Update README.md

README.md

@@ -22,12 +22,17 @@
 8. [Upsert Documents](#upsert-documents)
 10. [Using Firecrawl (Scrape Websites into RAG)](#using-firecrawl-scrape-websites-into-rag)
 11. [Using Cloud LLM APIs (Optional)](#using-cloud-llm-apis-optional)
-12. [Deploy Chatbot to Users](#deploy-chatbot-to-users)
+12. [Multilingual Support & Embeddings](#multilingual-support--embeddings)
+    - [How Language Works in the Pipeline](#how-language-works-in-the-pipeline)
+    - [The Golden Rule of Embeddings](#the-golden-rule-of-embeddings)
+    - [Choosing an Embedding Model](#choosing-an-embedding-model)
+    - [North African & Middle Eastern Users](#north-african--middle-eastern-users)
+13. [Deploy Chatbot to Users](#deploy-chatbot-to-users)
    - [Direct Link](#direct-link)
    - [Embed in Website](#embed-in-website)
    - [Session Management](#session-management)
    - [Rate Limiting](#rate-limiting)
-13. [Troubleshooting](#troubleshooting)
+14. [Troubleshooting](#troubleshooting)
 
 ---
 
@@ -572,6 +577,166 @@ curl http://localhost:3002/v1/scrape \
  
 Your self-hosted Firecrawl API is now at `http://localhost:3002`
  
+---
+ 
+## Multilingual Support & Embeddings
+ 
+### How Language Works in the Pipeline
+ 
+Your chatbot has two separate language-handling stages that behave differently:
+ 
+| Stage | Component | What it Does |
+|---|---|---|
+| **Retrieval** | Embedding model + Faiss | Finds relevant doc chunks |
+| **Generation** | LLM (Gemini, Ollama, etc.) | Writes the answer |
+ 
+The LLM (Gemini 2.5 Flash, qwen2.5, etc.) is naturally multilingual — it will respond in whatever language the user writes in. The weak point is always **retrieval**: if the embedding model can't match a French or Arabic query to English docs, the LLM receives empty context and responds with "I am not sure."
+ 
+```
+User asks in Arabic
+      ↓
+Embedding model converts query to vector
+      ↓
+Faiss searches English docs → poor match → returns nothing
+      ↓
+LLM receives empty context → "Hmm, I am not sure" ❌
+```
+ 
+---
+ 
+### The Golden Rule of Embeddings
+ 
+> **The model used during Upsert MUST be the same model used during Chat.**
+ 
+Each embedding model has its own internal "language" for converting text to vectors. They are not compatible with each other.
+ 
+```
+Upsert with nomic-embed-text:
+"product warranty" → [0.23, 0.87, 0.12, ...]
+ 
+Chat query with text-embedding-004:
+"product warranty" → [0.91, 0.04, 0.67, ...]
+ 
+Faiss compares these → completely different → no match ❌
+```
+ 
+If you switch embedding models you **must delete the old index and re-upsert**:
+ 
+```bash
+rm /root/.flowise/vectorstore/faiss.index
+rm /root/.flowise/vectorstore/faiss.json
+# Then hit Upsert again in Flowise
+```
+ 
+---
+ 
+### Choosing an Embedding Model
+ 
+| Model | Where | Arabic | French | Darija | Code-switching |
+|---|---|---|---|---|---|
+| `nomic-embed-text` | Ollama | ⚠️ Partial | ✅ Good | ❌ Poor | ❌ Poor |
+| `mxbai-embed-large` | Ollama | ✅ Good | ✅ Good | ⚠️ Partial | ⚠️ Partial |
+| `bge-m3` | Ollama | ✅ Excellent | ✅ Excellent | ⚠️ Partial | ✅ Best local |
+| `text-embedding-004` | Google API | ✅ Good | ✅ Good | ⚠️ Partial | ⚠️ Partial |
+| `text-embedding-3-small` | OpenAI API | ✅ Good | ✅ Good | ⚠️ Partial | ⚠️ Partial |
+| `multilingual-e5-large` | Ollama | ✅ Excellent | ✅ Excellent | ⚠️ Partial | ✅ Best local |
+ 
+**Recommendation:** Switch to `bge-m3` for the best multilingual retrieval without any API cost:
+ 
+```bash
+ollama pull bge-m3
+```
+ 
+Then update the **Ollama Embeddings** node model name to `bge-m3:latest`, delete the old index, and re-upsert.
+ 
+---
+ 
+### North African & Middle Eastern Users
+ 
+North African users (Algeria, Morocco, Tunisia) often write in **code-switched** messages mixing Arabic dialect (Darija) with French in the same sentence:
+ 
+```
+"كيفاش نdir la configuration?"
+"le produit مايخدمش properly"
+"comment تاع l'installation?"
+```
+ 
+This is called **code-switching** and no embedding model handles it perfectly — Darija is underrepresented in all training datasets. Use a combination of strategies:
+ 
+#### Strategy 1: System Prompt (Do This First)
+ 
+In **Conversational Retrieval QA Chain** → **Additional Parameters** → **System Message**:
+ 
+```
+You are a helpful product support assistant for North African users.
+Users may write in Arabic (Modern Standard or Darija dialect),
+French, or a mix of both languages in the same message.
+ 
+When you receive a message:
+1. Understand it regardless of the language mix
+2. Answer in the same language(s) the user wrote in
+3. If the retrieved context is insufficient, use your general
+   knowledge to help the user as best as possible
+4. Never respond with "I am not sure" without first attempting
+   to answer based on your knowledge
+```
+ 
+Your LLM (especially Gemini) understands Darija + French mixing very well — this prompt prevents it from giving up when retrieval returns weak results.
+ 
+#### Strategy 2: Add Mixed-Language Content to Your Docs
+ 
+Write a FAQ section in your Document Store using the way your users actually type:
+ 
+```
+كيفاش نinstalli le produit؟ / Comment installer le produit?
+→ Go to Settings → Install → follow the steps...
+ 
+le produit مايخدمش / Le produit ne fonctionne pas
+→ First check that your internet connection is active...
+ 
+واش فيه version جديدة؟ / Y a-t-il une nouvelle version?
+→ Check the Updates section in your dashboard...
+```
+ 
+This ensures even a weaker embedding model can match queries because the vocabulary overlaps directly with the docs.
+ 
+#### Strategy 3: Switch to bge-m3 (Best Local Option)
+ 
+```bash
+ollama pull bge-m3
+```
+ 
+`bge-m3` is trained on 100+ languages with strong cross-lingual alignment. It handles mixed-language sentences better than any other locally available model.
+ 
+#### Combined Recommended Setup for North Africa
+ 
+```
+1. Pull bge-m3:        ollama pull bge-m3
+2. Update Ollama Embeddings node → model: bge-m3:latest
+3. Add the system prompt above to your QA Chain
+4. Add Darija/French mixed FAQ to your Document Store
+5. Delete old index:   rm /root/.flowise/vectorstore/faiss.*
+6. Re-upsert
+```
+ 
+---
+ 
+### ❌ Bot Says "I am not sure" in Other Languages
+ 
+This is a retrieval failure, not an LLM failure. Debug it in order:
+ 
+**1. Check your system prompt is set** — the LLM needs permission to answer from its own knowledge when context is thin.
+ 
+**2. Check your embedding model** — if you're using `nomic-embed-text` with non-English queries, switch to `bge-m3`.
+ 
+**3. Verify the index exists after re-upserting:**
+```bash
+ls /root/.flowise/vectorstore/
+# Must show: faiss.index  faiss.json
+```
+ 
+**4. Add multilingual content to your docs** — even a few translated FAQ entries dramatically improve retrieval for those languages.
+
 ---
 
 ## Troubleshooting