Update README.md

2026-05-21 15:30:06 +02:00 · 2026-05-21 15:30:06 +02:00 · 4dc9f7669e
commit 4dc9f7669e
parent d6dd3a1b1e
1 changed files with 91 additions and 3 deletions
--- a/README.md
+++ b/README.md
@ -20,13 +20,14 @@
   - [Step 5: Ollama Chat Model](#step-5-ollama-chat-model)
   - [Step 6: Buffer Memory](#step-6-buffer-memory)
 8. [Upsert Documents](#upsert-documents)
-9. [Using Cloud LLM APIs (Optional)](#using-cloud-llm-apis-optional)
-10. [Deploy Chatbot to Users](#deploy-chatbot-to-users)
+10. [Using Firecrawl (Scrape Websites into RAG)](#using-firecrawl-scrape-websites-into-rag)
+11. [Using Cloud LLM APIs (Optional)](#using-cloud-llm-apis-optional)
+12. [Deploy Chatbot to Users](#deploy-chatbot-to-users)
   - [Direct Link](#direct-link)
   - [Embed in Website](#embed-in-website)
   - [Session Management](#session-management)
   - [Rate Limiting](#rate-limiting)
-11. [Troubleshooting](#troubleshooting)
+13. [Troubleshooting](#troubleshooting)

 ---

@ -486,6 +487,93 @@ This limits each IP address to 20 messages per 60 seconds.

 ---

+## Using Firecrawl (Scrape Websites into RAG)
+ 
+Firecrawl lets you scrape entire websites and feed the content directly into your Document Store as a knowledge base. This is useful if your software documentation lives on a website or wiki.
+ 
+---
+ 
+### What Firecrawl Does
+ 
+```
+Your website / docs URL
+        ↓
+  Firecrawl crawls all pages
+        ↓
+  Returns clean Markdown text
+        ↓
+  Loaded into Document Store
+        ↓
+  Indexed into Faiss for RAG
+```
+ 
+---
+ 
+### Option A: Firecrawl Cloud (Easiest)
+ 
+1. Sign up at [https://firecrawl.dev](https://firecrawl.dev)
+2. Get your API key from the dashboard
+3. Add the API key to Flowise:
+   - Go to **Settings** → **API Keys** → **Add Credential**
+   - Choose **Firecrawl API** and paste your key
+---
+ 
+### Option B: Self-Host Firecrawl (No API Cost)
+ 
+If you want to keep everything local and free:
+ 
+#### Requirements
+ 
+```bash
+# Install Docker and Docker Compose
+sudo apt-get install -y docker.io docker-compose
+```
+ 
+#### Setup
+ 
+```bash
+# Clone Firecrawl repository
+git clone https://github.com/mendableai/firecrawl.git
+cd firecrawl
+ 
+# Copy environment file
+cp .env.example .env
+```
+ 
+Edit the `.env` file:
+ 
+```bash
+nano .env
+```
+ 
+Set these values at minimum:
+ 
+```env
+NUM_WORKERS_PER_QUEUE=8
+PORT=3002
+HOST=0.0.0.0
+REDIS_URL=redis://redis:6379
+PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/scrape
+```
+ 
+Start Firecrawl:
+ 
+```bash
+docker-compose up -d
+```
+ 
+Verify it's running:
+ 
+```bash
+curl http://localhost:3002/v1/scrape \
+  -H "Content-Type: application/json" \
+  -d '{"url": "https://example.com"}'
+```
+ 
+Your self-hosted Firecrawl API is now at `http://localhost:3002`
+ 
+---
+
 ## Troubleshooting

 ### ❌ `fetch failed` when chatting