diff --git a/README.md b/README.md index 510ad6f..a39d2e3 100644 --- a/README.md +++ b/README.md @@ -20,13 +20,14 @@ - [Step 5: Ollama Chat Model](#step-5-ollama-chat-model) - [Step 6: Buffer Memory](#step-6-buffer-memory) 8. [Upsert Documents](#upsert-documents) -9. [Using Cloud LLM APIs (Optional)](#using-cloud-llm-apis-optional) -10. [Deploy Chatbot to Users](#deploy-chatbot-to-users) +10. [Using Firecrawl (Scrape Websites into RAG)](#using-firecrawl-scrape-websites-into-rag) +11. [Using Cloud LLM APIs (Optional)](#using-cloud-llm-apis-optional) +12. [Deploy Chatbot to Users](#deploy-chatbot-to-users) - [Direct Link](#direct-link) - [Embed in Website](#embed-in-website) - [Session Management](#session-management) - [Rate Limiting](#rate-limiting) -11. [Troubleshooting](#troubleshooting) +13. [Troubleshooting](#troubleshooting) --- @@ -486,6 +487,93 @@ This limits each IP address to 20 messages per 60 seconds. --- +## Using Firecrawl (Scrape Websites into RAG) + +Firecrawl lets you scrape entire websites and feed the content directly into your Document Store as a knowledge base. This is useful if your software documentation lives on a website or wiki. + +--- + +### What Firecrawl Does + +``` +Your website / docs URL + ↓ + Firecrawl crawls all pages + ↓ + Returns clean Markdown text + ↓ + Loaded into Document Store + ↓ + Indexed into Faiss for RAG +``` + +--- + +### Option A: Firecrawl Cloud (Easiest) + +1. Sign up at [https://firecrawl.dev](https://firecrawl.dev) +2. Get your API key from the dashboard +3. Add the API key to Flowise: + - Go to **Settings** → **API Keys** → **Add Credential** + - Choose **Firecrawl API** and paste your key +--- + +### Option B: Self-Host Firecrawl (No API Cost) + +If you want to keep everything local and free: + +#### Requirements + +```bash +# Install Docker and Docker Compose +sudo apt-get install -y docker.io docker-compose +``` + +#### Setup + +```bash +# Clone Firecrawl repository +git clone https://github.com/mendableai/firecrawl.git +cd firecrawl + +# Copy environment file +cp .env.example .env +``` + +Edit the `.env` file: + +```bash +nano .env +``` + +Set these values at minimum: + +```env +NUM_WORKERS_PER_QUEUE=8 +PORT=3002 +HOST=0.0.0.0 +REDIS_URL=redis://redis:6379 +PLAYWRIGHT_MICROSERVICE_URL=http://playwright-service:3000/scrape +``` + +Start Firecrawl: + +```bash +docker-compose up -d +``` + +Verify it's running: + +```bash +curl http://localhost:3002/v1/scrape \ + -H "Content-Type: application/json" \ + -d '{"url": "https://example.com"}' +``` + +Your self-hosted Firecrawl API is now at `http://localhost:3002` + +--- + ## Troubleshooting ### ❌ `fetch failed` when chatting