# Gitea Sheets Sync Synchronizes Gitea issue events into a Google Sheets document with a production-ready worker, idempotency, and Google Sheets styling. This repository contains a small HTTP ingress for Gitea webhooks, an internal event bus, a background worker that processes issue events, and a Google Sheets adapter that persists canonical `Issues`, `Events`, `Comments`, `TimeEntries`, and `DeadLetters` sheets. Table of Contents - Overview - Architecture - Production checklist - Environment variables - Google authentication - Google Sheets schema - Running locally - Deploying to Render - Operational details (health, readiness, logging) - Security - Troubleshooting - Contributing Overview -------- This service listens for Gitea webhooks, verifies them using HMAC, extracts metadata, and records structured rows in a Google Sheets spreadsheet. It implements: - HMAC verification for webhook authenticity. - Event idempotency via `eventHash` and updating existing `Events` rows (no duplicate audit rows). - Per-issue queueing and retries with exponential backoff for Google Sheets API calls. - Styling and sizing of created sheets (frozen header, formats, widths, colors). - Time entry aggregation from Gitea's `/times` endpoint on issue close. Architecture ------------ - `src/server.js` — Express HTTP server exposing webhook endpoints and health/readiness endpoints. - `src/routes/` — Route definitions (wire HTTP to controllers). - `src/controllers/webhookController.js` — Verifies HMAC signatures and publishes normalized events to the internal bus. - `src/lib/eventBus.js` — Lightweight in-process pub/sub used to decouple HTTP and background workers. - `src/workers/issueProcessor.js` — Subscribes to events and orchestrates persistence and external calls. - `src/adapters/sheetsAdapter.js` — Google Sheets adapter: ensures sheets exist, enforces headers, styles sheets, appends/updates rows, and provides idempotent operations. - `src/lib/logger.js` — Structured logging helper (integrated with `morgan`). - `src/config/index.js` — Centralized configuration loaded from environment variables. Production checklist -------------------- Before deploying to production, ensure: - You have a Google service account with `Editor` access to the target spreadsheet (or at least `spreadsheets` write access). - The spreadsheet id is configured via `GOOGLE_SHEET_ID`. - `GITEA_WEBHOOK_SECRET` is set and configured in your Gitea webhook settings. - For time aggregation, `GITEA_APPLICATION_TOKEN` (personal access token or application token) is set. - All secrets are stored in your platform's secret store (do not commit `.env` or JSON keys). - Health/readiness probes are configured in the host to use `/health` and `/ready`. Environment variables --------------------- Required: - `GOOGLE_SHEET_ID` — Spreadsheet ID where sheets will be created/managed. - `GITEA_WEBHOOK_SECRET` — HMAC secret used to verify incoming webhooks. At least one of the Google auth options: - `GOOGLE_SERVICE_ACCOUNT_JSON` — Full service account JSON string (recommended to store as a secret). OR - `GOOGLE_APPLICATION_CREDENTIALS` — Path to a service account JSON file available to the process. OR - Legacy compatibility: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`. Optional: - `PORT` — Port to bind (default: `3000`). - `GITEA_APPLICATION_TOKEN` — Token used to call Gitea `/times` API for time aggregation. - `GITEA_API_BASE` — Base URL for your Gitea server (e.g., `https://gitea.example.com`). If not provided, the adapter attempts to derive it from the webhook payload. - `GOOGLE_SHEET_DEFAULT_ROWS` — Default number of rows when creating a sheet (default `100`). - `GOOGLE_EVENTS_SHEET`, `GOOGLE_ISSUES_SHEET`, `GOOGLE_COMMENTS_SHEET`, `GOOGLE_TIME_ENTRIES_SHEET`, `GOOGLE_DEADLETTERS_SHEET` — override sheet names. Google authentication --------------------- The adapter supports three ways to authenticate with Google: 1. `GOOGLE_SERVICE_ACCOUNT_JSON` — provide the full service account JSON as an environment variable (recommended to base64-encode in CI/CD and decode when injecting). 2. `GOOGLE_APPLICATION_CREDENTIALS` — file path to a service account JSON on disk. 3. Legacy: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`. Grant the service account Editor access to the target spreadsheet (Share the sheet to the `client_email` from the service account JSON). Google Sheets schema -------------------- The adapter creates and manages the following sheets (tabs) with canonical headers and basic styling: - `Issues` — canonical issue rows. Header columns: - `Number`, `Title`, `State`, `Labels`, `Assignees`, `CommentsCount`, `CreatedAt`, `CreatedBy`, `UpdatedAt`, `ClosedAt`, `ClosedBy`, `LastEvent`, `LastEventAt`, `LastEventHash`, `LastActor`, `IssueURL`, `EstimatedDays`, `Importance`, `TotalTimeMinutes`, `TimeSummary`, `_IssueKey` - `Events` — audit log of processing attempts. Header columns: - `ProcessedAt`, `EventHash`, `EventType`, `IssueKey`, `Actor`, `Summary`, `Status` - When `eventHash` matches an existing event, the adapter updates the `Status` and `ProcessedAt` in-place instead of appending duplicates. - `Comments` — issue comments. Header columns: - `ProcessedAt`, `IssueNumber`, `Actor`, `CommentURL` - `TimeEntries` — aggregated time entries per issue, per user. Header columns: - `ProcessedAt`, `IssueKey`, `IssueNumber`, `Assignee`, `UserID`, `TotalMinutes` - The adapter converts the time values returned from Gitea (seconds) into minutes, rounded. It upserts rows by `IssueNumber`+`UserID` to avoid duplicates for the same issue/user. - `Pushes` — organization-wide push events. Header columns: - `ProcessedAt`, `EventHash`, `RepoFullName`, `Ref`, `Branch`, `Pusher`, `PusherEmail`, `CommitCount`, `HeadCommitId`, `HeadCommitMessage`, `CompareURL`, `RepoURL`, `CommitsSummary` - Push rows are idempotent by `eventHash` (if the same push webhook is received twice the existing row is updated instead of duplicated). - Push rows are idempotent by `eventHash` (if the same push webhook is received twice the existing row is updated instead of duplicated). - `CommitsSummary` is truncated to a configurable per-commit snippet length and a configurable number of commits; a `CommitsTruncated` column indicates truncation. - Per-commit rows can be enabled using `PUSH_COMMITS_ENABLED=true`. When enabled a `PushCommits` sheet is created with one row per commit (upserts on EventHash+CommitId). Configuration for push commits: - `PUSH_COMMITS_ENABLED` — set to `true` to write per-commit rows to `PushCommits`. - `PUSH_COMMIT_SNIPPET_LENGTH` — maximum characters per commit message snippet (default 80). - `PUSH_COMMITS_SUMMARY_LIMIT` — maximum number of commits included in the `CommitsSummary` (default 10). - `DeadLetters` — failed events for manual inspection. Header columns: - `ProcessedAt`, `EventHash`, `IssueKey`, `Error`, `PayloadBase64`, `Notes` All sheets are styled when created or when headers change: frozen header row, bold gray header background, column widths, date formatting (`yyyy-MM-dd HH:mm:ss`) for date columns, number formats and text wrapping for long fields. Running locally --------------- 1. Install dependencies: ```bash npm install ``` 2. Set environment variables locally (for development only — use `.env` or your shell): ```bash export GOOGLE_SHEET_ID=your_spreadsheet_id export GOOGLE_SERVICE_ACCOUNT_JSON='{"type":...}' export GITEA_WEBHOOK_SECRET=your_secret export PORT=3000 ``` 3. Start the service: ```bash npm start # or: node src/server.js ``` 4. Health endpoints: - `GET /health` — returns 200 when server is running. - `GET /ready` — returns readiness status (checks Google Sheets auth + spreadsheet access). Testing webhooks locally ------------------------- To test webhooks, compute the HMAC sha256 signature using your `GITEA_WEBHOOK_SECRET` and send it in the `x-hub-signature-256` header in format `sha256=`. Example with Node (one-liner) to compute signature and send a sample payload: ```bash payload='{"action":"opened","issue":{"number":123}}' sig=$(node -e "const crypto=require('crypto'); const s=crypto.createHmac('sha256', process.env.GITEA_WEBHOOK_SECRET).update(process.argv[1]).digest('hex'); console.log('sha256='+s)" "$payload") curl -X POST http://localhost:3000/webhooks/gitea -H "content-type: application/json" -H "x-hub-signature-256: $sig" -d "$payload" ``` Deploying to Render -------------------- This service runs on Node and is Render-friendly. Minimal steps: 1. Create a new Web Service in Render and connect your Git repository. 2. Set the build command to: ``` npm ci ``` 3. Set the start command to: ``` npm start ``` 4. In Render's Dashboard, add the required environment variables (see Environment variables section). Provide `GOOGLE_SERVICE_ACCOUNT_JSON` or `GOOGLE_APPLICATION_CREDENTIALS` via Render's secret storage. Provide `GITEA_WEBHOOK_SECRET`, `GOOGLE_SHEET_ID`, and `GITEA_APPLICATION_TOKEN` (if you need time aggregation). 5. Set health check path to `/health` and readiness check to `/ready`. 6. (Optional) If you use `GOOGLE_APPLICATION_CREDENTIALS` and uploaded a secret file, ensure the file is available on disk to the process, or prefer `GOOGLE_SERVICE_ACCOUNT_JSON` as a Render secret. Notes for Render: - Bind port via `PORT` environment variable (Render sets this for you). The service listens on `process.env.PORT`. - Ensure the target spreadsheet is shared with the service account email. Operational details ------------------- - Logging: The service uses structured JSON-like logs via the internal logger and `morgan` for HTTP access logs. - Retries: The `sheetsAdapter` retries transient Google API errors with exponential backoff and jitter. - Idempotency: Events are identified by `eventHash`. The worker avoids reprocessing and the `Events` sheet is updated in-place for repeated processing attempts. - Graceful shutdown: The server supports SIGINT/SIGTERM and drains adapter queues before exit. Security -------- - Do not commit secrets or service account JSON files. Use the platform secrets store. - Rotate service-account keys if you believe any secret was leaked. - Limit the service account access to only the spreadsheet it needs to reduce blast radius. Troubleshooting --------------- - Authorization errors with Google Sheets: ensure the service account has access to the spreadsheet and the JSON credentials are valid. - `ready` endpoint returns false: check that `GOOGLE_SHEET_ID` is set and the service account has access. - Time aggregation failures: verify `GITEA_APPLICATION_TOKEN` is set and the Gitea instance `GITEA_API_BASE` is reachable from the service. Contributing / Development notes -------------------------------- - This codebase is organized into small, testable modules. To add business logic or change mapping rules, modify `src/adapters/sheetsAdapter.js` and `src/workers/issueProcessor.js`. - Keep `ensureSheet()` idempotent — it is safe to run multiple times and only modifies headers/styles when needed. Files of interest ----------------- - `src/server.js` — server bootstrap and route mounting. - `src/routes/giteaWebhook.js` — webhook route. - `src/controllers/webhookController.js` — HMAC verification and event normalization. - `src/workers/issueProcessor.js` — background processing and Gitea `/times` integration. - `src/adapters/sheetsAdapter.js` — Google Sheets I/O, header enforcement, styling, and upsert helpers. Contact ------- For questions about the code or deployment, open an issue in the repository.