220 lines
10 KiB
Markdown
220 lines
10 KiB
Markdown
# Gitea Sheets Sync
|
|
|
|
Synchronizes Gitea issue events into a Google Sheets document with a production-ready worker, idempotency, and Google Sheets styling.
|
|
|
|
This repository contains a small HTTP ingress for Gitea webhooks, an internal event bus, a background worker that processes issue events, and a Google Sheets adapter that persists canonical `Issues`, `Events`, `Comments`, `TimeEntries`, and `DeadLetters` sheets.
|
|
|
|
Table of Contents
|
|
- Overview
|
|
- Architecture
|
|
- Production checklist
|
|
- Environment variables
|
|
- Google authentication
|
|
- Google Sheets schema
|
|
- Running locally
|
|
- Deploying to Render
|
|
- Operational details (health, readiness, logging)
|
|
- Security
|
|
- Troubleshooting
|
|
- Contributing
|
|
|
|
Overview
|
|
--------
|
|
This service listens for Gitea webhooks, verifies them using HMAC, extracts metadata, and records structured rows in a Google Sheets spreadsheet. It implements:
|
|
|
|
- HMAC verification for webhook authenticity.
|
|
- Event idempotency via `eventHash` and updating existing `Events` rows (no duplicate audit rows).
|
|
- Per-issue queueing and retries with exponential backoff for Google Sheets API calls.
|
|
- Styling and sizing of created sheets (frozen header, formats, widths, colors).
|
|
- Time entry aggregation from Gitea's `/times` endpoint on issue close.
|
|
|
|
Architecture
|
|
------------
|
|
|
|
- `src/server.js` — Express HTTP server exposing webhook endpoints and health/readiness endpoints.
|
|
- `src/routes/` — Route definitions (wire HTTP to controllers).
|
|
- `src/controllers/webhookController.js` — Verifies HMAC signatures and publishes normalized events to the internal bus.
|
|
- `src/lib/eventBus.js` — Lightweight in-process pub/sub used to decouple HTTP and background workers.
|
|
- `src/workers/issueProcessor.js` — Subscribes to events and orchestrates persistence and external calls.
|
|
- `src/adapters/sheetsAdapter.js` — Google Sheets adapter: ensures sheets exist, enforces headers, styles sheets, appends/updates rows, and provides idempotent operations.
|
|
- `src/lib/logger.js` — Structured logging helper (integrated with `morgan`).
|
|
- `src/config/index.js` — Centralized configuration loaded from environment variables.
|
|
|
|
Production checklist
|
|
--------------------
|
|
|
|
Before deploying to production, ensure:
|
|
|
|
- You have a Google service account with `Editor` access to the target spreadsheet (or at least `spreadsheets` write access).
|
|
- The spreadsheet id is configured via `GOOGLE_SHEET_ID`.
|
|
- `GITEA_WEBHOOK_SECRET` is set and configured in your Gitea webhook settings.
|
|
- For time aggregation, `GITEA_APPLICATION_TOKEN` (personal access token or application token) is set.
|
|
- All secrets are stored in your platform's secret store (do not commit `.env` or JSON keys).
|
|
- Health/readiness probes are configured in the host to use `/health` and `/ready`.
|
|
|
|
Environment variables
|
|
---------------------
|
|
|
|
Required:
|
|
- `GOOGLE_SHEET_ID` — Spreadsheet ID where sheets will be created/managed.
|
|
- `GITEA_WEBHOOK_SECRET` — HMAC secret used to verify incoming webhooks.
|
|
|
|
At least one of the Google auth options:
|
|
- `GOOGLE_SERVICE_ACCOUNT_JSON` — Full service account JSON string (recommended to store as a secret). OR
|
|
- `GOOGLE_APPLICATION_CREDENTIALS` — Path to a service account JSON file available to the process. OR
|
|
- Legacy compatibility: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`.
|
|
|
|
Optional:
|
|
- `PORT` — Port to bind (default: `3000`).
|
|
- `GITEA_APPLICATION_TOKEN` — Token used to call Gitea `/times` API for time aggregation.
|
|
- `GITEA_API_BASE` — Base URL for your Gitea server (e.g., `https://gitea.example.com`). If not provided, the adapter attempts to derive it from the webhook payload.
|
|
- `GOOGLE_SHEET_DEFAULT_ROWS` — Default number of rows when creating a sheet (default `100`).
|
|
- `GOOGLE_EVENTS_SHEET`, `GOOGLE_ISSUES_SHEET`, `GOOGLE_COMMENTS_SHEET`, `GOOGLE_TIME_ENTRIES_SHEET`, `GOOGLE_DEADLETTERS_SHEET` — override sheet names.
|
|
|
|
Google authentication
|
|
---------------------
|
|
|
|
The adapter supports three ways to authenticate with Google:
|
|
|
|
1. `GOOGLE_SERVICE_ACCOUNT_JSON` — provide the full service account JSON as an environment variable (recommended to base64-encode in CI/CD and decode when injecting).
|
|
2. `GOOGLE_APPLICATION_CREDENTIALS` — file path to a service account JSON on disk.
|
|
3. Legacy: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`.
|
|
|
|
Grant the service account Editor access to the target spreadsheet (Share the sheet to the `client_email` from the service account JSON).
|
|
|
|
Google Sheets schema
|
|
--------------------
|
|
|
|
The adapter creates and manages the following sheets (tabs) with canonical headers and basic styling:
|
|
|
|
- `Issues` — canonical issue rows. Header columns:
|
|
- `Number`, `Title`, `State`, `Labels`, `Assignees`, `CommentsCount`, `CreatedAt`, `CreatedBy`, `UpdatedAt`, `ClosedAt`, `ClosedBy`, `LastEvent`, `LastEventAt`, `LastEventHash`, `LastActor`, `IssueURL`, `EstimatedDays`, `Importance`, `TotalTimeMinutes`, `TimeSummary`, `_IssueKey`
|
|
|
|
- `Events` — audit log of processing attempts. Header columns:
|
|
- `ProcessedAt`, `EventHash`, `EventType`, `IssueKey`, `Actor`, `Summary`, `Status`
|
|
- When `eventHash` matches an existing event, the adapter updates the `Status` and `ProcessedAt` in-place instead of appending duplicates.
|
|
|
|
- `Comments` — issue comments. Header columns:
|
|
- `ProcessedAt`, `IssueNumber`, `Actor`, `CommentURL`
|
|
|
|
- `TimeEntries` — aggregated time entries per issue, per user. Header columns:
|
|
- `ProcessedAt`, `IssueKey`, `IssueNumber`, `Assignee`, `UserID`, `TotalMinutes`
|
|
- The adapter converts the time values returned from Gitea (seconds) into minutes, rounded. It upserts rows by `IssueNumber`+`UserID` to avoid duplicates for the same issue/user.
|
|
|
|
- `DeadLetters` — failed events for manual inspection. Header columns:
|
|
- `ProcessedAt`, `EventHash`, `IssueKey`, `Error`, `PayloadBase64`, `Notes`
|
|
|
|
All sheets are styled when created or when headers change: frozen header row, bold gray header background, column widths, date formatting (`yyyy-MM-dd HH:mm:ss`) for date columns, number formats and text wrapping for long fields.
|
|
|
|
Running locally
|
|
---------------
|
|
|
|
1. Install dependencies:
|
|
|
|
```bash
|
|
npm install
|
|
```
|
|
|
|
2. Set environment variables locally (for development only — use `.env` or your shell):
|
|
|
|
```bash
|
|
export GOOGLE_SHEET_ID=your_spreadsheet_id
|
|
export GOOGLE_SERVICE_ACCOUNT_JSON='{"type":...}'
|
|
export GITEA_WEBHOOK_SECRET=your_secret
|
|
export PORT=3000
|
|
```
|
|
|
|
3. Start the service:
|
|
|
|
```bash
|
|
npm start
|
|
# or: node src/server.js
|
|
```
|
|
|
|
4. Health endpoints:
|
|
|
|
- `GET /health` — returns 200 when server is running.
|
|
- `GET /ready` — returns readiness status (checks Google Sheets auth + spreadsheet access).
|
|
|
|
Testing webhooks locally
|
|
-------------------------
|
|
|
|
To test webhooks, compute the HMAC sha256 signature using your `GITEA_WEBHOOK_SECRET` and send it in the `x-hub-signature-256` header in format `sha256=<hex>`.
|
|
|
|
Example with Node (one-liner) to compute signature and send a sample payload:
|
|
|
|
```bash
|
|
payload='{"action":"opened","issue":{"number":123}}'
|
|
sig=$(node -e "const crypto=require('crypto'); const s=crypto.createHmac('sha256', process.env.GITEA_WEBHOOK_SECRET).update(process.argv[1]).digest('hex'); console.log('sha256='+s)" "$payload")
|
|
curl -X POST http://localhost:3000/webhooks/gitea -H "content-type: application/json" -H "x-hub-signature-256: $sig" -d "$payload"
|
|
```
|
|
|
|
Deploying to Render
|
|
--------------------
|
|
|
|
This service runs on Node and is Render-friendly. Minimal steps:
|
|
|
|
1. Create a new Web Service in Render and connect your Git repository.
|
|
2. Set the build command to:
|
|
|
|
```
|
|
npm ci
|
|
```
|
|
|
|
3. Set the start command to:
|
|
|
|
```
|
|
npm start
|
|
```
|
|
|
|
4. In Render's Dashboard, add the required environment variables (see Environment variables section). Provide `GOOGLE_SERVICE_ACCOUNT_JSON` or `GOOGLE_APPLICATION_CREDENTIALS` via Render's secret storage. Provide `GITEA_WEBHOOK_SECRET`, `GOOGLE_SHEET_ID`, and `GITEA_APPLICATION_TOKEN` (if you need time aggregation).
|
|
|
|
5. Set health check path to `/health` and readiness check to `/ready`.
|
|
|
|
6. (Optional) If you use `GOOGLE_APPLICATION_CREDENTIALS` and uploaded a secret file, ensure the file is available on disk to the process, or prefer `GOOGLE_SERVICE_ACCOUNT_JSON` as a Render secret.
|
|
|
|
Notes for Render:
|
|
- Bind port via `PORT` environment variable (Render sets this for you). The service listens on `process.env.PORT`.
|
|
- Ensure the target spreadsheet is shared with the service account email.
|
|
|
|
Operational details
|
|
-------------------
|
|
|
|
- Logging: The service uses structured JSON-like logs via the internal logger and `morgan` for HTTP access logs.
|
|
- Retries: The `sheetsAdapter` retries transient Google API errors with exponential backoff and jitter.
|
|
- Idempotency: Events are identified by `eventHash`. The worker avoids reprocessing and the `Events` sheet is updated in-place for repeated processing attempts.
|
|
- Graceful shutdown: The server supports SIGINT/SIGTERM and drains adapter queues before exit.
|
|
|
|
Security
|
|
--------
|
|
|
|
- Do not commit secrets or service account JSON files. Use the platform secrets store.
|
|
- Rotate service-account keys if you believe any secret was leaked.
|
|
- Limit the service account access to only the spreadsheet it needs to reduce blast radius.
|
|
|
|
Troubleshooting
|
|
---------------
|
|
|
|
- Authorization errors with Google Sheets: ensure the service account has access to the spreadsheet and the JSON credentials are valid.
|
|
- `ready` endpoint returns false: check that `GOOGLE_SHEET_ID` is set and the service account has access.
|
|
- Time aggregation failures: verify `GITEA_APPLICATION_TOKEN` is set and the Gitea instance `GITEA_API_BASE` is reachable from the service.
|
|
|
|
Contributing / Development notes
|
|
--------------------------------
|
|
|
|
- This codebase is organized into small, testable modules. To add business logic or change mapping rules, modify `src/adapters/sheetsAdapter.js` and `src/workers/issueProcessor.js`.
|
|
- Keep `ensureSheet()` idempotent — it is safe to run multiple times and only modifies headers/styles when needed.
|
|
|
|
Files of interest
|
|
-----------------
|
|
|
|
- `src/server.js` — server bootstrap and route mounting.
|
|
- `src/routes/giteaWebhook.js` — webhook route.
|
|
- `src/controllers/webhookController.js` — HMAC verification and event normalization.
|
|
- `src/workers/issueProcessor.js` — background processing and Gitea `/times` integration.
|
|
- `src/adapters/sheetsAdapter.js` — Google Sheets I/O, header enforcement, styling, and upsert helpers.
|
|
|
|
Contact
|
|
-------
|
|
For questions about the code or deployment, open an issue in the repository.
|