gitea-sheets-sync/README.md

Gitea Sheets Sync

Synchronizes Gitea issue events into a Google Sheets document with a production-ready worker, idempotency, and Google Sheets styling.

This repository contains a small HTTP ingress for Gitea webhooks, an internal event bus, a background worker that processes issue events, and a Google Sheets adapter that persists canonical Issues, Events, Comments, TimeEntries, and DeadLetters sheets.

Table of Contents

• Overview
• Architecture
• Production checklist
• Environment variables
• Google authentication
• Google Sheets schema
• Running locally
• Deploying to Render
• Operational details (health, readiness, logging)
• Security
• Troubleshooting
• Contributing

Overview

This service listens for Gitea webhooks, verifies them using HMAC, extracts metadata, and records structured rows in a Google Sheets spreadsheet. It implements:

• HMAC verification for webhook authenticity.
• Event idempotency via eventHash and updating existing Events rows (no duplicate audit rows).
• Per-issue queueing and retries with exponential backoff for Google Sheets API calls.
• Styling and sizing of created sheets (frozen header, formats, widths, colors).
• Time entry aggregation from Gitea's /times endpoint on issue close.

Architecture

• src/server.js — Express HTTP server exposing webhook endpoints and health/readiness endpoints.
• src/routes/ — Route definitions (wire HTTP to controllers).
• src/controllers/webhookController.js — Verifies HMAC signatures and publishes normalized events to the internal bus.
• src/lib/eventBus.js — Lightweight in-process pub/sub used to decouple HTTP and background workers.
• src/workers/issueProcessor.js — Subscribes to events and orchestrates persistence and external calls.
• src/adapters/sheetsAdapter.js — Google Sheets adapter: ensures sheets exist, enforces headers, styles sheets, appends/updates rows, and provides idempotent operations.
• src/lib/logger.js — Structured logging helper (integrated with morgan).
• src/config/index.js — Centralized configuration loaded from environment variables.

Production checklist

Before deploying to production, ensure:

• You have a Google service account with Editor access to the target spreadsheet (or at least spreadsheets write access).
• The spreadsheet id is configured via GOOGLE_SHEET_ID.
• GITEA_WEBHOOK_SECRET is set and configured in your Gitea webhook settings.
• For time aggregation, GITEA_APPLICATION_TOKEN (personal access token or application token) is set.
• All secrets are stored in your platform's secret store (do not commit .env or JSON keys).
• Health/readiness probes are configured in the host to use /health and /ready.

Environment variables

Required:

• GOOGLE_SHEET_ID — Spreadsheet ID where sheets will be created/managed.
• GITEA_WEBHOOK_SECRET — HMAC secret used to verify incoming webhooks.

At least one of the Google auth options:

• GOOGLE_SERVICE_ACCOUNT_JSON — Full service account JSON string (recommended to store as a secret). OR
• GOOGLE_APPLICATION_CREDENTIALS — Path to a service account JSON file available to the process. OR
• Legacy compatibility: GOOGLE_PRIVATE_KEY and GOOGLE_SERVICE_ACCOUNT_EMAIL.

Optional:

• PORT — Port to bind (default: 3000).
• GITEA_APPLICATION_TOKEN — Token used to call Gitea /times API for time aggregation.
• GITEA_API_BASE — Base URL for your Gitea server (e.g., https://gitea.example.com). If not provided, the adapter attempts to derive it from the webhook payload.
• GOOGLE_SHEET_DEFAULT_ROWS — Default number of rows when creating a sheet (default 100).
• GOOGLE_EVENTS_SHEET, GOOGLE_ISSUES_SHEET, GOOGLE_COMMENTS_SHEET, GOOGLE_TIME_ENTRIES_SHEET, GOOGLE_DEADLETTERS_SHEET — override sheet names.

Google authentication

The adapter supports three ways to authenticate with Google:

1. GOOGLE_SERVICE_ACCOUNT_JSON — provide the full service account JSON as an environment variable (recommended to base64-encode in CI/CD and decode when injecting).
2. GOOGLE_APPLICATION_CREDENTIALS — file path to a service account JSON on disk.
3. Legacy: GOOGLE_PRIVATE_KEY and GOOGLE_SERVICE_ACCOUNT_EMAIL.

Grant the service account Editor access to the target spreadsheet (Share the sheet to the client_email from the service account JSON).

Google Sheets schema

The adapter creates and manages the following sheets (tabs) with canonical headers and basic styling:

• Issues — canonical issue rows. Header columns:

  • Number, Title, State, Labels, Assignees, CommentsCount, CreatedAt, CreatedBy, UpdatedAt, ClosedAt, ClosedBy, LastEvent, LastEventAt, LastEventHash, LastActor, IssueURL, EstimatedDays, Importance, TotalTimeMinutes, TimeSummary, _IssueKey

• Events — audit log of processing attempts. Header columns:

  • ProcessedAt, EventHash, EventType, IssueKey, Actor, Summary, Status
  • When eventHash matches an existing event, the adapter updates the Status and ProcessedAt in-place instead of appending duplicates.

• Comments — issue comments. Header columns:

  • ProcessedAt, IssueNumber, Actor, CommentURL

• TimeEntries — aggregated time entries per issue, per user. Header columns:

  • ProcessedAt, IssueKey, IssueNumber, Assignee, UserID, TotalMinutes
  • The adapter converts the time values returned from Gitea (seconds) into minutes, rounded. It upserts rows by IssueNumber+UserID to avoid duplicates for the same issue/user.

• DeadLetters — failed events for manual inspection. Header columns:

  • ProcessedAt, EventHash, IssueKey, Error, PayloadBase64, Notes

All sheets are styled when created or when headers change: frozen header row, bold gray header background, column widths, date formatting (yyyy-MM-dd HH:mm:ss) for date columns, number formats and text wrapping for long fields.

Running locally

1. Install dependencies:

npm install

2. Set environment variables locally (for development only — use .env or your shell):

export GOOGLE_SHEET_ID=your_spreadsheet_id
export GOOGLE_SERVICE_ACCOUNT_JSON='{"type":...}'
export GITEA_WEBHOOK_SECRET=your_secret
export PORT=3000

3. Start the service:

npm start
# or: node src/server.js

4. Health endpoints:

• GET /health — returns 200 when server is running.
• GET /ready — returns readiness status (checks Google Sheets auth + spreadsheet access).

Testing webhooks locally

To test webhooks, compute the HMAC sha256 signature using your GITEA_WEBHOOK_SECRET and send it in the x-hub-signature-256 header in format sha256=<hex>.

Example with Node (one-liner) to compute signature and send a sample payload:

payload='{"action":"opened","issue":{"number":123}}'
sig=$(node -e "const crypto=require('crypto'); const s=crypto.createHmac('sha256', process.env.GITEA_WEBHOOK_SECRET).update(process.argv[1]).digest('hex'); console.log('sha256='+s)" "$payload")
curl -X POST http://localhost:3000/webhooks/gitea -H "content-type: application/json" -H "x-hub-signature-256: $sig" -d "$payload"

Deploying to Render

This service runs on Node and is Render-friendly. Minimal steps:

1. Create a new Web Service in Render and connect your Git repository.
2. Set the build command to:

npm ci

3. Set the start command to:

npm start

4. In Render's Dashboard, add the required environment variables (see Environment variables section). Provide GOOGLE_SERVICE_ACCOUNT_JSON or GOOGLE_APPLICATION_CREDENTIALS via Render's secret storage. Provide GITEA_WEBHOOK_SECRET, GOOGLE_SHEET_ID, and GITEA_APPLICATION_TOKEN (if you need time aggregation).

5. Set health check path to /health and readiness check to /ready.

6. (Optional) If you use GOOGLE_APPLICATION_CREDENTIALS and uploaded a secret file, ensure the file is available on disk to the process, or prefer GOOGLE_SERVICE_ACCOUNT_JSON as a Render secret.

Notes for Render:

• Bind port via PORT environment variable (Render sets this for you). The service listens on process.env.PORT.
• Ensure the target spreadsheet is shared with the service account email.

Operational details

• Logging: The service uses structured JSON-like logs via the internal logger and morgan for HTTP access logs.
• Retries: The sheetsAdapter retries transient Google API errors with exponential backoff and jitter.
• Idempotency: Events are identified by eventHash. The worker avoids reprocessing and the Events sheet is updated in-place for repeated processing attempts.
• Graceful shutdown: The server supports SIGINT/SIGTERM and drains adapter queues before exit.

Security

• Do not commit secrets or service account JSON files. Use the platform secrets store.
• Rotate service-account keys if you believe any secret was leaked.
• Limit the service account access to only the spreadsheet it needs to reduce blast radius.

Troubleshooting

• Authorization errors with Google Sheets: ensure the service account has access to the spreadsheet and the JSON credentials are valid.
• ready endpoint returns false: check that GOOGLE_SHEET_ID is set and the service account has access.
• Time aggregation failures: verify GITEA_APPLICATION_TOKEN is set and the Gitea instance GITEA_API_BASE is reachable from the service.

Contributing / Development notes

• This codebase is organized into small, testable modules. To add business logic or change mapping rules, modify src/adapters/sheetsAdapter.js and src/workers/issueProcessor.js.
• Keep ensureSheet() idempotent — it is safe to run multiple times and only modifies headers/styles when needed.

Files of interest

• src/server.js — server bootstrap and route mounting.
• src/routes/giteaWebhook.js — webhook route.
• src/controllers/webhookController.js — HMAC verification and event normalization.
• src/workers/issueProcessor.js — background processing and Gitea /times integration.
• src/adapters/sheetsAdapter.js — Google Sheets I/O, header enforcement, styling, and upsert helpers.

Contact

For questions about the code or deployment, open an issue in the repository.