TrizTech/gitea-sheets-sync
15
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

release(v1.0)

Browse Source
This commit is contained in:
Othmane Ataallah 2026-05-03 14:28:21 +01:00
commit 2cc258daba
15 changed files with 2912 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.gitignore vendored Normal file
View File

@ -0,0 +1,6 @@
# Ignore local environment and service account credentials
.env
service-account.json
# Node modules
node_modules/

219
README.md Normal file
View File

@ -0,0 +1,219 @@
# Gitea Sheets Sync
Synchronizes Gitea issue events into a Google Sheets document with a production-ready worker, idempotency, and Google Sheets styling.
This repository contains a small HTTP ingress for Gitea webhooks, an internal event bus, a background worker that processes issue events, and a Google Sheets adapter that persists canonical `Issues`, `Events`, `Comments`, `TimeEntries`, and `DeadLetters` sheets.
Table of Contents
- Overview
- Architecture
- Production checklist
- Environment variables
- Google authentication
- Google Sheets schema
- Running locally
- Deploying to Render
- Operational details (health, readiness, logging)
- Security
- Troubleshooting
- Contributing
Overview
--------
This service listens for Gitea webhooks, verifies them using HMAC, extracts metadata, and records structured rows in a Google Sheets spreadsheet. It implements:
- HMAC verification for webhook authenticity.
- Event idempotency via `eventHash` and updating existing `Events` rows (no duplicate audit rows).
- Per-issue queueing and retries with exponential backoff for Google Sheets API calls.
- Styling and sizing of created sheets (frozen header, formats, widths, colors).
- Time entry aggregation from Gitea's `/times` endpoint on issue close.
Architecture
------------
- `src/server.js` — Express HTTP server exposing webhook endpoints and health/readiness endpoints.
- `src/routes/` — Route definitions (wire HTTP to controllers).
- `src/controllers/webhookController.js` — Verifies HMAC signatures and publishes normalized events to the internal bus.
- `src/lib/eventBus.js` — Lightweight in-process pub/sub used to decouple HTTP and background workers.
- `src/workers/issueProcessor.js` — Subscribes to events and orchestrates persistence and external calls.
- `src/adapters/sheetsAdapter.js` — Google Sheets adapter: ensures sheets exist, enforces headers, styles sheets, appends/updates rows, and provides idempotent operations.
- `src/lib/logger.js` — Structured logging helper (integrated with `morgan`).
- `src/config/index.js` — Centralized configuration loaded from environment variables.
Production checklist
--------------------
Before deploying to production, ensure:
- You have a Google service account with `Editor` access to the target spreadsheet (or at least `spreadsheets` write access).
- The spreadsheet id is configured via `GOOGLE_SHEET_ID`.
- `GITEA_WEBHOOK_SECRET` is set and configured in your Gitea webhook settings.
- For time aggregation, `GITEA_APPLICATION_TOKEN` (personal access token or application token) is set.
- All secrets are stored in your platform's secret store (do not commit `.env` or JSON keys).
- Health/readiness probes are configured in the host to use `/health` and `/ready`.
Environment variables
---------------------
Required:
- `GOOGLE_SHEET_ID` — Spreadsheet ID where sheets will be created/managed.
- `GITEA_WEBHOOK_SECRET` — HMAC secret used to verify incoming webhooks.
At least one of the Google auth options:
- `GOOGLE_SERVICE_ACCOUNT_JSON` — Full service account JSON string (recommended to store as a secret). OR
- `GOOGLE_APPLICATION_CREDENTIALS` — Path to a service account JSON file available to the process. OR
- Legacy compatibility: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`.
Optional:
- `PORT` — Port to bind (default: `3000`).
- `GITEA_APPLICATION_TOKEN` — Token used to call Gitea `/times` API for time aggregation.
- `GITEA_API_BASE` — Base URL for your Gitea server (e.g., `https://gitea.example.com`). If not provided, the adapter attempts to derive it from the webhook payload.
- `GOOGLE_SHEET_DEFAULT_ROWS` — Default number of rows when creating a sheet (default `100`).
- `GOOGLE_EVENTS_SHEET`, `GOOGLE_ISSUES_SHEET`, `GOOGLE_COMMENTS_SHEET`, `GOOGLE_TIME_ENTRIES_SHEET`, `GOOGLE_DEADLETTERS_SHEET` — override sheet names.
Google authentication
---------------------
The adapter supports three ways to authenticate with Google:
1. `GOOGLE_SERVICE_ACCOUNT_JSON` — provide the full service account JSON as an environment variable (recommended to base64-encode in CI/CD and decode when injecting).
2. `GOOGLE_APPLICATION_CREDENTIALS` — file path to a service account JSON on disk.
3. Legacy: `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL`.
Grant the service account Editor access to the target spreadsheet (Share the sheet to the `client_email` from the service account JSON).
Google Sheets schema
--------------------
The adapter creates and manages the following sheets (tabs) with canonical headers and basic styling:
- `Issues` — canonical issue rows. Header columns:
- `Number`, `Title`, `State`, `Labels`, `Assignees`, `CommentsCount`, `CreatedAt`, `CreatedBy`, `UpdatedAt`, `ClosedAt`, `ClosedBy`, `LastEvent`, `LastEventAt`, `LastEventHash`, `LastActor`, `IssueURL`, `EstimatedDays`, `Importance`, `TotalTimeMinutes`, `TimeSummary`, `_IssueKey`
- `Events` — audit log of processing attempts. Header columns:
- `ProcessedAt`, `EventHash`, `EventType`, `IssueKey`, `Actor`, `Summary`, `Status`
- When `eventHash` matches an existing event, the adapter updates the `Status` and `ProcessedAt` in-place instead of appending duplicates.
- `Comments` — issue comments. Header columns:
- `ProcessedAt`, `IssueNumber`, `Actor`, `CommentURL`
- `TimeEntries` — aggregated time entries per issue, per user. Header columns:
- `ProcessedAt`, `IssueKey`, `IssueNumber`, `Assignee`, `UserID`, `TotalMinutes`
- The adapter converts the time values returned from Gitea (seconds) into minutes, rounded. It upserts rows by `IssueNumber`+`UserID` to avoid duplicates for the same issue/user.
- `DeadLetters` — failed events for manual inspection. Header columns:
- `ProcessedAt`, `EventHash`, `IssueKey`, `Error`, `PayloadBase64`, `Notes`
All sheets are styled when created or when headers change: frozen header row, bold gray header background, column widths, date formatting (`yyyy-MM-dd HH:mm:ss`) for date columns, number formats and text wrapping for long fields.
Running locally
---------------
1. Install dependencies:
```bash
npm install
```
2. Set environment variables locally (for development only — use `.env` or your shell):
```bash
export GOOGLE_SHEET_ID=your_spreadsheet_id
export GOOGLE_SERVICE_ACCOUNT_JSON='{"type":...}'
export GITEA_WEBHOOK_SECRET=your_secret
export PORT=3000
```
3. Start the service:
```bash
npm start
# or: node src/server.js
```
4. Health endpoints:
- `GET /health` — returns 200 when server is running.
- `GET /ready` — returns readiness status (checks Google Sheets auth + spreadsheet access).
Testing webhooks locally
-------------------------
To test webhooks, compute the HMAC sha256 signature using your `GITEA_WEBHOOK_SECRET` and send it in the `x-hub-signature-256` header in format `sha256=<hex>`.
Example with Node (one-liner) to compute signature and send a sample payload:
```bash
payload='{"action":"opened","issue":{"number":123}}'
sig=$(node -e "const crypto=require('crypto'); const s=crypto.createHmac('sha256', process.env.GITEA_WEBHOOK_SECRET).update(process.argv[1]).digest('hex'); console.log('sha256='+s)" "$payload")
curl -X POST http://localhost:3000/webhooks/gitea -H "content-type: application/json" -H "x-hub-signature-256: $sig" -d "$payload"
```
Deploying to Render
--------------------
This service runs on Node and is Render-friendly. Minimal steps:
1. Create a new Web Service in Render and connect your Git repository.
2. Set the build command to:
```
npm ci
```
3. Set the start command to:
```
npm start
```
4. In Render's Dashboard, add the required environment variables (see Environment variables section). Provide `GOOGLE_SERVICE_ACCOUNT_JSON` or `GOOGLE_APPLICATION_CREDENTIALS` via Render's secret storage. Provide `GITEA_WEBHOOK_SECRET`, `GOOGLE_SHEET_ID`, and `GITEA_APPLICATION_TOKEN` (if you need time aggregation).
5. Set health check path to `/health` and readiness check to `/ready`.
6. (Optional) If you use `GOOGLE_APPLICATION_CREDENTIALS` and uploaded a secret file, ensure the file is available on disk to the process, or prefer `GOOGLE_SERVICE_ACCOUNT_JSON` as a Render secret.
Notes for Render:
- Bind port via `PORT` environment variable (Render sets this for you). The service listens on `process.env.PORT`.
- Ensure the target spreadsheet is shared with the service account email.
Operational details
-------------------
- Logging: The service uses structured JSON-like logs via the internal logger and `morgan` for HTTP access logs.
- Retries: The `sheetsAdapter` retries transient Google API errors with exponential backoff and jitter.
- Idempotency: Events are identified by `eventHash`. The worker avoids reprocessing and the `Events` sheet is updated in-place for repeated processing attempts.
- Graceful shutdown: The server supports SIGINT/SIGTERM and drains adapter queues before exit.
Security
--------
- Do not commit secrets or service account JSON files. Use the platform secrets store.
- Rotate service-account keys if you believe any secret was leaked.
- Limit the service account access to only the spreadsheet it needs to reduce blast radius.
Troubleshooting
---------------
- Authorization errors with Google Sheets: ensure the service account has access to the spreadsheet and the JSON credentials are valid.
- `ready` endpoint returns false: check that `GOOGLE_SHEET_ID` is set and the service account has access.
- Time aggregation failures: verify `GITEA_APPLICATION_TOKEN` is set and the Gitea instance `GITEA_API_BASE` is reachable from the service.
Contributing / Development notes
--------------------------------
- This codebase is organized into small, testable modules. To add business logic or change mapping rules, modify `src/adapters/sheetsAdapter.js` and `src/workers/issueProcessor.js`.
- Keep `ensureSheet()` idempotent — it is safe to run multiple times and only modifies headers/styles when needed.
Files of interest
-----------------
- `src/server.js` — server bootstrap and route mounting.
- `src/routes/giteaWebhook.js` — webhook route.
- `src/controllers/webhookController.js` — HMAC verification and event normalization.
- `src/workers/issueProcessor.js` — background processing and Gitea `/times` integration.
- `src/adapters/sheetsAdapter.js` — Google Sheets I/O, header enforcement, styling, and upsert helpers.
Contact
-------
For questions about the code or deployment, open an issue in the repository.

32
SECRETS.md Normal file
View File

@ -0,0 +1,32 @@
Secrets & deployment
This repository contained a committed Google service account and a local `.env` file with private keys. Those files were removed from the repository.
How to provide credentials securely in your environment:
Options (pick one):
1) Set `GOOGLE_APPLICATION_CREDENTIALS` to the path of a service account JSON file on the host (do NOT commit the file):
- Upload the JSON to a secure location on the host (e.g. `/etc/secrets/gitea-sync-sa.json`) and set:
export GOOGLE_APPLICATION_CREDENTIALS=/etc/secrets/gitea-sync-sa.json
2) Set `GOOGLE_SERVICE_ACCOUNT_JSON` to the full JSON string (base64 encoding recommended):
- Encode the JSON and provide it as an env var in your process manager or CI/CD secret store:
export GOOGLE_SERVICE_ACCOUNT_JSON='{"type":...}'
- For CI/CD you can store it as a secret and map it into the runtime.
3) Legacy: set `GOOGLE_PRIVATE_KEY` and `GOOGLE_SERVICE_ACCOUNT_EMAIL` separately as env vars. This is supported for backward compatibility but less convenient.
Other notes:
- Do NOT commit `.env` or service account files. Add them to project `.gitignore` (done).
- Rotate service account keys immediately if these keys were used publicly.
- Use a secret manager (AWS Secrets Manager, GCP Secret Manager, Vault) in production.
Webhook secret:
- Provide `GITEA_WEBHOOK_SECRET` as an env var in your runtime environment.
- Do NOT commit it to the repository.

1349
package-lock.json generated Normal file
View File

File diff suppressed because it is too large Load Diff

20
package.json Normal file
View File

@ -0,0 +1,20 @@
{
"name": "gitea-sheets-sync",
"version": "1.0.0",
"description": "",
"license": "ISC",
"author": "",
"type": "module",
"main": "src/server.js",
"scripts": {
"start": "node src/server.js",
"test": "echo \"Error: no test specified\" && exit 1"
},
"dependencies": {
"axios": "^1.15.2",
"dotenv": "^17.4.2",
"express": "^5.2.1",
"googleapis": "^171.4.0",
"morgan": "^1.10.1"
}
}

683
src/adapters/sheetsAdapter.js Normal file
View File

@ -0,0 +1,683 @@
import { google } from "googleapis";
import axios from "axios";
import fs from "fs";
import path from "path";
import logger from "../lib/logger.js";
import config from "../config/index.js";
const ISSUES_SHEET = process.env.GOOGLE_ISSUES_SHEET || "Issues";
const EVENTS_SHEET = process.env.GOOGLE_EVENTS_SHEET || "Events";
const COMMENTS_SHEET = process.env.GOOGLE_COMMENTS_SHEET || "Comments";
// Issues: display-focused header (internal _IssueKey kept at end for lookups)
const ISSUES_HEADER = [
"Number",
"Title",
"State",
"Labels",
"Assignees",
"CommentsCount",
"CreatedAt",
"CreatedBy",
"UpdatedAt",
"ClosedAt",
"ClosedBy",
"LastEvent",
"LastEventAt",
"LastEventHash",
"LastActor",
"IssueURL",
"EstimatedDays",
"Importance",
"TotalTimeMinutes",
"TimeSummary",
"_IssueKey",
];
const EVENTS_HEADER = ["ProcessedAt", "EventHash", "EventType", "IssueKey", "Actor", "Summary", "Status"];
const DEADLETTERS_SHEET = process.env.GOOGLE_DEADLETTERS_SHEET || "DeadLetters";
const DEADLETTERS_HEADER = ["ProcessedAt", "EventHash", "IssueKey", "Error", "PayloadBase64", "Notes"];
const COMMENTS_HEADER = process.env.GOOGLE_COMMENTS_HEADER ? JSON.parse(process.env.GOOGLE_COMMENTS_HEADER) : ["ProcessedAt", "IssueNumber", "Actor", "CommentURL"];
const TIME_ENTRIES_SHEET = process.env.GOOGLE_TIME_ENTRIES_SHEET || "TimeEntries";
const TIME_ENTRIES_HEADER = ["ProcessedAt", "IssueKey", "IssueNumber", "Assignee", "UserID", "TotalMinutes"];
// Default number of rows to keep on a freshly-created sheet. Can be overridden with env var
const DEFAULT_SHEET_ROWS = Number(process.env.GOOGLE_SHEET_DEFAULT_ROWS || 100);
const ensuredSheets = new Set();
function colNumberToLetter(n) {
let s = "";
while (n > 0) {
const m = (n - 1) % 26;
s = String.fromCharCode(65 + m) + s;
n = Math.floor((n - m) / 26);
}
return s;
}
async function ensureSheet(sheets, spreadsheetId, sheetName, header) {
const cacheKey = `${spreadsheetId}:${sheetName}`;
if (ensuredSheets.has(cacheKey)) return;
// Ensure sheet exists
const meta = await requestWithRetry(() => sheets.spreadsheets.get({ spreadsheetId, fields: "sheets.properties" }));
const sheetsList = (meta && meta.data && meta.data.sheets) || [];
const exists = sheetsList.some((s) => s.properties && s.properties.title === sheetName);
let sheetCreated = false;
if (!exists) {
await requestWithRetry(() =>
sheets.spreadsheets.batchUpdate({
spreadsheetId,
requestBody: { requests: [{ addSheet: { properties: { title: sheetName } } }] },
})
);
logger.info("sheetsAdapter created sheet", { sheetName });
sheetCreated = true;
}
// Ensure header
const headerRange = `${sheetName}!A1:${colNumberToLetter(header.length)}1`;
const res = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range: headerRange }));
const values = (res && res.data && res.data.values) || [];
const firstRow = values[0] || [];
// If header cells are missing or different, overwrite
let needSet = false;
if (firstRow.length < header.length) needSet = true;
else {
for (let i = 0; i < header.length; i++) {
if (String(firstRow[i] || "").trim() !== header[i]) {
needSet = true;
break;
}
}
}
if (needSet) {
await requestWithRetry(() =>
sheets.spreadsheets.values.update({
spreadsheetId,
range: headerRange,
valueInputOption: "RAW",
requestBody: { values: [header] },
})
);
logger.info("sheetsAdapter set header for sheet", { sheetName });
}
// Apply styling only when the sheet was newly created or header was reset
if (needSet || sheetCreated) {
try {
// re-fetch to get the sheetId
const meta2 = await requestWithRetry(() => sheets.spreadsheets.get({ spreadsheetId, fields: "sheets.properties" }));
const sheetObj = (meta2 && meta2.data && meta2.data.sheets || []).find((s) => s.properties && s.properties.title === sheetName);
const sheetId = sheetObj && sheetObj.properties && sheetObj.properties.sheetId;
if (sheetId != null) {
const requests = [];
// Freeze the header row, and set grid size (rows/columns) to remove unused space
requests.push({
updateSheetProperties: {
properties: { sheetId, gridProperties: { frozenRowCount: 1, rowCount: DEFAULT_SHEET_ROWS, columnCount: header.length } },
fields: "gridProperties.frozenRowCount,gridProperties.rowCount,gridProperties.columnCount",
},
});
// Header styling: light gray background + bold
requests.push({
repeatCell: {
range: { sheetId, startRowIndex: 0, endRowIndex: 1, startColumnIndex: 0, endColumnIndex: header.length },
cell: { userEnteredFormat: { backgroundColor: { red: 0.95, green: 0.95, blue: 0.95 }, textFormat: { bold: true } } },
fields: "userEnteredFormat(backgroundColor,textFormat)",
},
});
// sensible default widths per column name
const defaultWidthsByHeader = {
Number: 80,
Title: 360,
State: 90,
Labels: 200,
Assignees: 160,
CommentsCount: 90,
CreatedAt: 170,
CreatedBy: 140,
UpdatedAt: 170,
ClosedAt: 170,
ClosedBy: 140,
LastEvent: 120,
LastEventAt: 170,
LastEventHash: 220,
LastActor: 140,
IssueURL: 260,
EstimatedDays: 90,
Importance: 90,
TotalTimeMinutes: 110,
TimeSummary: 260,
_IssueKey: 220,
ProcessedAt: 170,
EventHash: 220,
EventType: 140,
Actor: 140,
Summary: 260,
Status: 120,
IssueNumber: 90,
Assignee: 160,
UserID: 120,
TotalMinutes: 110,
};
const dateCols = new Set(["CreatedAt", "UpdatedAt", "ClosedAt", "LastEventAt", "ProcessedAt"]);
const numberCols = new Set(["CommentsCount", "Number", "TotalTimeMinutes", "EstimatedDays", "Importance", "IssueNumber", "TotalMinutes"]);
const wrapCols = new Set(["Title", "Labels", "TimeSummary", "Summary", "IssueURL"]);
for (let i = 0; i < header.length; i++) {
const key = header[i];
const width = defaultWidthsByHeader[key] || 150;
// set column width
requests.push({
updateDimensionProperties: {
range: { sheetId, dimension: "COLUMNS", startIndex: i, endIndex: i + 1 },
properties: { pixelSize: width },
fields: "pixelSize",
},
});
// apply formats for the column (rows 2..10000)
const colRange = { sheetId, startRowIndex: 1, endRowIndex: 10000, startColumnIndex: i, endColumnIndex: i + 1 };
if (dateCols.has(key)) {
requests.push({
repeatCell: {
range: colRange,
cell: { userEnteredFormat: { numberFormat: { type: "DATE_TIME", pattern: "yyyy-MM-dd HH:mm:ss" }, horizontalAlignment: "CENTER" } },
fields: "userEnteredFormat(numberFormat,horizontalAlignment)",
},
});
} else if (numberCols.has(key)) {
requests.push({
repeatCell: {
range: colRange,
cell: { userEnteredFormat: { numberFormat: { type: "NUMBER", pattern: "#,##0" }, horizontalAlignment: "CENTER" } },
fields: "userEnteredFormat(numberFormat,horizontalAlignment)",
},
});
} else if (wrapCols.has(key)) {
requests.push({
repeatCell: {
range: colRange,
cell: { userEnteredFormat: { wrapStrategy: "WRAP", horizontalAlignment: "LEFT" } },
fields: "userEnteredFormat(wrapStrategy,horizontalAlignment)",
},
});
} else {
// default alignment
requests.push({
repeatCell: {
range: colRange,
cell: { userEnteredFormat: { horizontalAlignment: "CENTER" } },
fields: "userEnteredFormat(horizontalAlignment)",
},
});
}
}
if (requests.length > 0) {
await requestWithRetry(() => sheets.spreadsheets.batchUpdate({ spreadsheetId, requestBody: { requests } }));
logger.info("sheetsAdapter applied sheet styling", { sheetName });
}
}
} catch (e) {
logger.warn("sheetsAdapter styling failed", { sheetName, err: e && e.message ? e.message : e });
}
}
ensuredSheets.add(cacheKey);
}
// Simple per-key queue to serialize operations for the same issue number
const queues = new Map();
function enqueue(key, work) {
const prev = queues.get(key) || Promise.resolve();
const next = prev
.then(() => work())
.catch((err) => {
logger.error("sheetsAdapter queue worker error", { key, err: err && err.message ? err.message : err });
throw err;
})
.then((res) => {
if (queues.get(key) === next) queues.delete(key);
return res;
});
queues.set(key, next);
return next;
}
function parseRetryAfter(header) {
if (!header) return null;
const s = String(header).trim();
// If numeric, seconds
if (/^\d+$/.test(s)) return Number(s) * 1000;
// try parse date
const ts = Date.parse(s);
if (!isNaN(ts)) return Math.max(ts - Date.now(), 0);
return null;
}
function formatDate(iso) {
try {
const d = new Date(iso);
if (Number.isNaN(d.getTime())) return "";
const YYYY = d.getFullYear();
const MM = String(d.getMonth() + 1).padStart(2, "0");
const DD = String(d.getDate()).padStart(2, "0");
const hh = String(d.getHours()).padStart(2, "0");
const mm = String(d.getMinutes()).padStart(2, "0");
const ss = String(d.getSeconds()).padStart(2, "0");
return `${YYYY}-${MM}-${DD} ${hh}:${mm}:${ss}`;
} catch (e) {
return "";
}
}
function sleep(ms) {
return new Promise((r) => setTimeout(r, ms));
}
function isTransientError(err) {
const status = err && err.response && err.response.status;
if (!status) return false;
return status === 429 || status >= 500;
}
async function sheetsClient() {
// Prefer JSON creds via env
if (process.env.GOOGLE_SERVICE_ACCOUNT_JSON) {
let creds;
try {
creds = JSON.parse(process.env.GOOGLE_SERVICE_ACCOUNT_JSON);
} catch (e) {
throw new Error("Invalid GOOGLE_SERVICE_ACCOUNT_JSON");
}
const auth = new google.auth.GoogleAuth({ credentials: creds, scopes: ["https://www.googleapis.com/auth/spreadsheets"] });
const client = await auth.getClient();
return google.sheets({ version: "v4", auth: client });
}
// Key file path
if (process.env.GOOGLE_APPLICATION_CREDENTIALS) {
const auth = new google.auth.GoogleAuth({ keyFilename: process.env.GOOGLE_APPLICATION_CREDENTIALS, scopes: ["https://www.googleapis.com/auth/spreadsheets"] });
const client = await auth.getClient();
return google.sheets({ version: "v4", auth: client });
}
// local service-account.json fallback
const local = path.resolve(process.cwd(), "service-account.json");
if (fs.existsSync(local)) {
const raw = fs.readFileSync(local, "utf8");
const creds = JSON.parse(raw);
const auth = new google.auth.GoogleAuth({ credentials: creds, scopes: ["https://www.googleapis.com/auth/spreadsheets"] });
const client = await auth.getClient();
return google.sheets({ version: "v4", auth: client });
}
// legacy env vars
if (process.env.GOOGLE_PRIVATE_KEY && process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL) {
const creds = {
client_email: process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL,
private_key: process.env.GOOGLE_PRIVATE_KEY.replace(/\\n/g, "\n"),
};
const auth = new google.auth.GoogleAuth({ credentials: creds, scopes: ["https://www.googleapis.com/auth/spreadsheets"] });
const client = await auth.getClient();
return google.sheets({ version: "v4", auth: client });
}
throw new Error("No Google service account credentials available");
}
function mapIssueToRow(issue, action, meta = {}) {
// Compute normalized labels and assignees
const labels = Array.isArray(issue.labels) ? issue.labels.map((l) => (typeof l === 'string' ? l : l.name)).filter(Boolean) : [];
const labelStr = labels.map((s) => String(s).trim()).sort().join(",");
const assignees = issue.assignees || (issue.assignee ? [issue.assignee] : []);
const assigneeStr = Array.isArray(assignees) ? assignees.map((a) => (a && a.login) || (typeof a === 'string' ? a : '')).filter(Boolean).join(",") : "";
const extra = {};
if (issue.milestone) extra.milestone = issue.milestone;
const createdAt = issue.created_at || issue.createdAt || new Date().toISOString();
const updatedAt = issue.updated_at || issue.updatedAt || new Date().toISOString();
const closedAt = issue.closed_at || issue.closedAt || "";
// TimeSummary will be a JSON string mapping user->minutes
const timeSummary = meta.timeSummary || "";
return [
issue.number || "",
issue.title || "",
issue.state || "",
labelStr,
assigneeStr,
issue.comments || 0,
formatDate(createdAt),
(issue.user && issue.user.login) || meta.createdBy || "",
formatDate(updatedAt),
closedAt ? formatDate(closedAt) : "",
meta.closedBy || "",
action || "",
formatDate(new Date().toISOString()),
meta.eventHash || "",
meta.actor || (issue.user && issue.user.login) || "",
(issue.html_url || issue.url) || "",
meta.estimatedDays != null ? meta.estimatedDays : "",
meta.importance != null ? meta.importance : "",
meta.totalTimeMinutes != null ? meta.totalTimeMinutes : "",
timeSummary || "",
meta.issueKey || `${(issue.repository && issue.repository.full_name) || ''}#${issue.number}`,
];
}
async function requestWithRetry(fn, opts = {}) {
const maxRetries = opts.maxRetries || 4;
const base = opts.baseDelay || 500; // ms
const maxDelay = opts.maxDelay || 30000; // ms
let attempt = 0;
while (true) {
try {
return await fn();
} catch (err) {
attempt += 1;
if (!isTransientError(err) || attempt > maxRetries) throw err;
const ra = err.response && err.response.headers && err.response.headers["retry-after"];
const retryAfter = parseRetryAfter(ra);
let delay = retryAfter != null ? retryAfter : Math.min(base * Math.pow(2, attempt - 1), maxDelay);
// add jitter
delay = Math.floor(delay + Math.random() * 300);
logger.warn("sheetsAdapter transient error, retrying", { attempt, delay, err: err && err.message ? err.message : err });
await sleep(delay);
continue;
}
}
}
async function findRowIndexByIssueKey(sheets, spreadsheetId, issueKey) {
await ensureSheet(sheets, spreadsheetId, ISSUES_SHEET, ISSUES_HEADER);
// _IssueKey is last column in ISSUES_HEADER
const keyColIndex = ISSUES_HEADER.length; // 1-based
const keyColLetter = colNumberToLetter(keyColIndex);
const res = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range: `${ISSUES_SHEET}!${keyColLetter}:${keyColLetter}` }));
const values = (res && res.data && res.data.values) || [];
for (let i = 0; i < values.length; i++) {
if (String(values[i][0]) === String(issueKey)) return i + 1; // 1-based
}
return null;
}
export async function appendIssue(issue, meta = {}) {
const issueKey = meta.issueKey || `${(issue.repository && issue.repository.full_name) || ''}#${issue.number}`;
return enqueue(String(issueKey), async () => {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, ISSUES_SHEET, ISSUES_HEADER);
const row = mapIssueToRow(issue, meta.action, meta);
const range = `${ISSUES_SHEET}!A:${colNumberToLetter(ISSUES_HEADER.length)}`;
await requestWithRetry(() =>
sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } })
);
logger.info("sheetsAdapter appended issue", { issue: issueKey });
});
}
export async function upsertIssue(issue, meta = {}) {
const issueKey = meta.issueKey || `${(issue.repository && issue.repository.full_name) || ''}#${issue.number}`;
return enqueue(String(issueKey), async () => {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, ISSUES_SHEET, ISSUES_HEADER);
const row = mapIssueToRow(issue, meta.action, meta);
const rowIndex = await findRowIndexByIssueKey(sheets, spreadsheetId, issueKey);
if (rowIndex) {
const range = `${ISSUES_SHEET}!A${rowIndex}:${colNumberToLetter(ISSUES_HEADER.length)}${rowIndex}`;
await requestWithRetry(() =>
sheets.spreadsheets.values.update({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } })
);
logger.info("sheetsAdapter updated issue", { issue: issueKey, row: rowIndex });
} else {
const range = `${ISSUES_SHEET}!A:${colNumberToLetter(ISSUES_HEADER.length)}`;
await requestWithRetry(() =>
sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } })
);
logger.info("sheetsAdapter appended issue", { issue: issueKey });
}
});
}
export default { upsertIssue, appendIssue, appendEvent, appendComment, isEventKnown, getIssueByIssueKey, appendDeadLetter };
export async function appendEvent(evt = {}) {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, EVENTS_SHEET, EVENTS_HEADER);
const processedAt = formatDate(new Date().toISOString());
const eventHash = evt.eventHash || (evt.meta && evt.meta.eventHash) || "";
const eventType = evt.type || (evt.event && evt.event.action) || "";
const issueKey = evt.issueKey || (evt.meta && evt.meta.issueKey) || (evt.event && evt.event.repository && evt.event.issue ? `${evt.event.repository.full_name}#${evt.event.issue.number}` : "");
const actor = (evt.actor || (evt.event && evt.event.sender && evt.event.sender.login) || (evt.meta && evt.meta.actor)) || "";
const summary = evt.summary || (evt.event && evt.event.action) || "";
const status = evt.status || "pending";
// If eventHash exists in Events, update only the Status (and ProcessedAt)
if (eventHash) {
try {
const res = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range: `${EVENTS_SHEET}!B:B` }));
const values = (res && res.data && res.data.values) || [];
for (let i = 0; i < values.length; i++) {
if (String(values[i][0]) === String(eventHash)) {
const rowIndex = i + 1;
const statusCol = EVENTS_HEADER.indexOf('Status') + 1;
const processedAtCol = 1; // ProcessedAt
const statusRange = `${EVENTS_SHEET}!${colNumberToLetter(statusCol)}${rowIndex}`;
const processedAtRange = `${EVENTS_SHEET}!A${rowIndex}`;
await requestWithRetry(() => sheets.spreadsheets.values.update({ spreadsheetId, range: statusRange, valueInputOption: 'RAW', requestBody: { values: [[status]] } }));
await requestWithRetry(() => sheets.spreadsheets.values.update({ spreadsheetId, range: processedAtRange, valueInputOption: 'RAW', requestBody: { values: [[processedAt]] } }));
logger.info('sheetsAdapter updated event status', { issueKey, eventHash, status });
return;
}
}
} catch (e) {
logger.warn('Error checking for existing eventHash, will append', { err: e && e.message ? e.message : e });
}
}
const row = [processedAt, eventHash, eventType, issueKey, actor, summary, status];
const range = `${EVENTS_SHEET}!A:${colNumberToLetter(EVENTS_HEADER.length)}`;
await requestWithRetry(() => sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } }));
logger.info("sheetsAdapter appended event", { issueKey, eventHash });
}
export async function appendComment(comment = {}) {
try {
logger.info('appendComment called', { issueNumber: comment.issueNumber || (comment.comment && comment.comment.issue && comment.comment.issue.number), preview: (comment.body || (comment.comment && comment.comment.body) || '').slice(0, 120) });
} catch (e) {
// ignore
}
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, COMMENTS_SHEET, COMMENTS_HEADER);
const processedAt = formatDate(new Date().toISOString());
// prefer provided issueNumber, else try from comment.issue.number or event.issue.number
const issueNumber = comment.issueNumber || (comment.issue && comment.issue.number) || (comment.comment && comment.comment.issue && comment.comment.issue.number) || (comment.comment && comment.issue && comment.issue.number) || "";
const actor = (comment.actor || (comment.sender && comment.sender.login) || (comment.comment && ((comment.comment.user && comment.comment.user.login) || (comment.comment.sender && comment.comment.sender.login)))) || "";
const url = comment.url || (comment.comment && (comment.comment.html_url || comment.comment.url)) || "";
const row = [processedAt, issueNumber, actor, url];
const range = `${COMMENTS_SHEET}!A:${colNumberToLetter(COMMENTS_HEADER.length)}`;
await requestWithRetry(() => sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } }));
logger.info("sheetsAdapter appended comment", { issueNumber, actor });
}
export async function appendDeadLetter({ eventHash, issueKey, error, payload, notes } = {}) {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, DEADLETTERS_SHEET, DEADLETTERS_HEADER);
const processedAt = formatDate(new Date().toISOString());
const errStr = error && error.message ? error.message : String(error || '');
const payloadB64 = payload ? Buffer.from(typeof payload === 'string' ? payload : JSON.stringify(payload)).toString('base64') : '';
const row = [processedAt, eventHash || '', issueKey || '', errStr, payloadB64, notes || ''];
const range = `${DEADLETTERS_SHEET}!A:${colNumberToLetter(DEADLETTERS_HEADER.length)}`;
await requestWithRetry(() => sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: [row] } }));
logger.info("sheetsAdapter appended dead letter", { issueKey, eventHash });
}
export async function isEventKnown(eventHash) {
if (!eventHash) return false;
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, EVENTS_SHEET, EVENTS_HEADER);
const res = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range: `${EVENTS_SHEET}!B:B` }));
const values = (res && res.data && res.data.values) || [];
for (const row of values) {
if (String(row[0]) === String(eventHash)) return true;
}
return false;
}
export async function appendTimeEntries(issueKey, issueNumber, times = []) {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, TIME_ENTRIES_SHEET, TIME_ENTRIES_HEADER);
const processedAt = formatDate(new Date().toISOString());
// Aggregate seconds per userId (fallback to username if userId missing)
const perUserSeconds = {}; // key -> { userId, userName, seconds }
for (const t of times) {
const userId = (t.user_id || (t.user && (t.user.id || t.user.user_id)) || '') + '';
const userName = t.user_name || (t.user && (t.user.name || t.user.login || t.user.username)) || '';
const seconds = Number(t.time || 0);
const key = userId || userName || '__unknown__';
if (!perUserSeconds[key]) perUserSeconds[key] = { userId: userId || '', userName: userName || '', seconds: 0 };
perUserSeconds[key].seconds += seconds;
}
// Read existing time entries (skip header)
const existingRes = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range: `${TIME_ENTRIES_SHEET}!A2:${colNumberToLetter(TIME_ENTRIES_HEADER.length)}` })).catch(() => ({ data: { values: [] } }));
const existing = (existingRes && existingRes.data && existingRes.data.values) || [];
const toAppend = [];
const perUserMinutes = {};
for (const entryKey of Object.keys(perUserSeconds)) {
const { userId, userName, seconds } = perUserSeconds[entryKey];
const minutes = Math.round(Number(seconds || 0) / 60);
perUserMinutes[userName || userId || entryKey] = minutes;
// Find existing row for same IssueNumber and UserID
let found = false;
for (let i = 0; i < existing.length; i++) {
const row = existing[i] || [];
const rowIssueNumber = row[2] !== undefined ? String(row[2]) : '';
const rowUserId = row[4] !== undefined ? String(row[4]) : '';
if (String(rowIssueNumber) === String(issueNumber) && String(rowUserId) === String(userId)) {
// update TotalMinutes (column index)
const rowIndex = 2 + i; // because we started at A2
const totalMinutesCol = TIME_ENTRIES_HEADER.indexOf('TotalMinutes') + 1;
const range = `${TIME_ENTRIES_SHEET}!${colNumberToLetter(totalMinutesCol)}${rowIndex}`;
await requestWithRetry(() => sheets.spreadsheets.values.update({ spreadsheetId, range, valueInputOption: 'RAW', requestBody: { values: [[minutes]] } }));
found = true;
break;
}
}
if (!found) {
toAppend.push([processedAt, issueKey, issueNumber, userName, userId, minutes]);
}
}
if (toAppend.length > 0) {
const range = `${TIME_ENTRIES_SHEET}!A:${colNumberToLetter(TIME_ENTRIES_HEADER.length)}`;
await requestWithRetry(() => sheets.spreadsheets.values.append({ spreadsheetId, range, valueInputOption: "RAW", requestBody: { values: toAppend } }));
}
logger.info('sheetsAdapter appended/updated time entries', { issueKey, appended: toAppend.length, users: Object.keys(perUserMinutes).length });
return perUserMinutes;
}
export async function updateIssueFields(issueKey, updates = {}) {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, ISSUES_SHEET, ISSUES_HEADER);
const rowIndex = await findRowIndexByIssueKey(sheets, spreadsheetId, issueKey);
if (!rowIndex) return null;
const existing = await getIssueByIssueKey(issueKey);
const newRow = [];
for (let i = 0; i < ISSUES_HEADER.length; i++) {
const key = ISSUES_HEADER[i];
if (updates[key] !== undefined) newRow[i] = updates[key];
else newRow[i] = existing[key] !== undefined ? existing[key] : "";
}
const range = `${ISSUES_SHEET}!A${rowIndex}:${colNumberToLetter(ISSUES_HEADER.length)}${rowIndex}`;
await requestWithRetry(() => sheets.spreadsheets.values.update({ spreadsheetId, range, valueInputOption: 'RAW', requestBody: { values: [newRow] } }));
logger.info('sheetsAdapter updated issue fields', { issueKey });
return newRow;
}
export async function getIssueByIssueKey(issueKey) {
if (!issueKey) return null;
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
await ensureSheet(sheets, spreadsheetId, ISSUES_SHEET, ISSUES_HEADER);
const rowIndex = await findRowIndexByIssueKey(sheets, spreadsheetId, issueKey);
if (!rowIndex) return null;
const range = `${ISSUES_SHEET}!A${rowIndex}:${colNumberToLetter(ISSUES_HEADER.length)}${rowIndex}`;
const res = await requestWithRetry(() => sheets.spreadsheets.values.get({ spreadsheetId, range }));
const row = (res && res.data && res.data.values && res.data.values[0]) || [];
const obj = {};
for (let i = 0; i < ISSUES_HEADER.length; i++) {
obj[ISSUES_HEADER[i]] = row[i] !== undefined ? row[i] : "";
}
return obj;
}
export function pendingOperationsCount() {
return queues.size;
}
export async function waitForIdle(timeoutMs = 10000) {
const start = Date.now();
while (queues.size > 0) {
if (Date.now() - start > timeoutMs) {
throw new Error("timeout waiting for adapter idle");
}
// small sleep
// re-use sleep function defined above
// eslint-disable-next-line no-await-in-loop
await sleep(200);
}
}
export async function readinessProbe() {
try {
const sheets = await sheetsClient();
const spreadsheetId = process.env.GOOGLE_SHEET_ID || config.googleSheetId;
if (!spreadsheetId) throw new Error("GOOGLE_SHEET_ID not configured");
// do a lightweight get for spreadsheet metadata
await requestWithRetry(() => sheets.spreadsheets.get({ spreadsheetId, fields: "spreadsheetId" }), { maxRetries: 1 });
return true;
} catch (err) {
logger.warn("sheetsAdapter readiness probe failed", { err: err && err.message ? err.message : err });
return false;
}
}

8
src/config/index.js Normal file
View File

@ -0,0 +1,8 @@
const config = {
port: process.env.PORT ? Number(process.env.PORT) : 3000,
env: process.env.NODE_ENV || "development",
giteaWebhookSecret: process.env.GITEA_WEBHOOK_SECRET || "",
googleSheetId: process.env.GOOGLE_SHEET_ID || "",
};
export default config;

122
src/controllers/webhookController.js Normal file
View File

@ -0,0 +1,122 @@
import crypto from "crypto";
import { publish } from "../lib/eventBus.js";
import config from "../config/index.js";
import logger from "../lib/logger.js";
function verifyHmac(req) {
const signature = req.headers["x-hub-signature-256"] || req.headers["x-hub-signature"];
const secret = config.giteaWebhookSecret;
if (!signature || !secret || !req.rawBody) return false;
const hmac = crypto.createHmac("sha256", secret);
hmac.update(req.rawBody);
const digest = `sha256=${hmac.digest("hex")}`;
try {
const sigBuf = Buffer.from(signature);
const digBuf = Buffer.from(digest);
if (sigBuf.length !== digBuf.length) return false;
return crypto.timingSafeEqual(sigBuf, digBuf);
} catch (e) {
return false;
}
}
export async function handleWebhook(req, res, next) {
try {
if (!verifyHmac(req)) {
return res.status(401).json({ error: "Unauthorized" });
}
const event = req.body;
// compute event hash for idempotency
const eventHash = crypto.createHash("sha256").update(req.rawBody || JSON.stringify(req.body)).digest("hex");
// extract repo full name
const repoFullName = (event.repository && (event.repository.full_name || (event.repository.owner && event.repository.owner.login && event.repository.name && `${event.repository.owner.login}/${event.repository.name}`))) || "";
// parse meta (estimated days d, importance i) from frontmatter, labels, or title
const meta = { estimatedDays: null, importance: null, estimateSource: null, estimateRaw: null, eventHash, actor: (event.sender && event.sender.login) || null };
const body = event.issue && event.issue.body ? event.issue.body : "";
function parseFrontmatter(text) {
if (!text) return null;
const m = text.match(/^---\s*\n([\s\S]*?)\n---/);
if (!m) return null;
const block = m[1];
const lines = block.split(/\r?\n/);
const obj = {};
for (const line of lines) {
const idx = line.indexOf(":");
if (idx === -1) continue;
const key = line.slice(0, idx).trim();
const val = line.slice(idx + 1).trim();
obj[key] = val;
}
return obj;
}
function parseFromLabels(labels) {
if (!Array.isArray(labels)) return null;
const out = {};
for (const l of labels) {
const name = (typeof l === "string") ? l : (l.name || "");
const md = name.match(/d\s*[:=]\s*([0-9]+(?:[.,][0-9]+)?)/i);
if (md) out.d = md[1];
const mi = name.match(/i\s*[:=]\s*(\d+)/i);
if (mi) out.i = mi[1];
}
return Object.keys(out).length ? out : null;
}
function parseFromTitle(title) {
if (!title) return null;
const out = {};
const md = title.match(/d\s*=\s*([0-9]+(?:[.,][0-9]+)?)/i);
if (md) out.d = md[1];
const mi = title.match(/i\s*=\s*(\d+)/i);
if (mi) out.i = mi[1];
return Object.keys(out).length ? out : null;
}
// precedence: frontmatter > labels > title
const fm = parseFrontmatter(body);
if (fm && (fm.d || fm.i)) {
if (fm.d) {
const d = String(fm.d).replace(/,/g, '.');
meta.estimatedDays = Number(d);
meta.estimateRaw = fm.d;
}
if (fm.i) meta.importance = parseInt(fm.i, 10);
meta.estimateSource = 'frontmatter';
} else {
const lb = parseFromLabels(event.issue && event.issue.labels);
if (lb && (lb.d || lb.i)) {
if (lb.d) meta.estimatedDays = Number(String(lb.d).replace(/,/g, '.'));
if (lb.i) meta.importance = parseInt(lb.i, 10);
meta.estimateSource = 'label';
meta.estimateRaw = lb.d || lb.i || null;
} else {
const tt = parseFromTitle(event.issue && event.issue.title);
if (tt && (tt.d || tt.i)) {
if (tt.d) meta.estimatedDays = Number(String(tt.d).replace(/,/g, '.'));
if (tt.i) meta.importance = parseInt(tt.i, 10);
meta.estimateSource = 'title';
meta.estimateRaw = tt.d || tt.i || null;
}
}
}
const issueKey = repoFullName && event.issue ? `${repoFullName}#${event.issue.number}` : null;
// Publish event to internal bus for background processing.
publish("gitea.event", { event, headers: req.headers, meta: Object.assign(meta, { issueKey }) });
logger.info("Webhook received and published to event bus", { issueKey, eventHash });
return res.status(202).json({ status: "accepted" });
} catch (err) {
return next(err);
}
}

15
src/lib/eventBus.js Normal file
View File

@ -0,0 +1,15 @@
import EventEmitter from "events";
const bus = new EventEmitter();
function publish(topic, payload) {
bus.emit(topic, payload);
}
function subscribe(topic, handler) {
bus.on(topic, handler);
return () => bus.off(topic, handler);
}
export { publish, subscribe };
export default bus;

33
src/lib/logger.js Normal file
View File

@ -0,0 +1,33 @@
const pretty = process.env.LOG_PRETTY === "true" || process.env.NODE_ENV === "development";
function serializeMessage(args) {
if (!args || args.length === 0) return "";
if (args.length === 1) return args[0];
return args;
}
function base(level, args, meta) {
const ts = new Date().toISOString();
const msg = serializeMessage(args);
const payload = Object.assign({ timestamp: ts, level, msg }, meta || {});
if (pretty) {
// eslint-disable-next-line no-console
console.log(JSON.stringify(payload, null, 2));
} else {
// eslint-disable-next-line no-console
console.log(JSON.stringify(payload));
}
}
function createLogger(component) {
return {
info: (...args) => base("info", args, { component }),
warn: (...args) => base("warn", args, { component }),
error: (...args) => base("error", args, { component }),
debug: (...args) => base("debug", args, { component }),
child: (childName) => createLogger(`${component}:${childName}`)
};
}
const root = createLogger(undefined);
export default root;

6
src/middleware/errorHandler.js Normal file
View File

@ -0,0 +1,6 @@
import logger from "../lib/logger.js";
export default function errorHandler(err, req, res, next) {
logger.error(err && err.stack ? err.stack : err);
res.status(500).json({ error: "internal_error", message: err.message || "Internal server error" });
}

11
src/routes/giteaWebhook.js Normal file
View File

@ -0,0 +1,11 @@
import { Router } from "express";
import { handleWebhook } from "../controllers/webhookController.js";
const router = Router();
// The controller handles verification and publishes the event to the internal event bus.
// Business logic (Google Sheets writes) is handled by worker adapters and is intentionally
// separated and not implemented here.
router.post("/", handleWebhook);
export default router;

187
src/server.js Normal file
View File

@ -0,0 +1,187 @@
import dotenv from "dotenv";
dotenv.config();
import express from "express";
import morgan from "morgan";
import fs from "fs";
import path from "path";
import giteaWebhook from "./routes/giteaWebhook.js";
import issueProcessor, { isRunning as workerIsRunning } from "./workers/issueProcessor.js";
import errorHandler from "./middleware/errorHandler.js";
import logger from "./lib/logger.js";
import config from "./config/index.js";
import sheetsAdapter from "./adapters/sheetsAdapter.js";
const app = express();
// use morgan but route logs through structured logger
app.use(morgan("combined", { stream: { write: (s) => logger.info(s.trim()) } }));
// capture raw body for webhook signature verification on the gitea webhook route
app.use(
"/webhooks/gitea",
express.json({ verify: (req, res, buf) => (req.rawBody = buf) })
);
// regular JSON parser for other routes
app.use(express.json());
app.use("/webhooks/gitea", giteaWebhook);
app.get("/health", (req, res) => {
res.json({ status: "ok", uptime: process.uptime(), pid: process.pid, timestamp: new Date().toISOString() });
});
app.get("/ready", async (req, res) => {
const missing = [];
// config checks
if (!config.giteaWebhookSecret) missing.push("GITEA_WEBHOOK_SECRET");
if (!config.googleSheetId) missing.push("GOOGLE_SHEET_ID");
const hasGoogleCreds = !!(
process.env.GOOGLE_SERVICE_ACCOUNT_JSON ||
process.env.GOOGLE_APPLICATION_CREDENTIALS ||
(process.env.GOOGLE_PRIVATE_KEY && process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL) ||
fs.existsSync(path.resolve(process.cwd(), "service-account.json"))
);
if (!hasGoogleCreds) missing.push("Google service account credentials");
if (missing.length) {
logger.warn("Readiness check failed (missing config)", { missing });
return res.status(503).json({ ready: false, missing });
}
// active adapter readiness probe
try {
const ok = await sheetsAdapter.readinessProbe();
if (!ok) {
logger.warn("Readiness probe: sheets adapter not ready");
return res.status(503).json({ ready: false, missing: ["google_sheets_unreachable"] });
}
} catch (err) {
logger.warn("Readiness probe error", { err: err && err.message ? err.message : err });
return res.status(503).json({ ready: false, missing: ["google_sheets_unreachable"] });
}
// worker status
if (!workerIsRunning()) {
logger.warn("Readiness check failed: worker not running");
return res.status(503).json({ ready: false, missing: ["worker_not_running"] });
}
return res.json({ ready: true, uptime: process.uptime(), timestamp: new Date().toISOString() });
});
const PORT = config.port;
function validateConfig() {
const missing = [];
if (!config.giteaWebhookSecret) missing.push("GITEA_WEBHOOK_SECRET");
if (!config.googleSheetId) missing.push("GOOGLE_SHEET_ID");
const hasGoogleCreds = !!(
process.env.GOOGLE_SERVICE_ACCOUNT_JSON ||
process.env.GOOGLE_APPLICATION_CREDENTIALS ||
(process.env.GOOGLE_PRIVATE_KEY && process.env.GOOGLE_SERVICE_ACCOUNT_EMAIL) ||
fs.existsSync(path.resolve(process.cwd(), "service-account.json"))
);
if (!hasGoogleCreds) missing.push("Google service account credentials");
if (missing.length) {
logger.warn("Startup config warning: missing env vars:", { missing });
logger.warn("See SECRETS.md for secure configuration instructions.");
}
}
validateConfig();
// Start background workers (no business logic implemented yet)
issueProcessor.start();
// Central error handler
app.use(errorHandler);
const server = app.listen(PORT, "0.0.0.0", () => {
logger.info("Server running on port", { port: PORT });
});
// track active connections so we can force-close them on shutdown
const connections = new Set();
server.on("connection", (socket) => {
connections.add(socket);
socket.on("close", () => connections.delete(socket));
});
const SHUTDOWN_GRACE_MS = process.env.SHUTDOWN_GRACE_MS ? Number(process.env.SHUTDOWN_GRACE_MS) : 10000;
let shuttingDown = false;
async function shutdown(signal) {
if (shuttingDown) return;
shuttingDown = true;
logger.info("Shutdown initiated", { signal });
// stop accepting new connections
const closeServer = new Promise((resolve) => {
try {
server.close((err) => {
if (err) logger.warn("Error closing server", { err: err && err.message ? err.message : err });
else logger.info("HTTP server closed");
resolve();
});
} catch (e) {
logger.warn("Error during server.close", { err: e && e.message ? e.message : e });
resolve();
}
});
// schedule forced socket destruction after grace period
const forceTimeout = setTimeout(() => {
logger.warn("Forcing destruction of active connections after grace period");
for (const s of connections) {
try {
s.destroy();
} catch (e) {
// ignore
}
}
}, SHUTDOWN_GRACE_MS);
await closeServer;
// stop workers and drain adapter queues
try {
await issueProcessor.stop(SHUTDOWN_GRACE_MS);
} catch (e) {
logger.warn("Error stopping workers", { err: e && e.message ? e.message : e });
}
// final attempt to wait for adapter
try {
await sheetsAdapter.waitForIdle(SHUTDOWN_GRACE_MS);
} catch (e) {
logger.warn("Timeout waiting for adapter to become idle", { err: e && e.message ? e.message : e });
}
clearTimeout(forceTimeout);
logger.info("Shutdown complete, exiting");
// give a moment for logs to flush
setTimeout(() => process.exit(0), 100);
}
process.on("SIGINT", () => shutdown("SIGINT"));
process.on("SIGTERM", () => shutdown("SIGTERM"));
process.on("unhandledRejection", (reason) => {
logger.error("UnhandledRejection", { reason });
shutdown("unhandledRejection");
});
process.on("uncaughtException", (err) => {
logger.error("UncaughtException", { err: err && err.stack ? err.stack : err });
shutdown("uncaughtException");
});

10
src/services/googleSheets.js Normal file
View File

@ -0,0 +1,10 @@
// Deprecated placeholder for the removed Google Sheets business logic.
// The repository has been restructured: business logic should live in
// `src/adapters/*` and `src/workers/*`. This file intentionally does not
// perform any network calls.
export async function handleGiteaEvent(event) {
throw new Error(
"handleGiteaEvent is removed: implement Google Sheets integration in src/adapters/sheetsAdapter.js"
);
}

211
src/workers/issueProcessor.js Normal file
View File

@ -0,0 +1,211 @@
import { subscribe } from "../lib/eventBus.js";
import logger from "../lib/logger.js";
import * as sheetsAdapter from "../adapters/sheetsAdapter.js";
import axios from "axios";
let unsub = null;
let running = false;
async function handler(payload) {
try {
logger.info("Worker received event");
const event = payload && payload.event ? payload.event : payload;
const meta = payload.meta || {};
const headers = payload.headers || {};
const issue = event && event.issue;
if (!issue) {
logger.info("Event has no issue field; skipping processing");
return;
}
const issueKey = meta.issueKey || (event.repository && event.repository.full_name ? `${event.repository.full_name}#${issue.number}` : null);
// dedupe: if eventHash already exists in Events or matches issue.LastEventHash, skip
try {
const seen = await sheetsAdapter.isEventKnown(meta.eventHash);
if (seen) {
logger.info("Duplicate event detected in Events sheet; skipping", { issueKey, eventHash: meta.eventHash });
try {
await sheetsAdapter.appendEvent({ event, eventHash: meta.eventHash, issueKey, actor: meta.actor, summary: event.action || "", status: "duplicate", meta });
} catch (e) {
logger.warn("Failed to append duplicate event record", { err: e && e.message ? e.message : e });
}
return;
}
const existing = await sheetsAdapter.getIssueByIssueKey(issueKey);
if (existing && existing.LastEventHash && String(existing.LastEventHash) === String(meta.eventHash)) {
logger.info("Duplicate event detected by Issue.LastEventHash; skipping", { issueKey, eventHash: meta.eventHash });
try {
await sheetsAdapter.appendEvent({ event, eventHash: meta.eventHash, issueKey, actor: meta.actor, summary: event.action || "", status: "duplicate", meta });
} catch (e) {
logger.warn("Failed to append duplicate event record", { err: e && e.message ? e.message : e });
}
return;
}
} catch (e) {
logger.warn("Error during duplicate check; continuing processing", { err: e && e.message ? e.message : e });
}
// Append an event audit row (processing)
try {
await sheetsAdapter.appendEvent({ event, eventHash: meta.eventHash, issueKey, actor: meta.actor, summary: event.action || "", status: "processing", meta });
} catch (e) {
logger.warn("Failed to append processing event", { err: e && e.message ? e.message : e });
}
// If this is a comment event, append comment row and update issue
const giteaEvent = (headers && (headers['x-gitea-event'] || headers['x-gitea-event'.toLowerCase()])) || null;
const isComment = Boolean(event && event.comment) || giteaEvent === 'issue_comment';
try {
if (isComment) {
// Log comment detection for debugging
try {
logger.info('Detected comment event', {
issueKey,
commentId: event.comment && event.comment.id,
commentUser: event.comment && ((event.comment.user && event.comment.user.login) || (event.comment.sender && event.comment.sender.login)),
preview: event.comment && (event.comment.body ? event.comment.body.slice(0, 160) : ''),
});
} catch (lg) {
// ignore logging errors
}
try {
const issueNumber = (issue && issue.number) || (event && event.issue && event.issue.number) || null;
await sheetsAdapter.appendComment({ issueNumber, comment: event.comment, actor: (event.comment && ((event.comment.user && event.comment.user.login) || (event.comment.sender && event.comment.sender.login))) || meta.actor, body: (event.comment && event.comment.body) || '' });
logger.info('appendComment invoked', { issueNumber, commentId: event.comment && event.comment.id });
} catch (e) {
logger.warn("Failed to append comment", { err: e && e.message ? e.message : e });
}
}
await sheetsAdapter.upsertIssue(issue, Object.assign({}, meta, { action: event.action }));
logger.info("Worker processed issue", issue.number);
// If this was a comment event, ensure CommentsCount is accurate in Issues sheet
try {
if (isComment) {
let newCount = null;
if (issue && typeof issue.comments !== 'undefined' && issue.comments !== null) newCount = issue.comments;
if (newCount !== null) {
await sheetsAdapter.updateIssueFields(issueKey, { CommentsCount: String(newCount) });
} else {
const existing = await sheetsAdapter.getIssueByIssueKey(issueKey);
const existingCount = existing && existing.CommentsCount ? parseInt(existing.CommentsCount, 10) || 0 : 0;
await sheetsAdapter.updateIssueFields(issueKey, { CommentsCount: String(existingCount + 1) });
}
}
} catch (e) {
logger.warn('Failed to update CommentsCount', { err: e && e.message ? e.message : e, issueKey });
}
await sheetsAdapter.appendEvent({ event, eventHash: meta.eventHash, issueKey, actor: meta.actor, summary: event.action || "", status: "success", meta });
// If issue was closed, fetch time entries from Gitea and record per-assignee totals
try {
if (String(event.action).toLowerCase() === "closed") {
const token = process.env.GITEA_APPLICATION_TOKEN;
if (!token) {
logger.warn('GITEA_APPLICATION_TOKEN not configured; skipping time aggregation');
} else {
// derive owner/repo
const fullName = (event.repository && (event.repository.full_name || (event.repository.owner && `${event.repository.owner.login}/${event.repository.name}`))) || null;
if (!fullName) {
logger.warn('Cannot determine repository full_name for time aggregation', { event });
} else {
const parts = fullName.split('/');
const owner = parts[0];
const repo = parts.slice(1).join('/');
// derive API base
let apiBase = process.env.GITEA_API_BASE;
if (!apiBase) {
try {
if (event.repository && event.repository.html_url) {
const u = new URL(event.repository.html_url);
apiBase = `${u.protocol}//${u.host}`;
}
} catch (e) {
// ignore
}
}
if (!apiBase) {
logger.warn('Cannot determine Gitea API base URL; set GITEA_API_BASE to override');
} else {
const url = `${apiBase.replace(/\/$/, '')}/api/v1/repos/${encodeURIComponent(owner)}/${encodeURIComponent(repo)}/issues/${encodeURIComponent(issue.number)}/times`;
try {
const resp = await axios.get(url, { headers: { Authorization: `Bearer ${token}` }, timeout: 10000 });
const times = (resp && resp.data) || [];
const perUser = await sheetsAdapter.appendTimeEntries(issueKey, issue.number, times);
// compute total minutes and human summary
let total = 0;
let summary = '';
if (perUser && Object.keys(perUser).length > 0) {
total = Object.values(perUser).reduce((s, v) => s + Number(v || 0), 0);
summary = Object.entries(perUser).map(([u, m]) => `${u}:${m}min`).join('; ');
}
// update Issues row with totals and who closed
const closedBy = meta.actor || (event && event.sender && event.sender.login) || '';
await sheetsAdapter.updateIssueFields(issueKey, { TotalTimeMinutes: total, TimeSummary: summary, ClosedBy: closedBy });
logger.info('Recorded time entries for closed issue', { issueKey, total, summary });
} catch (e) {
logger.warn('Failed fetching time entries from Gitea', { err: e && e.message ? e.message : e, url });
}
}
}
}
}
} catch (e) {
logger.warn('Time aggregation failed', { err: e && e.message ? e.message : e });
}
} catch (err) {
logger.error("Worker error processing issue", { err: err && err.message ? err.message : err, issue: issueKey });
try {
await sheetsAdapter.appendEvent({ event, eventHash: meta.eventHash, issueKey, actor: meta.actor, summary: event.action || "", status: "failed", meta });
} catch (e2) {
logger.warn("Failed to append failure event", { err: e2 && e2.message ? e2.message : e2 });
}
// Also write to DeadLetters for manual inspection
try {
await sheetsAdapter.appendDeadLetter({ eventHash: meta.eventHash, issueKey, error: err, payload: event, notes: 'worker-processing-failed' });
} catch (e3) {
logger.warn("Failed to append dead letter", { err: e3 && e3.message ? e3.message : e3 });
}
}
} catch (err) {
logger.error("Worker error:", err && err.message ? err.message : err);
}
}
function start() {
if (running) return;
unsub = subscribe("gitea.event", handler);
running = true;
}
async function stop(timeoutMs = 10000) {
if (unsub) {
try {
unsub();
} catch (e) {
logger.warn("Error unsubscribing worker", { err: e && e.message ? e.message : e });
}
unsub = null;
}
running = false;
try {
await sheetsAdapter.waitForIdle(timeoutMs);
logger.info("Worker: drained adapter queues");
} catch (err) {
logger.warn("Worker: timeout waiting for adapter to drain", { err: err && err.message ? err.message : err });
}
}
export default { start, stop };
export function isRunning() {
return running;
}