Skip to content

Commit f589660

Browse files
feat(routines): add safe webhook setup and in-routine run management (#13637)
## Thinking Path > - Paperclip is the open source app people use to manage AI agents for work. > - Routines turn scheduled work and external events into tasks for an assigned agent. > - Webhook setup was disabled, and actor authentication rejected valid webhook bearer keys. > - Operators need to connect and test a sending app before events can start work. > - This pull request adds a guided setup with durable connection tests that cannot dispatch a task. > - It keeps trigger management, execution tasks, and activity within the routine. > - The benefit is a webhook that can be configured, verified, and operated from one place. ## Linked Issues or Issue Description Fixes #11937. Related: #13216 adds provider-specific Sentry support. This PR addresses general routine setup and ingress. #6841 addresses legacy secret bindings; this PR retains the existing secret service. **Current behavior** Webhook creation is disabled. Bearer deliveries can fail in agent authentication before the routine checks its key. Setup has no safe connection test. Runs and Activity send the operator away from the routine. **Proposed behavior** Choose a schedule or a webhook. Follow the setup steps, copy credentials or complete agent instructions, and test delivery without creating work. Finish setup to allow future events to start tasks. Edit or remove compact trigger cards, undo removal, and inspect tasks and activity inside the routine. **Reason and benefit** An operator can verify credentials and delivery before enabling automatic work. Durable setup state survives refreshes and restarts. Retry receipts prevent an old test event from starting work after activation. ## What Changed - Add a production trigger wizard using reusable Slack setup navigation and footer components. - Add schedule and webhook choices, one-time credentials, agent instructions, and live connection feedback. - Persist pending setup, test delivery receipts, connection status, and reversible trigger removal. - Keep setup checks free of routine runs, tasks, and agent wakeups. Preserve delivery idempotency after activation. - Add compact trigger cards, inline editing, key rotation, pause controls, removal, and Undo. - Keep Runs and Activity in the routine. Use the shared task list and compact activity rows. - Permit only exact public delivery POSTs through actor authentication. Retain webhook authentication, JSON-object validation, and log redaction. - Add production-backed Storybook states and focused server, database, and UI coverage. - Document signing modes, setup checks, retries, rotation, HTTPS ingress, and navigation. ## Verification - Full workspace typecheck, build, and token gates passed on the rebased branch. Storybook also builds. - Focused routine, middleware, logging, shared wizard, and UI coverage passes on the rebased branch: 195 tests across 14 files. The migration passed on a fresh PostgreSQL database and on two repeated applications. - Browser testing used the real app, database, and a deterministic process worker through Tailscale HTTPS and the current Cloud proxy code. - Verified rejected keys, safe setup deliveries, persisted state after restart, activation, retry deduplication, key rotation, schedule editing, removal, and Undo. - Fresh bearer and GitHub-signed deliveries created tasks that the worker checked out and completed. Runs and Activity stayed within the routine. - Current Cloud ingress tests passed. Public delivery POSTs passed through without a browser session; management routes remained gated. - All 54 current-head PR checks pass, including general and serialized tests, all eight browser E2E shards, typecheck, build, runner checks, security checks, and the canary dry run. Two optional Storybook jobs are skipped by workflow conditions. - Greptile is 5/5 on commit `7ea63a61e`, with no unresolved review threads. The stale connection-status finding is fixed and covered by a regression test. - No production deployment was performed. ## Risks - Migration 0281 adds three trigger columns and a test-receipt table. It is additive and safe to reapply. Apply it before running the new server. Existing triggers remain live by default. - Requests without delivery IDs are new events after activation. Senders must reuse an event's delivery ID for retries. - Completed webhooks keep normal dispatch behavior. Their management connection check can start work; the UI states this. - Removing a trigger archives it. Undo restores the URL and credentials. Permanent deletion remains available through the existing API. - Public ingress must remain restricted to the delivery POST route. The tenant verifies credentials. Cloud sleeping-stack behavior is unchanged. - Shared setup components also serve Slack. Existing setup contracts and navigation tests cover that integration. - Senders must use application/json with an object. Other media types receive 415. ## Model Used OpenAI Codex, based on GPT-6, with reasoning, repository tools, shell execution, and browser testing. The exact deployment model ID and context-window size are not exposed in this session. ## Checklist - [x] I have included a thinking path that traces from project context to this change - [x] I have specified the model used (with version and capability details) - [x] I have checked ROADMAP.md and confirmed this PR does not duplicate planned core work - [x] I have searched GitHub for duplicate or related PRs and linked them above - [x] I have either (a) linked existing issues with `Fixes: #` / `Closes #` / `Refs #` OR (b) described the issue in-PR following the relevant issue template - [x] I have not referenced internal/instance-local Paperclip issues or links (only public GitHub `#NNN` / `github.com/paperclipai/paperclip` URLs) - [x] My branch name describes the change (e.g. `docs/...`, `fix/...`) and contains no internal Paperclip ticket id or instance-derived details - [x] I have run tests locally and they pass - [x] I have added or updated tests where applicable - [x] I have updated relevant documentation to reflect my changes - [x] I have considered and documented any risks above - [x] All Paperclip CI gates are green - [x] Greptile is 5/5 with no open P2s, recommendations, or follow-ups - [x] I will address all Greptile and reviewer comments before requesting merge --------- Co-authored-by: Paperclip <noreply@paperclip.ing>
1 parent 20d2611 commit f589660

57 files changed

Lines changed: 51027 additions & 835 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

doc/design/COMPONENT-INVENTORY.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -424,3 +424,7 @@ Independently addressable examples live under `Connections/In-task connections`
424424
- `AnnouncementCard`: image, eyebrow, headline, description, navigation links and dismissal; accepts an announcement and `onDismiss`.
425425
- `AnnouncementWell`: one app-shell placement that owns eligibility, dismissal sync, modal deferral and toast priority. Use only once in Layout.
426426
- Preview variants live in `/design-guide` and Storybook under `Announcements/AnnouncementCard`.
427+
428+
## Shared setup wizard (2026-09-19)
429+
430+
`ui/src/components/SetupWizard.tsx` extracts the Slack setup navigation into reusable numbered steps, a portal sidebar, an optional section-menu takeover outlet, and a single-row footer. `SetupWizardSidebarProvider` owns the portal target and takeover lifecycle. Chat exports retain their existing names and defaults for compatibility. The Design Guide demonstrates the components. The production routine trigger wizard and its Storybook previews share the sidebar takeover, navigation, and footer. `routine-triggers/TriggerWizard.tsx`, `TriggerCard.tsx`, and `WebhookFields.tsx` provide the shared trigger setup, compact editable cards, copyable credentials, and agent instructions.

docs/api/routines.md

Lines changed: 84 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -126,7 +126,11 @@ Three trigger kinds:
126126
}
127127
```
128128

129-
Signing modes: `bearer` (default), `hmac_sha256`. Replay window range: 30–86400 seconds (default 300).
129+
Signing modes: `bearer` (default), `hmac_sha256`, `github_hmac`, and `none`.
130+
The replay window applies to `hmac_sha256` only: 30–86400 seconds (default 300).
131+
Creating a webhook returns `trigger` and one-time `secretMaterial` containing
132+
`webhookUrl` and `webhookSecret`. Save the secret before closing the dialog.
133+
Routine details retain the URL; rotate the secret if its value is lost.
130134

131135
**API** — fires only when called explicitly via [Manual Run](#manual-run):
132136

@@ -184,7 +188,68 @@ Fires a run immediately, bypassing the schedule. Concurrency policy still applie
184188
POST /api/routine-triggers/public/{publicId}/fire
185189
```
186190

187-
Fires a webhook trigger from an external system. Requires a valid `Authorization` or `X-Paperclip-Signature` + `X-Paperclip-Timestamp` header pair matching the trigger's signing mode.
191+
Fires a webhook trigger from an external system without a Paperclip login. Send
192+
`Content-Type: application/json` and a JSON object. The trigger authenticates the
193+
request using its own secret; an agent or board API key is not a substitute.
194+
195+
| Mode | Headers and signature |
196+
|------|-----------------------|
197+
| `bearer` | `Authorization: Bearer <webhookSecret>` |
198+
| `hmac_sha256` | `X-Paperclip-Timestamp` (Unix seconds or milliseconds) and `X-Paperclip-Signature: sha256=<hex>`; HMAC-SHA256 over the timestamp string, a dot, and the exact body bytes |
199+
| `github_hmac` | `X-Hub-Signature-256: sha256=<hex>`; HMAC-SHA256 over the exact body bytes, without a timestamp. `X-Paperclip-Signature` is also accepted. Configure GitHub to send JSON. |
200+
| `none` | No signature. Anyone with the generated URL can fire the trigger; keep it private. |
201+
202+
For a bearer trigger:
203+
204+
```sh
205+
curl --fail-with-body "$WEBHOOK_URL" \
206+
-H "Authorization: Bearer $WEBHOOK_SECRET" \
207+
-H 'Content-Type: application/json' \
208+
-H 'Idempotency-Key: deployment-123' \
209+
--data-binary '{"event":"deployment","variables":{"environment":"staging"}}'
210+
```
211+
212+
For a timestamped HMAC trigger, sign and send the same bytes:
213+
214+
```js
215+
import { createHmac } from "node:crypto";
216+
const body = JSON.stringify({ variables: { environment: "staging" } });
217+
const timestamp = String(Math.floor(Date.now() / 1000));
218+
const signature = createHmac("sha256", process.env.WEBHOOK_SECRET)
219+
.update(`${timestamp}.`).update(body).digest("hex");
220+
const response = await fetch(process.env.WEBHOOK_URL, {
221+
method: "POST",
222+
headers: {
223+
"Content-Type": "application/json",
224+
"X-Paperclip-Timestamp": timestamp,
225+
"X-Paperclip-Signature": `sha256=${signature}`,
226+
},
227+
body,
228+
});
229+
console.log(response.status, await response.json());
230+
```
231+
232+
`202` returns the routine run, including its status and linked task. The task
233+
runs asynchronously, so acceptance does not mean the agent has finished.
234+
Concurrency policy can coalesce or skip a delivery while work is active.
235+
Payload fields or a nested `variables` object supply declared routine variables;
236+
nested values take precedence. The full payload is retained in run history.
237+
238+
For bearer, GitHub HMAC, and unsigned triggers, send a stable `Idempotency-Key`
239+
when retrying a delivery to receive the original run without creating another
240+
task. Timestamped HMAC rejects an identical signed delivery with `409`, even
241+
inside the replay window; stale timestamps and invalid secrets/signatures return
242+
`401`. Disabled triggers and paused/archived routines return `409`. Missing
243+
required variables return `422`; non-JSON media types return `415` and invalid
244+
JSON objects return `400`. Rotating a secret immediately invalidates the old one.
245+
246+
Cloud installations use their canonical public origin for generated URLs.
247+
Self-hosted installations should set `PAPERCLIP_PUBLIC_URL` to their HTTPS
248+
origin. The reverse proxy must forward this POST endpoint and its authorization,
249+
signature, timestamp, and idempotency headers without requiring a browser login.
250+
For local HTTPS testing, proxy an isolated test instance through Tailscale Serve;
251+
use Funnel only if the sender is outside the tailnet. Existing routine and agent
252+
execution controls still apply, including the isolated-worktree execution gate.
188253

189254
## List Runs
190255

@@ -216,3 +281,20 @@ active -> paused -> active
216281
```
217282

218283
Archived routines do not fire and cannot be reactivated.
284+
285+
## Routine detail navigation
286+
287+
The routine detail page keeps **Runs** and **Activity** in the routine sidebar. Runs lists the execution issues for that routine using the shared issue list, including issue status, priority, assignee, and search controls. Activity shows the routine, trigger, and run event timeline without leaving the routine page. The overview links to these same local tabs.
288+
289+
290+
## Webhook setup and connection checks
291+
292+
Create a webhook trigger with `setupPending: true` to configure it safely. While setup is pending, authenticated deliveries return `202` with `{ "status": "test_received", "test": true, "routineStarted": false, "linkedIssueId": null }`. They never create a routine run, task, or agent wakeup. This state survives refreshes and server restarts, and connection checks also work while the routine is paused. Invalid authentication still returns `401`.
293+
294+
Routine detail exposes `setupPending` and `lastWebhookDelivery` (`status`, `receivedAt`, and `test`) so the wizard can show live connection feedback. The secret is only returned at creation or rotation; it is never stored in browser draft storage or included in routine detail.
295+
296+
Finish setup with `PATCH /api/routine-triggers/{id}` and `{ "setupPending": false }`. Future deliveries use normal routine dispatch and still respect the routine's enabled state. Test events are not dispatched on activation. Retries with the same `Idempotency-Key`, GitHub `X-GitHub-Delivery`, or timestamp-HMAC replay key remain test receipts after activation. Send a unique delivery ID per event so a sender's retries can be recognized. Requests without a delivery ID are new events after activation.
297+
298+
For compatibility, API-created triggers without `setupPending: true` are immediately live. Completed triggers cannot be returned to setup mode. Checking a previously enabled webhook observes real deliveries and can start the routine; the management UI explains this difference.
299+
300+
Trigger cards support removal with Undo. `PATCH` with `{ "archived": true }` excludes a trigger from routine detail and scheduling, and rejects its webhook deliveries. Setting `archived` back to `false` restores the same URL and credentials. `DELETE` remains the permanent deletion API.
Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
CREATE TABLE IF NOT EXISTS "routine_webhook_test_receipts" (
2+
"id" uuid PRIMARY KEY DEFAULT gen_random_uuid() NOT NULL,
3+
"company_id" uuid NOT NULL,
4+
"trigger_id" uuid NOT NULL,
5+
"delivery_key_hash" text NOT NULL,
6+
"received_at" timestamp with time zone DEFAULT now() NOT NULL
7+
);
8+
--> statement-breakpoint
9+
ALTER TABLE "routine_triggers" ADD COLUMN IF NOT EXISTS "setup_pending" boolean DEFAULT false NOT NULL;--> statement-breakpoint
10+
ALTER TABLE "routine_triggers" ADD COLUMN IF NOT EXISTS "archived" boolean DEFAULT false NOT NULL;--> statement-breakpoint
11+
ALTER TABLE "routine_triggers" ADD COLUMN IF NOT EXISTS "last_webhook_delivery" jsonb;--> statement-breakpoint
12+
DO $$ BEGIN ALTER TABLE "routine_webhook_test_receipts" ADD CONSTRAINT "routine_webhook_test_receipts_company_id_companies_id_fk" FOREIGN KEY ("company_id") REFERENCES "public"."companies"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
13+
DO $$ BEGIN ALTER TABLE "routine_webhook_test_receipts" ADD CONSTRAINT "routine_webhook_test_receipts_trigger_id_routine_triggers_id_fk" FOREIGN KEY ("trigger_id") REFERENCES "public"."routine_triggers"("id") ON DELETE cascade ON UPDATE no action; EXCEPTION WHEN duplicate_object THEN NULL; END $$;--> statement-breakpoint
14+
CREATE UNIQUE INDEX IF NOT EXISTS "routine_webhook_test_receipts_delivery_uq" ON "routine_webhook_test_receipts" USING btree ("trigger_id","delivery_key_hash");

0 commit comments

Comments
 (0)