PUT and GET APIs for projects, credentials and collections, with policies that decide from the caller's scope

.claude/rules/comments.md

@@ -0,0 +1,109 @@
+---
+paths:
+  - "lib/**/*.ex"
+  - "test/**/*.exs"
+  - "assets/js/**/*.ts"
+  - "assets/js/**/*.tsx"
+---
+
+# Comments and docs
+
+A comment promises the reader something the code can't tell them. Every one
+that breaks the promise teaches people to skip the next, and agents read every
+comment and act on it, so a wrong one misleads twice. Fewer, true, plain.
+
+## When a comment earns its place
+
+Keep a comment only if it carries one of these:
+
+- a constraint the names don't show,
+- a decision a competent reader would otherwise reverse ("we deliberately
+  don't retry here"),
+- a workaround for something outside the codebase,
+- a pointer to where a rule comes from.
+
+Anything else goes. If a comment exists because the code is unclear, rename or
+extract instead.
+
+```elixir
+# Ecto's :utc_datetime type rejects non-zero microseconds, so truncate
+# before casting or the changeset fails.
+scheduled_at = DateTime.truncate(DateTime.utc_now(), :second)
+```
+
+## Which kind of comment
+
+- `@moduledoc`: what the module is for and when to reach for it. One or two
+  sentences is usually enough. Use `@moduledoc false` rather than a filler
+  sentence.
+- `@doc`: how to call the function and what happens. Read in IEx, editor
+  hovers and by agents, usually without the source. The first line is one
+  sentence that stands alone.
+- `#`: why this line is like this, for someone already in the body.
+
+In TypeScript, JSDoc on an export plays the part of `@doc`.
+
+## A `@doc` describes what the caller gets, not the steps inside
+
+Say what the inputs must be, every return shape including error tuples, and
+side effects the caller can't see (writes, broadcasts, audit rows). Don't walk
+through the body in order: that goes wrong the moment someone reorders it.
+
+```elixir
+@doc """
+Replaces a credential's body on behalf of `actor`.
+
+Returns `{:error, :unauthorized}` if the actor can't write credentials in the
+project, or `{:error, changeset}` if the body is invalid. The audit event is
+written in the same transaction.
+"""
+```
+
+## Explain a rule once, where it's enforced
+
+When several modules rely on the same rule, the explanation lives in the
+module that enforces it. The others say nothing, or one line pointing there if
+the caller would otherwise look wrong. Don't describe another module's
+behaviour in a comment; it goes wrong when that module changes and nothing
+flags it.
+
+## Write what will still be true next month
+
+| Goes stale | Lasts |
+|---|---|
+| Step-by-step account of the body | What the function promises |
+| Callers, routes ("used by the API controller") | Constraints on inputs |
+| Counts, line numbers, file paths | Why a rule exists |
+| "Currently", "for now", "new", "now also" | Error shapes and side effects |
+| Ticket, PR and refactor history | Link to the external source of a rule |
+
+Never narrate the change ("added for CON-331", "now also handles service
+accounts"). That is the commit message's job. Code reads as if it was always
+written that way.
+
+When you change behaviour, re-read the comments in that function and its
+`@doc`, and fix anything the change made false in the same commit.
+
+A TODO says what is left and why it isn't done now. "TODO: fix" is not a TODO.
+
+## Plain words
+
+- Full sentences a colleague would say out loud. If a reader would need a
+  follow-up question to decode it, rewrite it.
+- No coined shorthand: "the state rule", "project-less scope", "raw-actor
+  clause". Say the thing it stands for.
+- No filler: "simply", "note that", "it's worth noting", "this ensures that".
+- Don't weld clauses together with colons and semicolons. Two sentences, or
+  one simpler one.
+- No spaced em dashes; prefer a full stop.
+- `mix format` doesn't wrap comments. Reflow a paragraph after editing it so
+  every line but the last runs to 80 columns. In a `@doc` or `@moduledoc`, a
+  blank line separates ideas, and three or more parallel items go in a list.
+
+## Tests
+
+- Put the intent in the test name, not a comment above it.
+- Comment setup only when the reason isn't visible ("inserted out of order so
+  the query has to sort").
+- Don't restate an assertion. Do say why when it pins deliberate behaviour a
+  reader might "fix" (a refusal that looks like a bug).

CHANGELOG.md

@@ -23,6 +23,10 @@ and this project adheres to
   [#4718](https://github.com/OpenFn/lightning/issues/4718)
 - The workflow version hash no longer includes a trigger's `enabled` state
   [#5241](https://github.com/OpenFn/lightning/pull/5241)
+- Service-account token requests must now name the scopes they want. A request
+  to `POST /api/oauth/token` without `scope`, or with an empty one, is refused
+  with `invalid_scope` instead of getting every scope.
+  [#5239](https://github.com/OpenFn/lightning/pull/5239)
 
 ### Fixed
 

SERVICE_ACCOUNTS.md

@@ -5,8 +5,8 @@ signing in with a key pair instead of a password. Use one when infrastructure
 automation, rather than a person, creates the instance's users, including its
 first superuser.
 
-Each instance has at most one service account. For now it can only create,
-change and find users.
+Each instance has at most one service account. It can create, change and find
+users, and create, change and read projects, collections and credentials.
 
 > #### Still under development {: .warning}
 >
@@ -92,6 +92,7 @@ const { access_token } = await fetch(token_endpoint, {
     client_assertion_type:
       'urn:ietf:params:oauth:client-assertion-type:jwt-bearer',
     client_assertion: `${unsigned}.${signature}`,
+    scope: 'users:read users:write',
   }),
 }).then(response => response.json());
 ```
@@ -116,11 +117,28 @@ Lightning refuses a JSON body.
 If the token endpoint answers `invalid_client`, Lightning's logs say why. See
 [Logs](#logs).
 
+## What the token can do
+
+Send the access token as `Authorization: Bearer <token>`. What it may do depends
+on the scopes it carries:
+
+- `users:read` finds users, and `users:write` creates and changes them.
+- `projects:write` reads, creates and changes projects.
+- `collections:write` reads, creates and renames collections.
+- `credentials:write` reads, creates and replaces credentials.
+
+Ask for the scopes you need with the `scope` parameter when you request the
+token. Projects, collections and credentials have no read scopes: their `GET`
+routes need the same `:write` scope as their `PUT` routes.
+
+The service account acts on every project, as if it held every project role,
+owner included, but only as far as its token's scopes reach. It is never a
+member of a project and never shows in a project's members.
+
 ## Manage users
 
-Send the access token as `Authorization: Bearer <token>`. Creating and changing
-users needs the `users:write` scope, and finding them needs `users:read`. A
-token gets both unless you ask for fewer with a `scope` parameter.
+Creating and changing users needs the `users:write` scope, and finding them
+needs `users:read`.
 
 Create a user, here a superuser whose email is already verified. `role` is
 `user` or `superuser`, and defaults to `user`:
@@ -188,6 +206,112 @@ curl -X PATCH https://lightning.example.org/api/users/$USER_ID \
 A new role or password signs the user out of every browser session. The email
 can't be changed, and the API can't delete or disable a user.
 
+## Manage projects
+
+`GET /api/projects/:id` shows a project. `PUT /api/projects/:id` creates the
+project under that id, or, if a project already has it, replaces its name and
+description. Both need `projects:write`.
+
+To create a project, send its `name` and `description`, its `members` as
+`{email, role}` pairs, and `notify`. Each member must already be a user.
+`notify` is `true` by default, and decides whether Lightning emails the members
+it adds. `members` and `notify` only count when the project is created:
+
+```bash
+curl -X PUT https://lightning.example.org/api/projects/$PROJECT_ID \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "malaria-programme",
+    "description": "Malaria case reporting",
+    "members": [
+      {"email": "ada@example.com", "role": "owner"},
+      {"email": "grace@example.com", "role": "editor"}
+    ],
+    "notify": false
+  }'
+```
+
+Lightning answers 201 with the project and its members when it creates one, and
+200 when it replaces one.
+
+## Manage collections
+
+`GET` and `PUT /api/projects/:project_id/collections/:id` read and write a
+project's collection. They need `collections:write`. `PUT` creates the
+collection under that id, or renames the one that has it and keeps its items:
+
+```bash
+curl -X PUT https://lightning.example.org/api/projects/$PROJECT_ID/collections/$COLLECTION_ID \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{"name": "patients"}'
+```
+
+Lightning answers `{"data": {"id", "project_id", "name"}}`, with 201 for a new
+collection and 200 for a rename. The items inside a collection are not part of
+this API.
+
+## Manage credentials
+
+`GET` and `PUT /api/credentials/:id` read and write a credential. They need
+`credentials:write`. `PUT` creates the credential under that id, or replaces the
+name, schema and `main` body of the one that has it.
+
+A `PUT` must name the credential's `owner` by email, and the owner must be an
+existing user:
+
+```bash
+curl -X PUT https://lightning.example.org/api/credentials/$CREDENTIAL_ID \
+  -H "Authorization: Bearer $ACCESS_TOKEN" \
+  -H "Content-Type: application/json" \
+  -d '{
+    "name": "Acme DHIS2",
+    "schema": "raw",
+    "owner": "ada@example.com",
+    "credential_bodies": [
+      {"name": "main", "body": {"username": "admin", "password": "district"}}
+    ],
+    "project_credentials": [{"project_id": "'"$PROJECT_ID"'"}]
+  }'
+```
+
+- `credential_bodies` must be exactly one body, named `main`, holding an object.
+  Replacing a credential leaves any other body it has alone.
+- `project_credentials` is optional. A `PUT` only adds links, and never removes
+  one the credential already has. Linking a project needs `credentials:write`
+  only. Naming a project that doesn't exist answers 403.
+- The response never includes the credential's body. It lists every project the
+  credential is linked to, but names a linked project only if the token also
+  holds `projects:write`.
+- A service account can write credentials but never delete one. Listing,
+  creating with `POST` and deleting are for people: a service account's token
+  gets a 401 from those routes.
+
+## When Lightning refuses
+
+- 401 with `WWW-Authenticate: Bearer error="invalid_token"`: the access token is
+  expired or bad, or missing on `/api/users`. Elsewhere, a request with no token
+  gets a plain 401.
+- 403 `{"error": "insufficient_scope"}` with
+  `WWW-Authenticate: Bearer error="insufficient_scope"`: the token lacks the
+  scope the action needs. Lightning checks this first, so it comes before any
+  404 or 422, whether the project is missing or the body is wrong.
+- 404: the project, collection or credential doesn't exist.
+- 409 `{"error": "scheduled_for_deletion"}`: the project (on the projects and
+  collections routes) or the credential (on a credential `PUT`) is scheduled for
+  deletion. Lightning refuses changes until the deletion is cancelled or
+  completes.
+- 409 `{"error": "id_taken"}`: the id already belongs to a collection in another
+  project, or to a credential with a different owner. Lightning never moves
+  either.
+- 409 `{"error": "name_taken"}`: the project already has a collection with that
+  name, or the owner already has a credential with that name.
+- 422 `{"errors": {...}}`: the body doesn't fit. Each key names a field, or a
+  key Lightning doesn't accept, and lists what is wrong without quoting values.
+- 402 `{"error": "exceeds_limit"}`: creating a collection would go over the
+  instance's limit.
+
 ## Change or remove the key
 
 Generate a new key pair, set `SERVICE_ACCOUNT_PUBLIC_KEY` to the new public key,
@@ -214,6 +338,12 @@ there as "Service account", named by its key's thumbprint. It records:
 - `updated` for each user it changes, with the before and after of the email,
   names, role and confirmation. A new password shows only as `password_changed`.
   A change that changes nothing isn't recorded.
+- `created` for each project it creates, with `collaborator_added` for each
+  member it adds, and `name_updated` or `description_updated` when it changes
+  one.
+- `created` and `updated` for each collection it creates or renames.
+- `created` and `updated` for each credential it writes, and `added_to_project`
+  for each project it links.
 
 ### Logs
 

lib/lightning/actor.ex

@@ -0,0 +1,8 @@
+defmodule Lightning.Actor do
+  @moduledoc """
+  Whoever a context function records as having done something: a person or a
+  service account.
+  """
+
+  @type t :: Lightning.Accounts.User.t() | Lightning.ServiceAccount.t()
+end