Design: context accumulation governance for delegated agent workflows #2798
Replies: 2 comments 3 replies
|
Thanks for writing this up. The envelope model seems like the right abstraction for the in-process case, especially the separation between aggregation, derived artifacts, delegation restrictions, and an explicit declassification step. One narrow point I would suggest keeping visible in the design is the boundary between the in-process envelope and anything a downstream verifier or cross-agent runtime might consume later. If an envelope ID appears in agt-evidence receipts, it may be useful to make the receipt treat that ID as an opaque reference to external governance context, not as a portable schema for the full envelope contents. That would leave AGT free to define the in-process ContextEnvelope shape while still preserving a small verifier-facing join point for diverse evidence implementations. Then the follow-up for trust-boundary cases can focus on responsibilities like issuer, version/monotonicity, replay/downgrade rejection, and where verification happens, without forcing AGT to standardize every downstream evidence record format. |
|
I like this boundary and I think it's worth keeping visible. Holding the envelope ID opaque in receipts, rather than serializing the envelope shape, seems like it would let AGT evolve the in-process ContextEnvelope while the receipt commits to only a small join point. A couple of things I'd want to think through before settling it. One is lifetime: a receipt can outlive the context it points at, so it isn't obvious whether a reference should be allowed to dangle or the issuer guarantees resolvability for some window. The other is whether anything minimal should travel with the ID, since a pure opaque token isn't much use to a verifier that can't perform the governed lookup. There might be a middle option, a coarse non-sensitive attestation like a sensitivity tier alongside the reference, but I'm not sure where the line should sit. If that holds up, the trust-boundary follow-up reads more like a responsibilities contract than a schema: issuer, version and monotonicity, replay and downgrade rejection, and where verification happens. Does that match how you were thinking about it? |
|
Yes, the responsibilities-contract framing matches how I was thinking about it. For lifetime, I would keep the receipt from becoming the guarantor. The issuer of the opaque reference should define the retention or resolvability window, and consumers can treat an expired or unresolved reference as a verification outcome rather than a receipt-schema failure. For minimal metadata, I agree there may be a useful middle ground, but I would keep it coarse and non-sensitive: something like a routing hint or governance label that helps a verifier choose the right resolver or policy path, without serializing the ContextEnvelope or committing AGT to a downstream evidence schema. That keeps the receipt stable as a small join point, lets the in-process ContextEnvelope evolve independently, and makes the trust-boundary follow-up about issuer/version/monotonicity/replay/downgrade responsibilities rather than about standardizing the full governed record. |
|
That all lands for me, and I think it settles the shape. Keeping the receipt out of the guarantor role is the part I most agree with: the issuer of the opaque reference owns the retention and resolvability window, and a consumer that hits an expired or unresolved reference treats that as a verification outcome, not a malformed receipt. That keeps the receipt stable even as the in-process ContextEnvelope changes underneath it. On the minimal metadata, the coarse routing-hint framing is the right ceiling: something a verifier can use to pick the resolver or policy path, a governance label or sensitivity tier, with no part of the envelope contents serialized. The moment a receipt carries the envelope shape it has effectively standardized a downstream record format, which is exactly what we wanted to avoid. I've folded this into the PR. There's now an |
|
The aggregation problem is really interesting here. A runtime policy can correctly approve every individual action and still miss the security consequence of the sequence as a whole. That makes “is this action allowed?” only one part of the enforcement problem. I think the runtime needs to maintain enough state to evaluate not only the current action, but also what the agent has already accessed, what it is trying to combine, and where the resulting data is going. This is close to the direction we're exploring with Aegisora: the enforcement point sits immediately before execution, but the decision can incorporate runtime context rather than treating every tool call as an isolated event. Would be interested to see whether context accumulation eventually becomes part of the policy input itself rather than a separate monitoring layer. |
Uh oh!
There was an error while loading. Please reload this page.
This is a design writeup for the feature proposed in #2797. It covers the model, the invariants, a threat model, and the open questions. Feedback welcome, especially on ownership and on where the envelope schema should live.
Background
AGT already governs individual actions through policy evaluation, audit logging, data classification, trust, identity, and tool interception. Those controls are necessary, but they mostly evaluate requests as discrete events.
A multi-agent workflow can stay compliant at every individual step and still become unsafe through aggregation. A customer name is low sensitivity. Renewal status, usage analytics, support records, and financial indicators are each moderate. Combined, they can produce a highly sensitive customer risk profile, especially once an agent derives a conclusion like likely churn, financial distress, or account vulnerability.
So there is a gap: the policy engine can approve every individual action while the accumulated context crosses a sensitivity boundary. That gap gets sharper with delegation. A delegated agent might receive a harmless looking subtask, but the parent workflow may already carry constraints that should follow the work.
The idea
Treat context as a governed state that evolves during a workflow. A governance decision should consider not just the requested action, but the context state the action would produce if it succeeds.
The traditional flow looks like this:
flowchart LR A[Requested action] --> B[Policy evaluation] B --> C{Decision} C -->|allow| D[Execute] C -->|deny| E[Block] C -->|escalate| F[Human review]The proposed flow adds a context envelope and one more outcome:
flowchart LR A[Current context envelope] --> C[Accumulation] B[Requested action or delegation] --> C C --> D[Updated context envelope] D --> E[Policy evaluation] E --> F{Decision} F -->|allow| G[Execute and update envelope] F -->|constrain| H[Execute with downstream restrictions] F -->|deny| I[Block] F -->|escalate| J[Human review]The new outcome is constrain. Alongside allow, deny, and escalate, constrain means the action is permitted but the envelope it produces carries restrictions that bind later actions and delegations. At the engine level this is not a brand new verdict. It is realized as allow plus information-flow labels plus an obligation list the host enforces.
It builds on what AGT already has
This is additive. It composes with data classification, ABAC, policy evaluation, trust scoring, and audit lineage rather than replacing any of them.
DataClassification(public through top_secret) andDataLabel(categories like PII, PHI, PCI, GDPR, ITAR) live inagent_os/policies/data_classification.py, andABACPolicyalready enforces a max classification ceiling per datum. The envelope reuses these types rather than inventing a parallel vocabulary.result_labelsto a verdict and the host re-injects them on the next call. The envelope is basically the durable, workflow-scoped accumulation of those per-call labels.DelegationChain.validate()in the structural-authz integration already enforces that a delegatee's scopes are a subset of the delegator's. The job here is to make restriction inheritance uniform, not to invent attenuation.The context envelope
A structured record of accumulated governance state. It can attach to a session, a task, a delegation chain, or a derived artifact.
{ "context_envelope_id": "env_01JZ4A9K2M", "parent_context_envelope_id": "env_01JZ49Y8ND", "workflow_id": "wf_123", "agent_id": "agent.customer-success", "labels": ["pii", "financial", "behavioral"], "aggregate_sensitivity": "restricted", "derived_sensitivity": ["customer-risk-profile"], "delegation_depth": 2, "restrictions": ["no_external_delegation", "no_external_export", "no_memory_write"], "version": 7 }In AGT terms,
labelsmaps toDataLabel.categories,aggregate_sensitivityto a value on theDataClassificationladder, andagent_idto the existing agent DID.Aggregation versus inference
These are two different mechanisms and they need different controls. Aggregation is when a collection of individually permitted labels becomes restricted in combination. That is a counting or threshold problem. Inference is when low sensitivity inputs derive a higher sensitivity conclusion. That is a derivation tracking problem, and detecting an undeclared inference is undecidable in general.
The first cut governs aggregation. It does not try to detect inference. Instead it takes a conservative stance: any artifact produced while the envelope is already at or above a threshold inherits that classification, so detection only ever raises a label, it never gates on one.
Aggregation should not always elevate, either. Bounding false positives matters as much as catching real ones. A customer name plus a public SKU plus a public price list produces nothing new. Elevation should come from explicit, organization authored rules over label combinations, not from the mere fact that the envelope grew.
flowchart TD A[PII] --> D[Aggregation evaluator] B[Financial] --> D C[Behavioral] --> D D --> E[Restricted customer intelligence] E --> F[Downstream restrictions]Delegation
A delegated agent should never receive a less restrictive envelope than the parent permits. Reducing the data payload is fine. Silently stripping restrictions is not.
flowchart TD A[Parent envelope: restricted] --> B{Delegation request} B --> C[Redacted sub-context] B --> D[Inherited restrictions] C --> E[Child envelope] D --> E E --> F[Delegated agent]In established terms this is capability attenuation. Effective authority along a delegation chain is the intersection of the scopes at each hop, and authority cannot be amplified by delegating to a more privileged agent (the confused deputy problem). The first cut implements the restriction half of this as a grow-only union: a child's restrictions are the parent's plus whatever the child adds.
Governance invariants
Threat model
The guarantees here are only as strong as the instrumentation they see through, so it is worth being explicit about the adversary.
In scope: aggregation and constraint inheritance for non-adversarial agents operating within instrumented paths. Honest but buggy workflows, well intentioned agents that would otherwise accumulate sensitive context without noticing, and delegation that should carry restrictions forward. Within this scope the invariants are enforced.
Out of scope for the first cut, and acknowledged rather than closed:
Within scope, the invariants are enforced. Outside it, the system detects and records but does not prevent. Read the invariants with that boundary in mind.
Invariant 0, envelope authenticity: an envelope that crosses a trust boundary should be signed by the delegator and monotonically versioned, and a verifier should reject forged, downgraded, or replayed envelopes before accepting them. Without this, every invariant above is advisory across the mesh. The first cut runs in-process and defers this, which makes it the highest priority follow-up.
How a decision flows
sequenceDiagram participant Agent participant Gate as Governance gate participant Ctx as Envelope store participant Agg as Aggregation evaluator participant Policy as Policy engine participant Audit as Audit log participant Tool as Tool or delegated agent Agent->>Gate: Request action or delegation Gate->>Ctx: Load current envelope Tool-->>Gate: Result with result_labels Gate->>Agg: Fold actual result, recompute Agg-->>Gate: Updated labels, sensitivity, restrictions Gate->>Policy: Evaluate next action against the envelope Policy-->>Gate: allow, constrain, deny, or escalate Gate->>Audit: Record decision and transition Gate->>Ctx: Persist updated or child envelopeWorth calling out: accumulation happens after execution, not before. Folding in an action's actual
result_labelsonce it has run is something you can do. Projecting the labels of an output you have not produced yet is not, so the gate checks the next action against accumulated state rather than a guess.Authority intersection
flowchart LR A[Delegating context authority] --> E[Effective delegated authority] B[Delegated agent authority] --> E C[Projected context restrictions] --> E D[Organization policy] --> E E --> F{Delegation decision} F -->|safe| G[Delegate] F -->|unsafe| H[Deny or escalate]This is what stops authority laundering, where an agent hands work to a more privileged agent to get around a restriction.
Where it lines up with standards
It is mostly a data-governance and access-control mechanism, and notably it stores labels and classification levels rather than the underlying data, which keeps its own footprint small. It lines up with EU AI Act Article 12 (record-keeping) and Article 14 (human oversight), NIST 800-53 information flow and security attribute controls (AC-4, AC-16, AC-21), the NIST AI RMF functions, SOC 2 CC6 and CC7, GDPR Article 25, and HIPAA access and audit controls. None of those standards prescribes an aggregation control, which is the gap this fills.
Consequences
Benefits: closes the gap between action-level policy and workflow-level risk; improves governance for multi-agent delegation; lets sensitivity reflect accumulated knowledge; makes derived intelligence auditable; builds on the information-flow labels and delegation attenuation AGT already has.
Tradeoffs: it tracks context state across actions, which adds complexity to delegation and audit; aggregation rules need careful design to avoid ambiguity and over-escalation; declassification and minimization need explicit semantics so the system does not over-restrict forever.
Risks: poorly designed aggregation rules create false positives; weak inference handling misses sensitive derivations (which is why the first cut is honest about deferring inference); the envelope itself becomes a sensitive artifact that needs protection at its own classification.
Open questions
agent_os/policiesnext toDataClassification, with propagation hooks in the policy engine (labels) and inDelegationChain(restrictions), and no centralized store for the in-process first cut.Tracking issue: #2797.
All reactions