[igrigorik]

Status: working draft for the Bulk Product Discovery DWG. Related: #682, #550

This proposal adds a way for a Business to publish its catalog in bulk — the same Product records Catalog Search returns — as files a Platform can index, embed, and rank offline, then call Catalog Search and Lookup at serving time for authoritative price and availability. The feed is point-in-time; the live operations stay authoritative.

Built from UCP primitives: discovery is a capability whose config names an endpoint; market scope is the catalog context; records are catalog Products; access is UCP's existing Platform authentication; extensions compose as they do for Search and Lookup. The 10k-foot view...

/.well-known/ucp                      capability advertised beside search and lookup
  -> GET config.endpoint              feed list, provisioned per caller (public or authenticated)
       -> feed entry                  one per market context; id, format, interval, generated_at, manifest URL
            -> manifest               one per snapshot, immutable at its URL
                 -> files[]           jsonl+gzip or parquet, RFC 9530 digest per file

Staged design is factored into a domain-neutral mechanism, dev.ucp.common.feed (discovery, feed list, manifest, files, snapshot semantics, supplemental feeds), plus dev.ucp.shopping.catalog.feed as an extension that specifies the record type to catalog Products and adds filters defined by shopping service.

---

Checklist

• Capability: New schemas (Discovery, Cart, etc.) or extensions.
• Documentation: Updates to README, or documentations regarding schema or capabilities.
• I have followed the Contributing Guide (including Conventional Commits title requirements and ! for breaking changes).
• I have updated the documentation (if applicable).
• My changes pass all local linting and formatting checks.

[frmoretto]

The split between feed list and manifest, and immutable snapshots at their URL, look right to me.

Just two gaps I'd raise:

1. Terms of use for the feed itself.
The feed exists so a Platform can "index, embed, and rank" offline, but neither the feed list nor the manifest lets the Business say what the records may be used for: search indexing yes, model training no, kept for how long. expires_at is retention of the files, not a limit on use.

UCP already has the pieces. links[] has a well-known terms_of_service type, and policies[] (#572) is an open, reverse-DNS typed primitive with a required description.
Neither appears on the feed documents, and policies[] is defined for rules about items "at the time of purchase", addressed to the buyer rather than to the Platform consuming the data.
Allowing links[] and policies[] on the feed list or the manifest, with a policy type for data use, would close the gap without new machinery. For the use categories, the IETF AI Preferences vocabulary (draft-ietf-aipref-vocab) could be referenced rather than reinvented.

2. A record count per file.
A file entry carries only url and digest. I'd add an optional records:

• digest is SHOULD, and may be absent when the serving path re-encodes bytes in transit. A record count survives re-encoding, so a Platform keeps a per-file completeness check.
• For jsonl+gzip the count cannot be known without downloading and decompressing the whole file (Parquet at least has it in the footer). With records, a snapshot can be sized from the manifest alone, which matters at the scale in the design note (~3,000 files for a 100M-product catalog).
• Feeds can be public, so not every consumer will run an indexing pipeline. A consumer with a fixed budget can decide from the manifest whether and how much to fetch, and report how much of a snapshot it saw. Partial Fetch already covers what it must not conclude from that.

Happy to post schema and prose for either as review suggestions.

Francesco Marinoni Moretto

[BarryLzc]

The draft looks great! I've added a few suggestions for the feed delta push：

Advantages for both side

Business

• Reduce the load on business systems during large-scale product feed generation, especially for feeds containing millions of products.

Platform

• The platform does not need to fetch the entire product dataset on every request

Schema definition

{
  "generated_at": "2026-09-16T12:00:00Z",
  "context": {
    "address_country": "CA",
    "language": "en",
    "currency": "CAD"
  },
  "format": "jsonl+gzip",
  "mode": "snapshot", // snapshot | delta
  "latest_sequence": 10542, // The latest Delta Sequence
  "files": [
    { "url": "https://cdn.example.com/feeds/snapshot_10542.jsonl.gz", "sha256": "..." }
  ],
  
  // P1 Delta  (Additive)
  "delta": {
    "base_sequence": 10000,        // The latest base snapshot version upon which the current Delta chain is based
    "latest_sequence": 10542,      // The latest Delta Sequence
    "min_available_sequence": 10500, // Initial Delta sequence retained by the server (if exceeded, a full snapshot must be taken).
    "files": [
      {
        "sequence": 10541,
        "url": "https://cdn.example.com/feeds/delta_10541.jsonl.gz",
        "created_at": "2026-09-16T12:05:00Z"
      },
      {
        "sequence": 10542,
        "url": "https://cdn.example.com/feeds/delta_10542.jsonl.gz",
        "created_at": "2026-09-16T12:10:00Z"
      }
    ]
  }
}

Manifest Reponsibility

The Manifest retains its single snapshot responsibility while exposing an optional delta object for increment consumers

Field	Type	Description	Engineering Purpose & Validation Logic
base_sequence	Integer	Baseline Snapshot Sequence is the Sequence ID of the full snapshot on which this delta chain is anchored.	Baseline Alignment Check: Clients verify that their local dataset originates from the matching snapshot baseline. If the local sequence is lower than base_sequence, the delta chain has reset, requiring a full snapshot pull.
latest_sequence	Integer	Latest Delta Sequence is the Sequence ID of the most recently generated delta package in the system.	Synchronization Target: Indicates the highest sequence available, allowing clients to calculate their sync lag relative to the latest system state.
min_available_sequence	Integer	Minimum Available Sequence is the lowest Delta Sequence ID currently retained on the server.	Gap Detection (Fallback Trigger): If a client's local sequence < min_available_sequence, intermediate delta files have been purged (gap in the chain). The client MUST invalidate local data and fallback to a full snapshot.
files	Array	Delta File Manifest is an ordered array of delta file metadata sorted sequentially by sequence.	Download Queue: The client filters for entries where sequence > local_sequence and applies them strictly in ascending order.
↳sequence	Integer	The monotonically increasing sequence ID for a specific delta file.	Used for deterministic ordering and state tracking (cursor position).
↳url	String	The HTTPS endpoint serving the compressed delta file (.jsonl.gz / .parquet).	Direct payload retrieval path (CDN-friendly).
↳created_at	String	ISO 8601 timestamp representing when the delta file was generated.	Used for latency monitoring, auditing, and temporal validation.

Flow Chart

Client-Side Synchronization Decision Flowchart

Client A VS Client B Decision Sequence Diagram

ClientA: Manifest (base=10000, min_avail=10029, latest=10048)

ClientB -> Server: GET /list_feeds / manifest.json
Server --> ClientB: Manifest (base=10000, min_avail=10029, latest=10048)

== Client A Evaluation (10045 >= 10029) ==
note over ClientA
Validation Passed:
10045 >= min_available_sequence (10029)
Delta chain intact!
end note
ClientA -> Server: GET delta_10046.jsonl.gz
ClientA -> Server: GET delta_10047.jsonl.gz
ClientA -> Server: GET delta_10048.jsonl.gz
ClientA -> ClientA: Apply deltas sequentially\nUpdate local cursor = 10048

== Client B Evaluation (10015 < 10029) ==
note over ClientB
Validation Failed:
10015 < min_available_sequence (10029)
Gap detected! Files 10016~10028 purged.
end note
ClientB -> ClientB: Purge local cache / database
ClientB -> Server: GET snapshot_10000.jsonl.gz
ClientB -> ClientB: Load full snapshot\nUpdate local cursor = 10000
@enduml -->

Rules

Can Apply Delta = (Local Sequence >= base_sequence)

Can Apply Delta = (Local Sequence >= min_available_sequence) AND (Local Sequence <= latest_sequence)

[gsmith85]

On extension chaining: requiring record-level extensions to declare extends: "dev.ucp.shopping.catalog.feed" creates an artificial split with schemas like fulfillment and policy.return, which already canonically extend catalog.search or lookup.

I'd keep the manifest hierarchy scoped to the document: common.feed as the root, and catalog.feed as the record type extending it. Any extensions that decorate the Product record can keep their normal canonical parentage as advertised in the profile, without needing to synthesize a parallel inheritance tree just for feeds.

flowchart LR
    subgraph Current ["Current PR #866 (Split Inheritance)"]
        direction TB
        C_Root["dev.ucp.common.feed"] --> C_Feed["dev.ucp.shopping.catalog.feed"]
        C_Feed -->|"forced extends"| C_Ext1["dev.ucp.shopping.fulfillment"]
        C_Feed -->|"forced extends"| C_Ext2["dev.ucp.shopping.policy.return"]
        C_Search["dev.ucp.shopping.catalog.search"] -->|"canonical extends"| C_Ext1
    end
    subgraph Proposed ["Proposed (Decoupled Scope)"]
        direction TB
        P_Doc["dev.ucp.common.feed\n(Document Root)"] --> P_Feed["dev.ucp.shopping.catalog.feed\n(Record Type = Product)"]
        P_Feed -.->|"records conform to"| P_Canon["Canonical Product + Extensions\n(fulfillment, policy.return, etc.)"]
    end

Loading

[igrigorik]

This doesn’t require a separate extension or duplicated field definitions. An extension can target both Catalog Feed and live Catalog capabilities, reusing shared $defs while defining the appropriate shape for each surface—just as Fulfillment does across Cart and Checkout.

Declaring the feed parent makes that applicability explicit. We shouldn’t infer it from Search/Lookup parent, because an extension may intentionally expose different or non-overlapping data on those surfaces

[jamesandersen]

Great progress here @igrigorik; this brings together many of the concepts we've been discussing in the DWG so far nicely.

[vink28]

This section defines the mechanism clearly, but I think it would land much better with a worked example. As a reader I can follow that an extension declared in the manifest's ucp.capabilities adds typed fields to records, but it took me a while to work out what a Business actually has to do end to end — and that path is the one most likely to be walked by people new to UCP.

Something like the following, for a Business adding five apparel attributes:

  1. Publish an extension schema at an authority-bound URL on a domain the Business controls:

  {
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$id": "https://merchant.example.com/ucp/schemas/apparel.json",
    "name": "com.example.merchant.apparel",
    "$defs": {
      "dev.ucp.shopping.catalog.feed": {
        "allOf": [
          { "$ref": "https://ucp.dev/{version}/schemas/shopping/product.json" },
          { "type": "object",
            "properties": {
              "fabric": { "type": "string" },
              "care_instructions":  { "type": "string" },
              "fit":  { "type": "string", "enum": ["slim", "regular", "relaxed"] },
              "sustainability_rating": { "type": "integer", "minimum": 1, "maximum": 5 },
              "country_of_manufacture": { "type": "string" }
            } }
        ]
      }
    }
  }

  2. Declare it in the profile, alongside the feed capability and the record type:

  "capabilities": {
    "dev.ucp.common.feed": [
      { "version": "{version}", "spec": "...", "schema": "...",
        "config": { "endpoint": "https://merchant.example.com/ucp/feeds" } }
    ],
    "dev.ucp.shopping.catalog.feed": [
      { "version": "{version}", "spec": "...", "schema": "...",
        "extends": "dev.ucp.common.feed" }
    ],
    "com.example.merchant.apparel": [
      { "version": "{version}",
        "spec":    "https://merchant.example.com/docs/apparel",
        "schema":  "https://merchant.example.com/ucp/schemas/apparel.json",
        "extends": "dev.ucp.shopping.catalog.feed" }
    ]
  }

  3. Declare it in each manifest whose records carry those fields:

  "ucp": {
    "version": "{version}",
    "capabilities": {
      "dev.ucp.common.feed": [{ "version": "{version}" }],
      "dev.ucp.shopping.catalog.feed": [{ "version": "{version}", "extends": "dev.ucp.common.feed" }],
      "com.example.merchant.apparel":  [{ "version": "{version}", "extends": "dev.ucp.shopping.catalog.feed" }]
    }
  }

  4. Emit the fields inline in the record:

  {"id": "P100", "title": "Oxford Shirt", "price_range": {}, "variants": [], "fabric": "100% cotton", "care_instructions": "Machine wash 30", "fit":
  "slim", "sustainability_rating": 4, "country_of_manufacture": "PT"}

Writing it out surfaced one question I couldn't answer from the text. In the example above the extension extends dev.ucp.shopping.catalog.feed, so the five attributes exist only in the feed. If the Business wants the same attributes in live Catalog Search responses — which seems likely, given the records are described as "exactly as returned by Catalog Search" — does it need a second extension against dev.ucp.shopping.catalog.search, or is one extension expected to span both? Worth making explicit either way.

[YanisMtcr]

On alex-jansen's point about small in-place changes and JMathnov's question on file size: if a share λ of products changes on a given day and products are spread across files independently of how often they change, a file of n products is untouched with probability about (1 − λ)^n. At the roughly 1 to 3% of prices a day implied by published studies of online retailers (for example Cavallo 2018, NBER w25138), a file of a few hundred products almost always changes, so splitting a snapshot into smaller files and skipping unchanged ones saves little unless files hold only a handful of products, and grouping volatile products together helps only with small files. That suggests choosing file size for transfer and parallelism, and leaving small changes to deltas.

Two thoughts for the delta phase, on the minimal delta mode bennyblum proposed on #550, itself building on BarryLzc's proposal here:

1. Compaction can cost more than the deltas themselves: when deltas are folded into a new base, a Platform already in sync would download the new base again, or most of it. If a compacted snapshot could declare the base and deltas it is equivalent to, and carry the last delta it folds alongside, a Platform in sync could apply that delta and adopt the new base without fetching it.
2. The supplemental-feed merge rules (members replaced whole, null removing a member, identified lists merged on id) could define what upsert does to an existing record, so supplements and deltas share one merge model. A delta would still need what a supplement cannot do: inserting or deleting a record or a list element, and referencing its base snapshot.

[damaz91]

📋 Discussed in UCP Bulk Product Discovery Domain Working Group (DWG) Meeting - 2026-10-28

This pull request was referenced under the following agenda topic(s):

• Evaluating brand attribute in product schema (view section)
• Distinguishing barcodes from manufacturer part numbers (view section)
• Offer price effective dates and time-bounded pricing (view section)
• Variant identifying properties and taxonomy mapping (view section)
• Base pull request status and December 10 release timeline (view section)
• Compliance attributes and messaging tool framework (view section)

See the full meeting minutes for discussion notes and action items.

[damaz91]

📋 Discussed in UCP Shopping Tech Council (STC) Meeting - 2026-10-23

This pull request was referenced under the following agenda topic(s):

• Feeds Domain Working Group Progress (view section)

See the full meeting minutes for discussion notes and action items.

[igrigorik]

Thanks everyone for the great feedback and comments! I believe I've addressed and captured all the outcomes from the group and GH discussions. PTAL.

[frmoretto]

One small, non-blocking clarification on Supplemental Feeds. The common spec already requires a supplement to carry members whose values depend on members it changes, and the catalog example applies this to price_range.

Could unit_price be named there too? Its amount is computed from price and the measure, and the Variant schema says the Platform must not recompute it, so a supplement that changes price and leaves unit_price alone passes a stale value straight through. The currency invariant won't catch it when both feeds are in the same currency, say a DE base and an FR supplement in EUR. For EU merchants that value can end up as the unit price shown next to the selling price, which is required for many products (Directive 98/6/EC, Art. 3).

If you'd rather keep this PR narrow, a follow-up issue works too.

Francesco Marinoni Moretto

[igrigorik]

@frmoretto clarified in 7f1161e for that particular example. Ultimately, we cannot define or enumerate all the dependencies, and we do have prose that delegates this responsibility to the business that generates the feed.