You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Add a FileSearchRepository backed by sqlite-vec for semantic cosine retrieval on a file-backed SQLite database. It keeps portable SQL metadata and published-chunk visibility, normalizes embeddings before indexing, and records a provider identity so stores cannot silently switch indexes. Keyword and hybrid search continue through the bounded exact ranker. SQLite metadata and vector rows now publish in one transaction, so concurrent replicas cannot remove a winning attachment. Orphan cleanup rechecks rows under a writer lock and immediately reclaims stale rows and per-store tables; search skips empty stores and unpublished rows, and uses bounded exact ranking when sqlite-vec's 4,096-candidate limit is exceeded. The loadable extension path is deployment-controlled; #485 wires its configuration.
BEIR-sized corpus testing exposed the old 64 MiB serialized-corpus limit and top-k edge cases with duplicate chunks and near ties. This branch keeps the default exact-search limit at 64 MiB and gives sqlite-vec a 256 MiB limit, validates vector-index coverage, and adaptively fetches candidates to preserve the requested document count.
Review follow-up: the portable vector map is migration-owned; extension-specific index SQL lives in storage/file_search/sqlite_vec_index.rs, publication has one implementation, and writes use BEGIN IMMEDIATE without a process-wide mutex. Focused storage and schema tests passed; the original 3 sqlite-vec integration tests passed, including immediate orphan-row and orphan-table recovery.
SQLITE_VEC_EXTENSION_PATH=/tmp/vec0.so cargo test -p agentic-server-core --test sqlite_vec_repository -- --ignored: 4 passed in CI with sqlite-vec 0.1.9, covering publication, extreme finite embeddings, restart, filtered and duplicate-chunk search, deletion, concurrent attachment conflicts, unpublished rows, empty stores, and searches beyond the 4,096-candidate limit.
Maral's capacity regression: a new sqlite-vec integration test publishes a serialized store above 64 MiB, then exercises keyword, hybrid, and filtered semantic searches against it. The Rust CI run passed this test and the full ignored sqlite-vec integration suite.
CI downloads the pinned Linux sqlite-vec release asset, verifies its SHA-256, and runs the ignored integration tests.
Compared the provider's semantic search with OGX's published vector-mode results. Used the official BEIR test corpora, queries, and relevance judgments, title + body text, cl100k_base 512-token chunks with 128-token overlap, nomic-ai/nomic-embed-text-v1.5 embeddings, and top-10 document scoring with pytrec_eval. The 512/128 setting comes from OGX's documented benchmark config, but this was a local preparation, not OGX's full API ingestion path. A local harness loaded the prepared chunks into the actual SqliteVecRepository and called its search method for every judged query. sqlite-vec 0.1.9 ran on CPU.
Dataset
Documents / chunks / queries
nDCG@10 OGX vector → ours
Recall@10 OGX vector → ours
MAP@10 OGX vector → ours
NFCorpus
3,633 / 4,921 / 323
0.3106 → 0.3101
0.1482 → 0.1462
0.1153 → 0.1137
SciFact
5,183 / 6,604 / 300
0.6943 → 0.6968
0.8369 → 0.8296
0.6442 → 0.6496
ArguAna
8,674 / 9,278 / 1,406
0.3765 → 0.3928
0.7610 → 0.8037
0.2542 → 0.2624
All 2,029 searches succeeded. An independent exhaustive NumPy cosine calculation on the same embeddings gave identical aggregate nDCG@10 for SciFact and ArguAna; NFCorpus differed by 0.000084. This validates sqlite-vec against an exact-cosine oracle on our prepared vectors. The proximity to OGX's published vector scores is indicative, not exact parity: OGX's current batch ingestion path applies whitespace cleaning and appears to use 800/400-token file-processor defaults for automatic chunking, despite the benchmark config documenting 512/128. The published run does not include materialized chunk/embedding or per-query result fixtures, so its exact input and rankings cannot yet be compared.
Scope and limit: This is a repository-level semantic retrieval test with prepared embeddings; it does not exercise the Files/Vector Stores HTTP APIs, automatic ingestion, hybrid reranking, Responses file_search, or end-to-end answer generation. OGX's fourth BEIR dataset, FiQA, produces 63,129 chunks and exceeds this branch's 10,000-chunk corpus bound, so full four-dataset parity remains unvalidated.
## Summary
Add typed file, vector-store, and search contracts; pluggable
`FileSearchRepository` and `FileBlobStore` interfaces; portable SQL
metadata and bounded exact retrieval; and local blob storage. Store
listing batches current attachment counts and usage. Durable,
token-fenced cleanup intents cover uploads and deletions; expired leases
cannot let a publisher expose missing content. Recovery retains
abandoned-upload tombstones for later sweeps when a delayed write
finishes after an initial cleanup pass. Local staging is addressable by
file ID, and cancellation drains publication before cleanup. This is the
independently compilable foundation of the file-search stack; HTTP
routes and periodic cleanup are in #485. CI uses the macOS 15 Apple
Silicon runner because GitHub is retiring macOS 14 and its scheduled
brownout canceled wheel validation.
Review order: #483 → #484 → #485 → #486 → #34. PostgreSQL vector
indexing remains in follow-up #278.
## Test Plan
- `cargo test --offline -p agentic-server-core --test
file_search_service`: 31 passed, 1 PostgreSQL test ignored locally.
Includes two-service upload/recovery interleavings, restart recovery
after metadata rejection plus failed blob removal, delayed-staging
cancellation, and a late-write/failed-rollback regression.
- `cargo test --offline -p agentic-server-core --lib
storage::local_files::tests`: 5 passed, including absent-directory and
deterministic staging cleanup.
- `cargo test --offline -p agentic-server-core --lib
storage::schema::tests`: 6 passed, 4 PostgreSQL tests ignored locally;
PostgreSQL CI exercises the final schema.
- `cargo clippy --offline --all-targets -- -D warnings`, `cargo fmt --
--check`, and repository pre-commit hooks: passed.
- The official OpenAI Python SDK HTTP integration test is in #485 and
passed locally against the rebuilt final stack.
---------
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
The reason will be displayed to describe this comment to others. Learn more.
Warning
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds a sqlite-vec–backed FileSearchRepository to enable cosine semantic retrieval over a file-backed SQLite DB, plus CI coverage for the loadable extension integration test.
Changes:
Introduces SqliteVecRepository with transactional staging, orphan recovery, and semantic search via sqlite-vec.
Adds shared semantic candidate ranking logic for provider-native cosine results.
Adds an ignored integration test and a CI job step that downloads and verifies sqlite-vec, then runs the ignored test.
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Search candidate accumulation can retain gigabytes of unused embeddings, while attachment snapshot races, synchronous ranking on Tokio workers, and inconsistent byte-limit enforcement introduce additional reliability risks.
[P2] When the vector table already exists, this no-op CREATE can establish a read snapshot before the deferred transaction acquires SQLite's writer lock. If another connection commits before the map insert, the write upgrade fails with SQLITE_BUSY_SNAPSHOT, so valid concurrent attachments can return storage errors. Reserve the writer lock before this statement, as cleanup already does, and add a concurrent-attachment regression with an initialized table.
Thanks, the other items look resolved. One regression from splitting the limits: SQLITE_VEC_MAX_CORPUS_BYTES admits up to 256 MiB, but only unfiltered semantic search reads from the index. Filtered semantic search and the >4,096-candidate fallback (sqlite_vec/search.rs:96), and all keyword and hybrid searches (sqlite_vec/mod.rs:302 → FileSearchStorage::search), load the corpus through FileSearchStorage::chunks(), which still enforces the 64 MiB MAX_CORPUS_BYTES. A sqlite-vec store between 64 and 256 MiB is therefore accepted at ingestion, but hybrid (the default mode when embeddings are configured), keyword, and filtered searches on it return 503 "Selected corpus exceeds the exact retrieval capacity".
Could you either pass the provider's limit into chunks() for these paths, or keep sqlite-vec admission at 64 MiB until those modes are indexed? A test with a >64 MiB sqlite-vec store running a hybrid or filtered search would cover it.
## Summary
Expose the Files and vector-store upload, lifecycle, attachment, and search routes over the pluggable core service. Add server configuration for local file bytes, embeddings, and an optional trusted sqlite-vec extension path. Wire the routes into authentication and OpenAPI, with deployment documentation. Reconcile abandoned blob cleanup intents at startup and periodically while the gateway runs. Local routes enforce the configured API key and request-body ceiling; deletion tolerates a cleanup marker already removed by reconciliation, and startup logs provider setup diagnostics. This layer does not register the Responses `file_search` built-in tool.
Review order: #485 → #486 → #34. #483 and #484 are merged. PostgreSQL vector indexing remains in #278.
## Test Plan
- `cargo test -p agentic-server --test file_search_http_test`: 6 passed, including missing, incorrect, and valid API-key cases and configured multipart/JSON body limits.
- `cargo test -p agentic-server-core --test file_search_service`: 32 passed, 1 ignored, including the deletion cleanup-marker race regression.
- Full workspace `cargo test --offline` passed on the restacked #34 head containing these fixes.
- `cargo test -p agentic-server --lib openapi`: 19 passed.
- `cargo test -p agentic-server --bin agentic-server config_file::tests`: 37 passed.
- `cargo build --offline -p agentic-server --bin agentic-server` and `cargo clippy --offline -p agentic-server -p agentic-server-core --all-targets -- -D warnings`: passed.
- `openai==3.8.0` client E2E: upload, list/retrieve/content, vector-store attachment, filtered search, detach, and deletion passed against the rebuilt local gateway. The Rust CI job runs this script.
- Formatting and repository pre-commit hooks passed on signed-off commits. CI also runs PostgreSQL file-search persistence, sqlite-vec, and bounded PDF ingestion checks.
- OpenAI reference cassette parity for Files and vector-store routes remains unverified: this environment has no OpenAI API credential. The focused local gateway HTTP suite and OpenAI SDK client E2E cover the route lifecycle; no reference cassette was hand-authored.
Signed-off-by: Francisco Javier Arceo <4163062+franciscojavierarceo@users.noreply.github.com>
## Summary
Register and execute the Responses `file_search` built-in tool through the existing typed gateway tool loop. The handler searches the configured file-search service, keeps retrieved passages private for model continuation, exposes `file_search_call.results` only when requested, and restores typed file citations in completed and streamed output. This is the top layer of a smaller stack; the Files and vector-store APIs, provider contracts, SQLite-vec adapter, and protocol types are reviewed below it.
Review order: #483 (storage contracts and portable SQL/local providers) → #484 (SQLite-vec) → #485 (Files and vector-store HTTP APIs and provider configuration) → #486 (Responses protocol and stream lifecycle) → this PR. The existing PostgreSQL/pgvector follow-up is #278; later retrieval and lifecycle work remains stacked above it.
The storage service accepts injected `FileSearchRepository` and `FileBlobStore` implementations. The lower layers configure SQLite-vec for semantic retrieval; Milvus and remote blob stores remain adapter extension points.
## Test Plan
- `cargo test -p agentic-server-core --test file_search_tool_test`: 6 passed, covering blocking and streaming calls, citations, opt-in results, mixed client tools, failures, and selector validation.
- `cargo test -p agentic-server-core --lib`: 1,385 passed, 6 ignored on the corrected, restacked file tree. Full workspace `cargo test --offline` also passed on that tree.
- Changed-crate all-target Clippy with warnings denied, formatting, and all-file pre-commit hooks passed on the corrected, restacked file tree.
- The failing Rust CI check at `33b20ba` came from an output-item test fixture missing the typed `logprobs: []` field. The corrected citation projection test passes locally; CI is running on the cleanly restacked head.
- SQLite-vec review fixes from #484 are carried through the stack: atomic publication, safe orphan cleanup, empty/unpublished row handling, and exact fallback above sqlite-vec's candidate limit.
Signed-off-by: Francisco Javier Arceo <arceofrancisco@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
maralbahari
left a comment
Summary
Add a
FileSearchRepositorybacked by sqlite-vec for semantic cosine retrieval on a file-backed SQLite database. It keeps portable SQL metadata and published-chunk visibility, normalizes embeddings before indexing, and records a provider identity so stores cannot silently switch indexes. Keyword and hybrid search continue through the bounded exact ranker. SQLite metadata and vector rows now publish in one transaction, so concurrent replicas cannot remove a winning attachment. Orphan cleanup rechecks rows under a writer lock and immediately reclaims stale rows and per-store tables; search skips empty stores and unpublished rows, and uses bounded exact ranking when sqlite-vec's 4,096-candidate limit is exceeded. The loadable extension path is deployment-controlled; #485 wires its configuration.BEIR-sized corpus testing exposed the old 64 MiB serialized-corpus limit and top-k edge cases with duplicate chunks and near ties. This branch keeps the default exact-search limit at 64 MiB and gives sqlite-vec a 256 MiB limit, validates vector-index coverage, and adaptively fetches candidates to preserve the requested document count.
Review order: #483 → #484 → #485 → #486 → #34. PostgreSQL vector indexing remains in #278.
Test Plan
storage/file_search/sqlite_vec_index.rs, publication has one implementation, and writes useBEGIN IMMEDIATEwithout a process-wide mutex. Focused storage and schema tests passed; the original 3 sqlite-vec integration tests passed, including immediate orphan-row and orphan-table recovery.SQLITE_VEC_EXTENSION_PATH=/tmp/vec0.so cargo test -p agentic-server-core --test sqlite_vec_repository -- --ignored: 4 passed in CI with sqlite-vec 0.1.9, covering publication, extreme finite embeddings, restart, filtered and duplicate-chunk search, deletion, concurrent attachment conflicts, unpublished rows, empty stores, and searches beyond the 4,096-candidate limit.cargo test -p agentic-server-core --lib: 1,370 passed, 6 ignored.cargo clippy -p agentic-server-core --all-targets -- -D warnings,cargo fmt -- --check, and pre-commit hooks passed.BEIR retrieval evidence (2026-11-03, benchmark commit
fd6b574)Compared the provider's semantic search with OGX's published vector-mode results. Used the official BEIR test corpora, queries, and relevance judgments, title + body text,
cl100k_base512-token chunks with 128-token overlap,nomic-ai/nomic-embed-text-v1.5embeddings, and top-10 document scoring withpytrec_eval. The 512/128 setting comes from OGX's documented benchmark config, but this was a local preparation, not OGX's full API ingestion path. A local harness loaded the prepared chunks into the actualSqliteVecRepositoryand called itssearchmethod for every judged query. sqlite-vec 0.1.9 ran on CPU.All 2,029 searches succeeded. An independent exhaustive NumPy cosine calculation on the same embeddings gave identical aggregate nDCG@10 for SciFact and ArguAna; NFCorpus differed by 0.000084. This validates sqlite-vec against an exact-cosine oracle on our prepared vectors. The proximity to OGX's published vector scores is indicative, not exact parity: OGX's current batch ingestion path applies whitespace cleaning and appears to use 800/400-token file-processor defaults for automatic chunking, despite the benchmark config documenting 512/128. The published run does not include materialized chunk/embedding or per-query result fixtures, so its exact input and rankings cannot yet be compared.
Scope and limit: This is a repository-level semantic retrieval test with prepared embeddings; it does not exercise the Files/Vector Stores HTTP APIs, automatic ingestion, hybrid reranking, Responses
file_search, or end-to-end answer generation. OGX's fourth BEIR dataset, FiQA, produces 63,129 chunks and exceeds this branch's 10,000-chunk corpus bound, so full four-dataset parity remains unvalidated.