Skip to content
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 0 additions & 11 deletions kits/firestore-bigquery-export/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,3 @@
## Version 0.1.0

- Initial release of kit, see README for differences between the legacy extension and this kit
- fix: `eventarcpublishing.googleapis.com` and `roles/eventarc.publisher` are declared only when `EVENTARC_CHANNEL` is set in `.env`. A default install never publishes an event, yet every deploy prompted to enable the Eventarc Publishing API and declining aborted the deploy with `Must enable required APIs to deploy.` The extension declared only `bigquery.googleapis.com` and gained Eventarc publishing when the user opted into events at install; the kit now does the same. Detecting the channel needs firebase-tools 15.28.0 or later, which loads `.env` during deploy discovery; an older CLI skips the declarations even with `EVENTARC_CHANNEL` set, and every publish then fails with `PERMISSION_DENIED`, logged as a warning while the export continues. A blank `EVENTARC_CHANNEL` now disables events the same as an unset one.
- fix: cap `syncBigQuery` at `maxInstances: 500`, its declared `maxConcurrentDispatches`. At concurrency `1`, the default 100 instances would take only 100 of the 500 dispatches Cloud Tasks may have in flight, so the cap keeps capacity level with the dispatch limit. A dispatch above the instance ceiling waits up to about 10 seconds for a free instance, then gets a Cloud Run 429 that Cloud Tasks counts as a failed attempt and retries on the queue's schedule (5 attempts, 60s minimum backoff), slowing the queue while 429s continue. The handler never ran for such a dispatch, so a task that exhausts its attempts that way writes no `BACKUP_COLLECTION` row. At 0.1666 vCPU the cap needs about 84 vCPU of regional Cloud Run CPU quota, within the captured defaults (500 in most regions, 343 in `europe-west10` and `europe-west12`); the README describes the deploy failure a smaller quota produces and how to recover.
- fix: set `timeoutSeconds: 540` on `syncBigQuery`, `initBigQuerySync` and `setupBigQuerySync`, the extension's 1st gen task-function timeout, instead of the 2nd gen default of 60s. `fsexportbigquery` keeps 60s, which the deployed trigger ran at.
- fix: set every function's `cpu` to `"gcf_gen1"` (0.1666 vCPU), the extension's allocation, instead of the 1 vCPU that the 2nd gen default resolves to at 256MiB.
- fix: set every function's concurrency to `1`, and `fsexportbigquery` ingress to `ALLOW_INTERNAL_ONLY`, matching the extension. Previously, the kit inherited the 2nd gen defaults of concurrency `80` and ingress `ALLOW_ALL`. The task-queue functions `syncBigQuery`, `initBigQuerySync` and `setupBigQuerySync` declare no `ingressSettings`, so they inherit the default `ALLOW_ALL`, or whatever `setGlobalOptions({ ingressSettings })` sets; the deployed extension's task-queue functions ran with open ingress, and the `initBigQuerySync` HTTP POST needs it. At concurrency `1` a function serves as many requests at once as it has instances: the Cloud Run default of 100, unless your codebase sets `setGlobalOptions({ maxInstances })`.
- fix: an Eventarc publish failure no longer fails the function. A channel that was deleted or is unreachable answers `PERMISSION_DENIED`, and the publish was awaited before the work the function exists to do, so the document change was never exported. The publish failure is now logged as a warning and the invocation continues. Deploys also declared `eventarcpublishing.googleapis.com` unconditionally; the entry above makes that conditional on `EVENTARC_CHANNEL`.
- docs: the no-region fallback needs an explicit empty `DATABASE_REGION=` line in `.env`. Omitting the key entirely fails a non-interactive deploy, because the CLI resolves parameters before it reads their defaults. Corrects the note on the `DATABASE_REGION` placement fix below; no behavior change.
- fix: `USE_NEW_SNAPSHOT_QUERY_SYNTAX` and `EXCLUDE_OLD_DATA` take the extension's `yes` / `no` values again, so a config exported from the extension works unchanged. `WILDCARD_IDS` is unchanged: the extension already used `true` / `false` there.
- feat: reinstate the extension's Cloud Tasks write buffer. A failed inline BigQuery write now enqueues onto a new `syncBigQuery` task queue (5 attempts, 60s minimum backoff, throttled by the restored `MAX_DISPATCHES_PER_SECOND` param, default 100) instead of replaying the Firestore event through Eventarc redelivery for up to 24 hours; `MAX_ENQUEUE_ATTEMPTS` (default 3) is also back. The `onSuccess` event returns with the queue handler. Two behavior changes against earlier release candidates: a row that exhausts the queue is dropped unless `BACKUP_COLLECTION` is set (extension parity - the tracker backs the row up on every terminal insert failure, so configure a backup collection), and deleting or moving the functions can leave the Cloud Tasks queue behind. A failed enqueue is logged at error level, published as an `onError` event, and dropped, as in the extension; the trigger no longer declares `retry: true`, so nothing is redelivered through Eventarc. Export the new `syncBigQuery` function from your codebase entry, and deploy with Firebase CLI 15.28.0+ so the trigger can address its own queue (`FIREBASE_KIT_INSTANCE_ID`); requires firebase-admin 14.2.0+.
- fix: restore explicit function placement from `DATABASE_REGION`, now with the Firestore-location-to-Cloud-Run-region mapping. The `DATABASE_REGION` parameter is back and all three functions deploy to the region derived from it: regional locations pass through unchanged, and the multi-region locations map to a Cloud Run region (`nam5`/`nam7` to `us-central1`, `eur3` to `europe-west1`) instead of failing the deploy. With the parameter unset the functions still declare no region and the CLI falls back as before (`us-central1` by default, `FIREBASE_FUNCTIONS_DEFAULT_REGION` to override). Placement requires firebase-tools >= 15.28.0 (older CLIs do not load `.env` at discovery and keep the fallback). If your `.env` already carries `DATABASE_REGION` from an extension migration, upgrading to this version moves the functions to the mapped region on your next deploy, which deletes and recreates them.
- fix: stop deploying functions to the `DATABASE_REGION` value. Firestore multi-region locations (`eur3`, `nam5`, `nam7`) are not Cloud Run regions, so any multi-region database made every deploy fail. The `DATABASE_REGION` parameter is removed; the functions now declare no region and deploy to `us-central1` by default (set `FIREBASE_FUNCTIONS_DEFAULT_REGION` when deploying to choose another region), while the Firestore trigger is always pinned to the database's own region. `ExportConfig.location` is removed from the library surface.
Loading