Skip to content

FINERACT-2900: Let the send-events job drain a backlog within one run - #6590

Open
jenny1371 wants to merge 1 commit into
apache:developfrom
jenny1371:events-drain-batches
Open

jenny1371 wants to merge 1 commit into
apache:developfrom
jenny1371:events-drain-batches

Conversation

@jenny1371

@jenny1371 jenny1371 commented Oct 6, 2026 •

Copy link
Copy Markdown
Contributor

Description

JIRA: https://issues.apache.org/jira/browse/FINERACT-2900

SendAsynchronousEventsTasklet reads and sends exactly one batch per run and then returns FINISHED. With the defaults (cron every minute, external-event-batch-size = 1000) the external events job therefore sends at most 1000 events per minute, even though a run only takes 0.12-0.28 s.

This adds fineract.events.external.max-batches-per-run (env FINERACT_EXTERNAL_EVENTS_MAX_BATCHES_PER_RUN, default 1, so the current behavior is unchanged). A run reads up to that many batches. Each batch is sent and marked as sent before the next one is read, and the run stops as soon as a batch is not full (backlog drained). If a send fails, the failed batch stays TO_BE_SENT and the run stops, exactly as today.

Measured (custom image built from this branch, PostgreSQL + Kafka, backlog of 12,000 TO_BE_SENT events, batch size 1000, default cron)

Setting Result
max-batches-per-run=1 (control, today's behavior) exactly 1000 events per run, runs 60 s apart, so 12,000 events take 12 minutes
max-batches-per-run=20 all 12,000 events sent in one run, about 4 s of job time

Per 1000-event batch (DEBUG timings): read 10-15 ms, message creation 90-300 ms, Kafka send ~25 ms, mark as sent 10-20 ms.

Tests

SendAsynchronousEventsTaskletTest: default reads one batch; exactly N batches when the backlog is larger; stops at a partial batch; a failure on the 2nd batch marks only the first batch as sent. ./gradlew :fineract-provider:spotlessCheck and the test class pass on JDK 25.

Checklist

  • Write the commit message as per our guidelines
  • Acknowledge that we will not review PRs that are not passing the build ("green") - it is your responsibility to get a proposed PR to pass the build, not primarily the project's maintainers.
  • Create/update unit or integration tests for verifying the changes made.
  • Follow our coding conventions.
  • Add required Swagger annotation and update API documentation (no API change)
  • This PR must not be a "code dump".
  • If merging this PR resolves a JIRA issue, I will mark that issue as resolved and set "Fix Version/s" appropriately.

SendAsynchronousEventsTasklet read and sent exactly one batch per run and then
returned FINISHED. With the defaults (cron every minute, external-event-batch-size
1000) the job therefore sent at most 1000 events per minute, about 17 events/s,
although a run takes only 0.12-0.28 s.

Add fineract.events.external.max-batches-per-run (default 1, which keeps the
current behavior). A run now reads up to that many batches, sending and marking
each batch as sent before reading the next, and stops early as soon as a batch is
not full (the backlog is drained). If a send fails, the failed batch stays
TO_BE_SENT and the run stops, exactly as before.

Tests: default keeps one batch per run; exactly N batches when the backlog is
larger; stops at a partial batch; a failure in the second batch marks only the
first batch as sent.
@jenny1371
jenny1371 force-pushed the events-drain-batches branch from ebc7469 to 7a7eb9c Compare October 9, 2026 04:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant