TASK: Add local SIT distributed tracing with Zipkin

[ramioooz]

Objective

Add a local-SIT distributed tracing baseline for the HTTP platform path so operators can correlate one request across API Gateway and downstream Spring services.

Parent

• Parent story: #64
• Sprint: Sprint 6 - Operational Resilience and Observability

Scope

• Add the Micrometer tracing bridge and Zipkin reporter only to services that participate in the HTTP request path.
• Add configurable SIT Zipkin endpoint and sampling settings through config-repo.
• Add a local SIT Zipkin deployment and ClusterIP service in infra-sit.
• Preserve existing correlation-ID behavior and keep trace/span identifiers out of business payloads.
• Document local verification and explicitly defer AWS X-Ray/OpenTelemetry Collector mapping.

Acceptance Criteria

• API Gateway and participating HTTP services emit trace spans when tracing is enabled.
• SIT sends spans to an in-cluster Zipkin endpoint without public exposure.
• Sampling and endpoint settings are environment-configurable and disabled or safely bounded outside SIT.
• A controlled gateway request can be correlated by trace ID across at least two services.
• Helm lint/template, affected Maven verification, and a focused smoke check pass.
• No new CI workflow or broad redundant unit-test suite is introduced.

Dependencies and Rollout Notes

• Depends on the existing structured logging/correlation baseline in STORY: Add structured logging, correlation IDs, and Zipkin tracing #64.
• Related runtime work: STORY: Implement centralized logging with OpenSearch #91 and TASK: Collect Kubernetes logs with Fluent Bit #94 remain separate centralized-log collection concerns.
• UAT/PROD tracing backend and AWS mapping remain deferred with Sprint 7.

[ramioooz]

Zipkin tracing implementation PRs\n\nOpened the implementation as separate reviewable PRs:\n\n- infra-sit#37: local SIT Zipkin 3.6.1 ClusterIP deployment with in-memory storage\n- config-repo#45: shared SIT exporter endpoint, sampling, and safe default\n- api-gateway#25: managed Spring Boot Zipkin starter\n- customer-service#36: managed Spring Boot Zipkin starter\n- account-service#41: managed Spring Boot Zipkin starter\n- transaction-service#23: managed Spring Boot Zipkin starter\n- mfa-service#16: managed Spring Boot Zipkin starter\n- ledger-service#21: managed Spring Boot Zipkin starter\n\nLocal verification before PR creation:\n- Helm lint/template passed for the Zipkin chart.\n- Existing test suites passed: Gateway 20, Customer 7, Account 64, Transaction 89, MFA 46, Ledger 27.\n- No new tests or CI workflows were added.\n\nMerge order:\n1. infra-sit#37\n2. config-repo#45\n3. service PRs in any order: api-gateway#25, customer-service#36, account-service#41, transaction-service#23, mfa-service#16, ledger-service#21\n4. Roll out Zipkin and the rebuilt services in SIT, then verify one gateway request across at least two services.\n\nThe tracking item remains In progress until the SIT trace-correlation evidence is captured.

[ramioooz]

The handoff record was refreshed in .github PR #241 with the tracing scope, PR links, verification counts, and post-merge SIT evidence required before closure.

[ramioooz]

Follow-up hardening was pushed to the six service PRs: the Zipkin exporter is disabled by default for standalone/test execution and is enabled only by the shared SIT profile. Existing suites were rerun successfully with the same counts; no new tests or CI jobs were added.

[ramioooz]

All implementation PR CI is now green and every PR is open, non-draft, and mergeable. The Zipkin PR set is ready for review. After merge, rollout order remains infra-sit#37, config-repo#45, then the six service PRs, followed by SIT trace-correlation verification.

[ramioooz]

Tracing implementation wave is complete in reviewable PRs. The existing task remains In progress until the PRs are merged and SIT correlation is verified.

New service coverage:

• Auth Service: feat: add opt-in Zipkin tracing to auth-service auth-service#10
• Payment Service: feat: add opt-in Zipkin tracing payment-service#11
• Notification Service: feat: add opt-in Zipkin tracing to notification-service notification-service#12

Existing tracing wave:

• API Gateway: feat: enable Zipkin tracing in API Gateway api-gateway#25
• Customer Service: feat: enable Zipkin tracing in Customer Service customer-service#36
• Account Service: feat: enable Zipkin tracing in Account Service account-service#41
• Transaction Service: feat: enable Zipkin tracing in Transaction Service transaction-service#23
• MFA Service: feat: enable Zipkin tracing in MFA Service mfa-service#16
• Ledger Service: feat: enable Zipkin tracing in Ledger Service ledger-service#21
• SIT Zipkin backend: feat: add local SIT Zipkin tracing backend infra-sit#37
• SIT exporter configuration: feat: configure SIT Zipkin tracing exporter config-repo#45

All new service PRs passed local ./mvnw verify, Helm lint/template validation, and git diff --check. GitHub Maven checks are still pending; Helm checks are green. No redundant tests, CI jobs, routes, or business behavior were added.

After merge, deploy the Zipkin backend/config and all instrumented services to SIT, exercise one gateway-to-service request, and verify the trace appears in Zipkin before closing this task.

[ramioooz]

SIT verification update (2026-09-06)

• Zipkin candidate chart is deployed and healthy in digital-bank-sit; /health reports UP.
• Rebuilt merged API Gateway and Auth Service tracing images from origin/main with temporary local tags and deployed them to SIT.
• A controlled gateway login produced a single trace containing spans from both api-gateway and auth-service; Zipkin service discovery lists both.
• Rollout exposed a chart security-context defect because the image user is symbolic; infra-sit PR #39 sets numeric UID/GID 1000 while retaining runAsNonRoot.
• .github PR #245 records this evidence in the handoff.

The task remains open until PR #39 is reviewed/merged and the trace is repeated with the normal release tags.

[ramioooz]

2026-09-06 candidate verification update:

• The reviewed infra-sit PR STORY: Define service naming, ports, and repository conventions #39 candidate was deployed in the local Docker Desktop SIT cluster with the numeric Zipkin UID/GID correction. Zipkin became Ready and accepted a real API Gateway-to-Auth Service correlated trace across both service spans.
• This is pre-merge runtime evidence only. Repeat the same check with the normal release-tag chart after PR STORY: Define service naming, ports, and repository conventions #39 merges; keep this task open until then.

[ramioooz]

Post-merge SIT verification (2026-09-06)

• Merged infra-sit#39 was deployed successfully; Zipkin is 1/1 Ready with zero restarts.
• Zipkin /health reports UP and its in-memory storage is healthy.
• Zipkin service discovery currently includes account-service, api-gateway, auth-service, and transaction-service.
• The normal release-tag deployment is therefore healthy after the numeric UID/GID correction.

The task remains open until one fresh gateway request is correlated across at least two service spans after the merged release rollout. AWS tracing and collector mapping remain deferred as documented.

[ramioooz]

Fresh post-merge trace evidence (2026-09-06)

• Sent GET /account-service/actuator/health through the SIT API Gateway after deploying the merged release tags.
• Response was 200 OK; Account Service reported both liveness and readiness groups UP.
• Zipkin trace ID: 6a9d60564d687c479f9c8a6dc06da9fe.
• The trace contains the API Gateway server span, the Gateway client span for route account-service-health, and the Account Service server span.

This satisfies the local SIT cross-service trace-correlation acceptance criterion. AWS tracing and collector mapping remain deferred. The task can now be closed with the Sprint 6 observability work item.