test: add AgentBakerSvc load testing and profiling

[Sri Harsha (SriHarsha001)]

What

• add an open-loop AgentBakerSvc load generator with steady-rate, phased burst, and synchronized concurrency-wave schedules
• add a synthetic Linux bootstrap fixture and per-request JSONL/percentile reporting
• distinguish terminal attempts (completed) from delivered capacity (successful / successfulRps) and report success-only latency
• assign stable nonzero sequence IDs across default and file-based schedules
• add an opt-in pprof listener on a separate address for CPU, heap, allocation, goroutine, block, mutex, and trace collection
• document local load-test and profiling workflows, starting with a 1 RPS smoke test and incremental ramp

Why

AgentBakerSvc timed out during concentrated traffic bursts. This tooling provides a reproducible way to measure throughput, latency, overload behavior, and Go runtime resource consumption before tuning concurrency, resources, HPA, and minimum replicas.

completed counts every request attempt that reached a terminal outcome, including client timeouts. Use successful and successfulRps for delivered capacity.

Local observations

• 2 RPS for 10 seconds completed with 20 successful responses and no overloads or timeouts, both with and without pprof
• 10 RPS for 30 seconds produced 28 successes, 74 HTTP 503 responses, and 198 client timeouts
• all overload responses included Retry-After: 3; scheduling delay stayed below about 2 ms, confirming the generator kept pace
• these local results demonstrate the 30-request concurrency limiter and overload behavior; they are not universal production capacity measurements

Safety

• pprof is disabled by default
• pprof uses a separate listener and is not registered on the API router
• the built-in request fixture is synthetic
• profiling is intended only for isolated test environments
• generated result artifacts are intentionally excluded from the change

Validation

• GOPROXY=direct go test -count=1 ./hack/agentbakersvc-load ./apiserver ./cmd/starter
• GOPROXY=direct go vet ./hack/agentbakersvc-load ./apiserver ./cmd/starter
• git diff --check

[github-actions]

Windows Unit Test Results

  3 files   17 suites   1m 14s ⏱️
533 tests 533 ✅ 0 💤 0 ❌
536 runs  536 ✅ 0 💤 0 ❌

Results for commit 10dd1a1.

♻️ This comment has been updated with latest results.

[github-actions]

Package	Line Rate	Health
github.com/Azure/agentbaker/aks-node-controller	72%	➖
github.com/Azure/agentbaker/aks-node-controller/common	100%	✔
github.com/Azure/agentbaker/aks-node-controller/helpers	71%	➖
github.com/Azure/agentbaker/aks-node-controller/parser	89%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/gpu	100%	✔
github.com/Azure/agentbaker/aks-node-controller/pkg/nodeconfigutils	67%	➖
github.com/Azure/agentbaker/aks-node-controller/utils	0%	❌
github.com/Azure/agentbaker/apiserver	40%	❌
github.com/Azure/agentbaker/cmd	0%	❌
github.com/Azure/agentbaker/cmd/starter	0%	❌
github.com/Azure/agentbaker/fuzz/api	0%	❌
github.com/Azure/agentbaker/hack/agentbakersvc-load	70%	➖
github.com/Azure/agentbaker/hotfix/render-nodecustomdata	0%	❌
github.com/Azure/agentbaker/pkg/agent	76%	➖
github.com/Azure/agentbaker/pkg/agent/datamodel	76%	➖
github.com/Azure/agentbaker/pkg/agent/toggles	0%	❌
github.com/Azure/agentbaker/pkg/vhdbuilder/datamodel	88%	✔
Summary	74% (6633 / 9002)	➖

[github-actions]

Package	Line Rate	Health
staging_cse_windows Package 1	65%	➖
debug	0%	❌
provisioningscripts	2%	❌
parts_windows Package 1	76%	➖
test	0%	❌
windows	21%	❌
Summary	37% (1598 / 6790)	❌

[github-actions]

Package	Line Rate	Health
shellspec spec	21%	❌
Summary	21% (3032 / 14138)	❌