Skip to content

Add s3 multi-agent aggregation example - #8424

Open
kashif wants to merge 17 commits into
huggingface:mainfrom
kashif:examples-s3-aggregation
Open

kashif wants to merge 17 commits into
huggingface:mainfrom
kashif:examples-s3-aggregation

Conversation

@kashif

@kashif kashif commented Nov 17, 2026 •

Copy link
Copy Markdown
Collaborator

What does this PR do?

Adds examples/async_grpo_s3_aggregation, a small replication of the information-aggregation task from s3. Four Qwen3-0.6B agents message each other to pool private numbers. One AsyncGRPOTrainer rollout is a whole team episode, from a custom AsyncRolloutLoop.

After 50 updates of 128 teams, held-out team success goes from 9.4% to 49.5%. The README has the run commands and results.

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • Did you read the contributor guideline, Pull Request section?
  • Was this discussed/approved via a GitHub issue? Please add a link to it if that's the case.
  • Did you make sure to update the documentation with your changes?
  • Did you write any new necessary tests?

AI writing disclosure

  • No AI usage: the PR was written entirely by a human.
  • AI-assisted: some parts were suggested or improved by AI, but the PR was written and reviewed by a human.
  • AI-generated: the PR was mostly or fully generated by an AI tool.

Who can review?

Anyone in the community is free to review the PR once the tests have passed.


Note

Low Risk
Adds an isolated examples package and documentation only; training logic lives in example subclasses and does not modify library training code paths.

Overview
Adds examples/async_grpo_s3_aggregation, an end-to-end replication of the s3 information-aggregation task: four agents with private numbers coordinate via tooling on a message board, and one AsyncGRPO rollout is a full team episode.

The example includes a token-level harness (harness/task.py, harness/team.py), training via TeamRolloutLoop / TeamTrainer (whole-team generate_one, length-weighted compute_advantages, loss scaling so each team counts equally per the paper’s Eq. 3), plus evaluate.py and a README with commands and reported results (~55% held-out success after 50 updates).

Documentation is updated in async_grpo_trainer.md (multi-agent pointer), example_overview.md (index row), and paper_index.md (s3 section with run command). No core TRL trainer API changes—only the new example and docs.

Reviewed by Cursor Bugbot for commit 072879f. Bugbot is set up for automated code reviews on this repo. Configure here.

@bot-ci-comment

bot-ci-comment Bot commented Nov 17, 2026 •

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Pages changed in this PR:

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

Comment thread docs/source/example_overview.md Resolved

@albertvillanova albertvillanova left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kashif!

Before approving:

  • Please add an s3 subsection to docs/source/paper_index.md pointing to this example, as the TailSFT entry does for examples/tail_sft_gsm8k.
  • The failing test jobs are #8429 (exit 137), fixed on main by #8430: merging main should turn CI green.

@qgallouedec qgallouedec left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really cool. I haven't run myself, but very relevant considering our current effort around swarms

@cursor cursor Bot left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Stale Bugbot comment from a previous run.

@edbeeching edbeeching mentioned this pull request Nov 19, 2026
4 of 8 tasks
@qgallouedec

Copy link
Copy Markdown
Member

Thanks Kashif, great fit for an example. A few things

  1. The results come from an earlier version of the code (README). Could you rerun the final script and share the checkpoint and Trackio run? Or an image plot. The paper index entry also describes the old batching.
  2. Examples aren't tested, so the _score_group and compute_loss overrides will break silently. Team weighting fits in compute_advantages. I'd drop the kept all-equal groups and the loss override, and list them as deviations.
  3. I'd drop analyze_episodes.py, transcript.py and test_harness.py. No other example has files like these, and CI never runs the test.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 039f2ec. Configure here.

@sergiopaniego

Copy link
Copy Markdown
Member

Thanks @kashif, great to have the first multi-agent example in TRL! A few things:

  1. --teams-per-update is approximate. gradient_accumulation_steps = teams_per_update // 2 assumes two teams per 40,960-token micro-batch, but a team forwards about 17k tokens, so it's ~2.4 teams per micro-batch (~150 per update) and varies with team length. The README and help text should say so.

  2. Script header. The script imports the local harness package, so uv run on it alone fails. I'd drop the PEP 723 header. Also, --report-to defaults to none; async_grpo_math defaults to trackio.

  3. Diagram. This is the first multi-agent example and the README only shows it in prose. A picture of "one rollout = one team episode" would help a lot.

  4. Docs link. "Customizing the rollout loop" in async_grpo_trainer.md covers overriding generate_one and compute_advantages. Worth linking this example there.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants