[dependabot]

Bumps huggingface-hub from 2.1.1 to 2.3.0.

Release notes

Sourced from huggingface-hub's releases.

[v2.3.0] Bucket live-follow, faster HfFileSystem listings and more

🔄 Keep Bucket listings fresh with live_follow=True

HfFileSystem caches directory listings, so files added to a Bucket by another process did not show up until you called invalidate_cache(). With live_follow=True, HfFileSystem opens one background connection per listed Bucket to receive its file changes from the Hub. When a change arrives, it removes the cached listings it affects.

>>> from huggingface_hub import HfFileSystem
>>> hffs = HfFileSystem(live_follow=True)
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # starts following that bucket
>>> # ... another process uploads a file to the bucket ...
>>> hffs.ls("buckets/my-username/my-bucket/data", detail=False)  # refreshed as changes arrive
['buckets/my-username/my-bucket/data/train.csv', 'buckets/my-username/my-bucket/data/new-file.csv']

It also works through fsspec with fsspec.filesystem("hf", live_follow=True), and through the storage_options argument of libraries such as pandas and pyarrow. The option is off by default because it keeps a connection open for each followed Bucket. It only affects listings and metadata (ls, info, exists, glob, ...), because reading a file always fetches its content from the Hub. It is not available for model, dataset and Space repos: use refresh=True to see a new commit.

• [HfFileSystem] Add live-follow for Buckets by @​lhoestq in #5096

🚀 HfFileSystem lists large repos in parallel

Recursive listings such as find and glob used to be slow on large repos. The Hub returns a repo's file tree 1000 entries at a time, and each page can only be requested after the previous one has arrived. Listing datasets/allenai/objaverse (~800k files) took ~800 requests in a row, about 13 minutes.

HfFileSystem now splits recursive listings by sub-directory and lists them in parallel, with almost the same number of requests. A sub-tree that fits in a single page is never split. maxdepth is now applied during the listing, so the Hub only returns the levels you ask for.

datasets/allenai/objaverse	before	after
find(), 798,924 files	801 requests, 776 s	807 requests, 19.7 s
find(maxdepth=2), 328 entries	801 requests, 776 s	6 requests, 1.5 s

>>> from huggingface_hub import hffs
>>> files = hffs.glob("datasets/allenai/objaverse/**/*.glb", maxdepth=3)

The new max_workers argument of HfFileSystem sets the number of concurrent requests. It defaults to 16. Set it to 1 to list sequentially like before.

• [HfFileSystem] List large repos in parallel by @​lhoestq in #5153

📎 Attach images and videos to discussions and PRs

The new HfApi.upload_attachment(path) uploads an image, video or audio file and returns its URL, which you can embed in a discussion, PR description or comment. Before, this required calling an undocumented Hub endpoint by hand.

>>> from huggingface_hub import HfApi
>>> api = HfApi()
>>> url = api.upload_attachment("screenshot.png")
>>> api.comment_discussion("username/repo-name", 5, f"Here is the fix:\n\n![screenshot](https://github.com/huggingface/huggingface_hub/blob/HEAD/{url})")

... (truncated)

Commits

• 587e09a Release: v2.3.0
• ac4a5f0 Release: v2.3.0.rc1
• 64e9850 [HfFileSystem] Fix glob crash when the parent listing is outdated (#5237)
• b1066db Release: v2.3.0.rc0
• 0c7fd06 [HfApi] Support gated repos hiding requester username/email (#5047)
• c629fa3 [Jobs] Fix hf jobs run losing logs and ignoring Ctrl+C (#5162)
• 7858680 [HfFileSystem] List large repos in parallel (#5153)
• 75fbd6d fix(kernels): allow uploads and commits to kernel repos (#5233)
• e1f595d [CLI] Show the Job's scheduling status in hf jobs run (#5163)
• d03a05c Bump the actions group with 3 updates (#5228)
• Additional commits viewable in compare view