[DISCUSS] Native Iceberg REST Catalog / S3 Tables support in Ozone S3 Gateway

[MahmoudHassanen99]

Motivation
Storage engines are converging with table metadata seen in AWS S3 Tables, RustFS's embedded Iceberg REST catalog, and recent MinIO lakehouse sessions. Currently, running Iceberg on Ozone requires an external catalog (Apache Polaris, HMS) and an external RDBMS. For Ozone-centric lakehouses, this adds unnecessary operational overhead and an extra failure domain.

The Proposal
Implement an embedded, lightweight Iceberg REST Catalog directly inside Ozone's S3 Gateway (S3G):

• Expose standard Iceberg REST endpoints (/v1/{prefix}/...) on S3G alongside the S3 API.
• Leverage Ozone Manager (Ratis + RocksDB atomic batches) and S3 conditional writes for atomic metadata.json pointer swaps.
• Provide a single endpoint for engines (Trino, Spark, PyIceberg) covering metadata operations and data access, protected under Ozone's existing Ranger security context.

Questions

• Is the community open to an optional, embedded Iceberg REST catalog in S3G?

[MahmoudHassanen99]

Hi @adoroszlai and @ayushtkn,

Hope you are doing well
Following up on our previous PR collaboration, I wanted to ping you for a quick architectural gut-check on this idea.

With the recent industry shift toward the S3 Tables pattern (converging object storage with an embedded Iceberg REST catalog, as seen in AWS S3 Tables and RustFS), I am exploring prototyping a lightweight, embedded Iceberg REST catalog directly inside Ozone's S3 Gateway (S3G) leveraging Ozone's S3 conditional writes (HDDS-13919) for atomic commits.

Before spending time on a formal design doc or PoC PR, I would appreciate your early thoughts:

1. Does housing an optional Iceberg REST endpoint inside S3G align with Ozone's roadmap, or does the PMC prefer keeping Ozone strictly decoupled and delegating catalogs entirely to external projects like Apache Polaris?
2. If this aligns with Ozone's direction, would you recommend keeping this strictly inside S3G, or would you see value in backing it with an internal OM catalog table in the long run?

Thanks for your time and feedback

[chungen0126]

Thanks for bringing this up, @MahmoudHassanen99 .

In my view, embedding an Iceberg catalog isn't really the direction Ozone is heading right now. Maintaining a full-blown catalog is a ton of work and feels outside Ozone's current focus as a storage engine.

Our efforts are spending making sure Ozone integrates smoothly with dedicated external catalogs like Polaris.