cloud: add tiered storage docs - #23328
zhaoshangzi wants to merge 55 commits into
Conversation
ti-chi-bot
Bot
added
contribution
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds TiDB Cloud Essential Private Preview documentation for Tiered Storage, including concepts, architecture, configuration, operations, limitations, FAQ content, and table-of-contents navigation. It also ignores a local QodEr settings file. ChangesTiered Storage documentation
Local tooling ignore
Estimated code review effort: 2 (Simple) | ~10 minutes Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 14
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: 8c969826-2f18-4a5e-9ea9-0868fa38a3a4
📒 Files selected for processing (5)
TOC-tidb-cloud.mdtidb-cloud/tieredstorage_concepts.mdtidb-cloud/tieredstorage_faq.mdtidb-cloud/tieredstorage_limitations.mdtidb-cloud/tieredstorage_operations.md
|
|
||
| ## 1 What is Tiered Storage | ||
|
|
||
| Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Replace etc. with explicit examples.
Vale rejects the current wording. Use a precise list without changing the technical meaning.
Suggested replacement
-Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally.
+Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage, such as S3 or OSS, keeping only metadata and on-demand cached hot data segments locally.As per path instructions, this is an exact replacement for the contiguous changed line.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. | |
| Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage, such as S3 or OSS, keeping only metadata and on-demand cached hot data segments locally. |
🧰 Tools
🪛 GitHub Check: vale
[failure] 12-12:
[vale] reported by reviewdog 🐶
[PingCAP.Latin] Use 'such as' instead of 'etc.'.
Raw Output:
{"message": "[PingCAP.Latin] Use 'such as' instead of 'etc.'.", "location": {"path": "tidb-cloud/tieredstorage_concepts.md", "range": {"start": {"line": 12, "column": 310}}}, "severity": "ERROR"}
Sources: Path instructions, Linters/SAST tools
|
|
||
| Tiered Storage is a **table-level and partition-level storage tiering capability** on TiDB Cloud Essential, designed for infrequently accessed data. Users can set a table or partition to the IA (Infrequent Access) storage class. The system automatically stores the full data in remote object storage (S3/OSS, etc.), keeping only metadata and on-demand cached hot data segments locally. | ||
|
|
||
| **In a nutshell**: An IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Remove spaces around the em dash.
Suggested replacement
-**In a nutshell**: An IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables.
+**In a nutshell**: An IA table is still a regular table from the application layer—all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables.As per path instructions, this is an exact replacement for the contiguous changed line.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| **In a nutshell**: An IA table is still a regular table from the application layer — all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. | |
| **In a nutshell**: An IA table is still a regular table from the application layer—all query, transaction, backup, and recovery semantics remain unchanged. The difference lies in cost and performance: local disk usage drops significantly, but on a cold read (when the local cache is missing), data must be fetched from remote object storage, resulting in higher latency than Standard tables. |
🧰 Tools
🪛 GitHub Check: vale
[failure] 14-14:
[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.
Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud/tieredstorage_concepts.md", "range": {"start": {"line": 14, "column": 83}}}, "severity": "ERROR"}
Sources: Path instructions, Linters/SAST tools
| - [ ] The data access frequency of the table/partition has been confirmed to be declining from a business perspective | ||
| - [ ] The table can be changed to a partitioned table, because cold/hot data separation is easier to manage with partitioned tables | ||
| - [ ] For regular table cold/hot separation, hot data accounts for less than 10% of the table | ||
| - [ ] Cold data access frequency is very low, e.g., query QPS does not exceed 10 concurrent (to avoid saturating object storage bandwidth) |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🟠 Major | ⚡ Quick win
Separate QPS, concurrency, and throughput limits.
These lines currently treat “10 concurrent” as equivalent to a QPS or throughput limit. Define each dimension independently and explain the assumptions behind any conversion between them; otherwise users cannot determine whether their workload is within the documented safety limits.
tidb-cloud/tieredstorage_concepts.md#L60-L60: replace the QPS-based wording with an explicit concurrent-request limit.tidb-cloud/tieredstorage_limitations.md#L30-L30: document the1 GiB/sthroughput limit separately from the maximum concurrent request count.
📍 Affects 2 files
tidb-cloud/tieredstorage_concepts.md#L60-L60(this comment)tidb-cloud/tieredstorage_limitations.md#L30-L30
| - **Standard layer**: Three replicas are each stored on the local disks of three TiKV nodes | ||
| - **IA layer**: Three replicas are each uploaded to object storage independently in IA format | ||
|
|
||
| Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Restrict IA SSTable writes to the documented L1+ path.
The concepts document explicitly says memtable and L0 remain local; only eligible L1+ files are opened in IA mode after flush or compaction.
Suggested replacement
-Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared.
+Each replica on each TiKV node runs its own independent LSM-tree, performing independent flush and compaction operations. Memtable and L0 writes remain on the local hot path; after flush or compaction, eligible L1+ SST files are written to S3 in IA format. The three replicas produce their own independent SST files—three separate objects in S3, not shared.As per path instructions, this is an exact replacement for the contiguous changed line.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. | |
| Each replica on each TiKV node runs its own independent LSM-tree, performing independent flush and compaction operations. Memtable and L0 writes remain on the local hot path; after flush or compaction, eligible L1+ SST files are written to S3 in IA format. The three replicas produce their own independent SST files—three separate objects in S3, not shared. |
🧰 Tools
🪛 GitHub Check: vale
[failure] 43-43:
[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.
Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud/tieredstorage_faq.md", "range": {"start": {"line": 43, "column": 291}}}, "severity": "ERROR"}
Source: Path instructions
|
|
||
| Each replica on each TiKV node runs its own independent LSM-Tree, performing independent flush and compaction operations. When a table switches to IA storage class, all subsequently generated SST files are written to S3 in IA type. The three replicas produce their own independent SST files — three separate objects in S3, not shared. | ||
|
|
||
| IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Replace the ambiguous “how many copies exist.”
Suggested replacement
-IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer.
+IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes how each node stores its own copy, not the number of copies. The Raft write and replication flow is identical to the Standard layer.As per path instructions, this is an exact replacement for the contiguous changed line.
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes "how each node stores its own copy," not "how many copies exist." The Raft write and replication flow is identical to the Standard layer. | |
| IA is a **storage format/location optimization**, not a **replica reduction mechanism**. It changes how each node stores its own copy, not the number of copies. The Raft write and replication flow is identical to the Standard layer. |
🧰 Tools
🪛 GitHub Check: vale
[warning] 45-45:
[vale] reported by reviewdog 🐶
[PingCAP.Ambiguous] Consider using a clearer word than 'many' because it may cause confusion.
Raw Output:
{"message": "[PingCAP.Ambiguous] Consider using a clearer word than 'many' because it may cause confusion.", "location": {"path": "tidb-cloud/tieredstorage_faq.md", "range": {"start": {"line": 45, "column": 147}}}, "severity": "INFO"}
Sources: Path instructions, Linters/SAST tools
ti-chi-bot
Bot
removed
the
missing-translation-status
lilin90
added
area/tidb-cloud
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
There was a problem hiding this comment.
Actionable comments posted: 7
♻️ Duplicate comments (1)
tidb-cloud/tiered-storage-concepts.md (1)
66-66:⚠️ Potential issue | 🟠 MajorKeep request concurrency and throughput as separate limits.
tidb-cloud/tiered-storage-concepts.md#L66-L66: replace the QPS wording with separate limits for no more than 10 concurrent cold-read requests and no more than 1 GiB/s aggregate throughput.tidb-cloud/tiered-storage-limitations.md#L36-L36: split the combined1 GiB/s (≤ 10 concurrent)value into separate throughput and concurrency rows.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Pro
Run ID: ac2090ff-f7bd-475c-a51f-7f1766f3ff3e
📒 Files selected for processing (6)
.gitignoreTOC-tidb-cloud-byoc.mdtidb-cloud/tiered-storage-concepts.mdtidb-cloud/tiered-storage-faq.mdtidb-cloud/tiered-storage-limitations.mdtidb-cloud/tiered-storage-operations.md
|
|
||
| ## What happens to IA tables when the object store (S3) experiences an outage? | ||
|
|
||
| IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Do not state that every IA table becomes unavailable during an object-store outage.
The concepts document says hot data can remain in the local IA cache. During an outage, uncached reads and operations requiring object storage can fail, but cached data is not necessarily unavailable.
Suggested replacement
-IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted.
+IA tables will be affected: reads for uncached data and operations requiring object storage may fail, while cached data may remain readable. If object-storage bandwidth is saturated, IA read/write performance will also be impacted.📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| IA tables will be affected and become unavailable — since all data resides remotely, read requests must fetch from S3. Additionally, if S3 bandwidth is saturated, IA read/write performance will also be impacted. | |
| IA tables will be affected: reads for uncached data and operations requiring object storage may fail, while cached data may remain readable. If object-storage bandwidth is saturated, IA read/write performance will also be impacted. |
🧰 Tools
🪛 GitHub Check: vale
[failure] 26-26:
[vale] reported by reviewdog 🐶
[PingCAP.EmDash] Don't put a space before or after a dash.
Raw Output:
{"message": "[PingCAP.EmDash] Don't put a space before or after a dash.", "location": {"path": "tidb-cloud/tiered-storage-faq.md", "range": {"start": {"line": 26, "column": 50}}}, "severity": "ERROR"}
Connor1996
left a comment
Connor1996
left a comment
There was a problem hiding this comment.
Why is the doc merged to release-8.5? Should be master
| | Range Columns partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | | ||
| | List partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | | ||
| | List Columns partitioned table | Supported | Must use `ENGINE_ATTRIBUTE` | | ||
| | Hash partitioned table | **Not supported** | — | |
There was a problem hiding this comment.
It has an ambiguity. Table-level IA is supported and applies to all partitions. TiDB rejects only partition-scoped storage-class definitions for HASH/KEY partitions.
| summary: Learn about Tiered Storage on TiDB Cloud BYOC/Premium/Essential, including its concepts, architecture, use cases, and read amplification. | ||
| --- | ||
|
|
||
| # Tiered Storage Concepts |
There was a problem hiding this comment.
| # Tiered Storage Concepts | |
| # Tiered Storage Overview |
lilin90
added
the
do-not-merge/hold
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Several moderate documentation inconsistencies and ambiguities remain unresolved.
Get a fresh assessment by requesting another Copilot review.
Review effort: Lite
Findings: 7
Open (7)
What changed in this PR
Adds private-preview Tiered Storage documentation for TiDB Cloud Premium and BYOC.
Changes:
- Adds navigation entries and five Tiered Storage guides.
- Documents configuration, monitoring, limitations, FAQs, and billing.
- Adds IA cache-level billing guidance.
| File | Summary |
|---|---|
TOC-tidb-cloud-premium.md |
Adds Tiered Storage navigation. |
TOC-tidb-cloud-byoc.md |
Adds Tiered Storage navigation. |
tidb-cloud/tiered-storage-overview.md |
Documents concepts, architecture, and scenarios. |
tidb-cloud/tiered-storage-observability.md |
Documents monitoring and metrics. |
tidb-cloud/tiered-storage-limitations.md |
Documents limitations and operational risks. |
tidb-cloud/tiered-storage-guide.md |
Documents configuration and operations. |
tidb-cloud/tiered-storage-faq.md |
Answers common Tiered Storage questions. |
tidb-cloud/tidb-cloud-billing.md |
Documents IA cache billing effects. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
|
|
||
| ## Can IA tables execute `UPDATE`/`DELETE`? | ||
|
|
||
| Yes. An `UPDATE` operation first loads the corresponding data from object storage into the IA cache, performs the modification, and writes a new SST file, the same flow as a regular `UPDATE`. Performance is affected by cold reads. |
| Only one copy is stored on Amazon S3, and all three replicas share the same object. | ||
|
|
||
| In the cloud storage engine architecture, SST/blob data files have only one copy on object storage (S3/DFS) to begin with: files are uploaded once by flush/compaction, the S3 key contains no node/replica information, and the three Raft replicas reference the same file id through the Raft-replicated ChangeSet. The three-replica mechanism applies only to Raft logs, metadata, and each node's local cache, never to the data on object storage. | ||
|
|
||
| **Cost implications**: The storage volume on S3 is always about 1x the data size (it does not multiply with the replica count). What the IA tier saves is local disk usage on each node; data durability is guaranteed by the object storage itself, independent of the replica count. |
| ### Partitioned table DDL | ||
|
|
||
| Partitioned tables **do not support** the `STORAGE_CLASS` syntactic sugar and must use `ENGINE_ATTRIBUTE`. | ||
|
|
||
| Partition attributes support three selector types (cannot be mixed) plus a table-level default: |
| - New metrics: | ||
| - `Row-based IA Storage` — The storage space of data in the IA storage class | ||
| - `Row-based Standard Storage` — Total Standard table space | ||
| - Relationship: `Row-based Storage` = `Row-based IA Storage` + `Row-based Standard Storage` |
| Since shared physical clusters have limited object storage bandwidth, IA cold storage access must comply with the following limits: | ||
|
|
||
| | Constraint dimension | Limit | Reason | | ||
| |-|-|-| | ||
| | Single SQL cold read throughput | ≤ 100 MiB/s | Prevents one query from consuming excessive bandwidth | | ||
| | Total concurrent cold read throughput | ≤ 1 GiB/s (≤ 10 concurrent) | Protects other tenants in the cluster | |
|
|
||
| - `COMPLETED_REPLICAS` increases and `LAST_UPDATE_TIME` keeps advancing: the conversion is progressing normally and the data volume is simply large. | ||
| - `DURATION` keeps growing but `COMPLETED_REPLICAS` does not increase for a long time: the conversion might be stuck because of a system exception, such as a TiKV rolling restart, temporarily insufficient resources, or short-term object storage unavailability. | ||
| - `TOTAL_REPLICAS`, `COMPLETED_REPLICAS`, `PROGRESS`, and `LAST_UPDATE_TIME` all stay `NULL`: no successful observation has been made yet. These columns are populated and cleared together, so `LAST_UPDATE_TIME` cannot tell whether observations are still being attempted. Check `DURATION` instead: it always increases while the conversion is being tracked. If these columns stay `NULL` while `DURATION` keeps growing, polling is still running but has not returned a valid observation. |
| - [ ] The data access frequency of the table/partition has been confirmed to be declining from a business perspective | ||
| - [ ] The table can be changed to a partitioned table, because cold/hot data separation is easier to manage with partitioned tables | ||
| - [ ] For regular table cold/hot separation, hot data accounts for less than 10% of the table | ||
| - [ ] Cold data access frequency is very low, e.g., query QPS does not exceed 10 concurrent (to avoid saturating object storage bandwidth) |
|
Note As confirmed by @zhaoshangzi, we'll make the tiered storage docs public on docs.pingcap.com left navigation. So I updated the TOC files via dd40f09, f0c1159. |
|
As confirmed by @zhaoshangzi, we won't show tiered storage docs in the left navigation of docs.pingcap.com. So I moved related topics to |
First-time contributors' checklist
What is changed, added or deleted? (Required)
Which TiDB version(s) do your changes apply to? (Required)
Tips for choosing the affected version(s):
By default, CHOOSE MASTER ONLY so your changes will be applied to the next TiDB major or minor releases. If your PR involves a product feature behavior change or a compatibility change, CHOOSE THE AFFECTED RELEASE BRANCH(ES) AND MASTER.
For details, see tips for choosing the affected versions.
What is the related PR or file link(s)?
AI agent involvement
Do your changes match any of the following descriptions?
Summary by CodeRabbit