fix: preserve leading content tabs in list fences

[jovial-liu]

Marked version: master at 7e8754d60b3c37be30aa17106ada623d1f0cf9db

Markdown flavor: CommonMark / GitHub Flavored Markdown

Description

Addresses the leading content-tab case reported in #4050. Given "- ```\n \tfoo\n ```\n", the current list parser expands the content tab to two spaces before the fenced-code tokenizer receives it. This changes the rendered code and what a user copies from it. The fenced-code text should retain "\tfoo".

Keep the existing normalized list text and structural parsing, but track the spans created by original tabs. Restore a tab in built-in fenced-code text only when its complete expansion survives list and fence indentation removal. Partly consumed indentation tabs remain spaces. Nested lists carry the spans through their own indentation removal; recursive calls scope and restore the context, including same-source reentry and exceptions. Custom tokenizer output is preserved.

This does not attempt to resolve every tab-related parsing case in #4050. Existing tilde-fence indentation compensation, ordered-list interruption, and indented-code/blank-line edge cases are outside this change.

Validation

• Full local npm test passes on Node 26.8.1: 1,911 spec executions and 253 unit tests, including builds, module/type checks and ESLint. Unmodified master passes 1,907 specs and 199 units.
• Adds 54 unit cases: on unmodified master, 34 fail and 20 controls pass; all 54 pass with the patch. Covers ordered/nested/multiple fences, backticks/tilde, partly consumed tabs, task lists, CRLF, normalized raw text, custom tokenizers, reentry and exception reuse.
• Adds two exact HTML fixture scenarios, contributing four spec executions. Both fail on unmodified master and match byte-for-byte with the patch.
• A separate 31-case public API probe passes. A generated 1,172-observation comparison of code contents against commonmark 0.31.2 finds 972 newly matching observations, preserves all 136 previously matching observations, and retains 64 observations outside the addressed behavior. This is a code-content comparison, not a claim of full CommonMark conformance.

Performance cost

A local parse-only microbenchmark (one fixed input per workload, 15 warmups, 9 samples of 35 iterations, median time) measured tab-free lists at 0.316 → 0.327 ms (+3.3%, overlapping sample ranges), sparse-tab fences at 0.172 → 0.185 ms (+7.7%), dense-tab fences at 0.180 → 0.224 ms (+24.7%), and other tab-containing lists at 0.307 → 0.315 ms (+2.5%). These are small local workloads, not whole-library performance results; the dense-tab overhead is a tradeoff for preserving code content.

Implementation and validation used OpenAI Codex.

Contributor

• Test(s) exist to ensure functionality and minimize regression.
• no tests required for this PR.
• If submitting new feature, it has been documented in the appropriate places. (Bug fix.)

Committer

• CI is green (no forced merge required).
• Squash and Merge PR following conventional commit guidelines.

[vercel]

@jovial-liu is attempting to deploy a commit to the MarkedJS Team on Vercel.

A member of the Team first needs to authorize it.

[UziTech]

We shouldn't be calling lexer functions from a tokenizer (other than inlineTokens and blockTokens) since an extension can supply the lexer in a provideLexer hook. If you need something from the lexer it should be state or passed in like maskedSrc in emStrong.

[UziTech]

This is a breaking change if it updates a public function signature

[UziTech]

Don't underscore prefix functions. If it is public just remove the underscore. If it is private prefix with #.

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
marked-website	Ready	Preview	Oct 13, 2026 5:53pm UTC

[jovial-liu]

Thanks for the review. I've pushed a follow-up to address the three points. `blockTokens` is back to its original public signature. The tab ranges are now passed through lexer state when processing list items, so `Tokenizer` no longer calls `_getTabRanges` (or any new lexer method). I also removed that underscored method and moved the scoped tokenization logic into a private `#blockTokens` method. The previous state is restored in `finally`, including on reentrant calls. The existing regression cases are still in place. I haven't been able to run the updated PR's GitHub Actions checks yet—the workflow currently shows `action_required`—so I don't want to claim CI is green. Let me know if you'd rather see the state handled differently.

…

On Fri, 09 Oct 2026 00:14:15 -0700, Tony Brix ***@***.***> wrote: @UziTech requested changes on this pull request. In [src/Tokenizer.ts](#4129 (comment)): > @@ -157,7 +161,12 @@ export class _Tokenizer<ParserOutput = string, RendererOutput = string> { const cap = this.rules.block.fences.exec(src); if (cap) { const raw = cap[0]; - const text = indentCodeCompensation(raw, cap[3] || '', this.rules); + const sourceRanges = this.lexer?._getTabRanges?.(src); We shouldn't be calling lexer functions from a tokenizer (other than inlineTokens and blockTokens) since an extension can supply the lexer in a provideLexer hook. If you need something from the lexer it should be state or passed in like maskedSrc in emStrong. In [src/Lexer.ts](#4129 (comment)): > @@ -113,16 +150,22 @@ export class _Lexer<ParserOutput = string, RendererOutput = string> { /** * Lexing */ - blockTokens(src: string, tokens?: Token[], lastParagraphClipped?: boolean): Token[]; - blockTokens(src: string, tokens?: TokensList, lastParagraphClipped?: boolean): TokensList; - blockTokens(src: string, tokens: Token[] = [], lastParagraphClipped = false) { + blockTokens(src: string, tokens?: Token[], lastParagraphClipped?: boolean, tabRanges?: readonly TabRange[]): Token[]; This is a breaking change if it updates a public function signature In [src/Lexer.ts](#4129 (comment)): > @@ -28,6 +29,42 @@ export class _Lexer<ParserOutput = string, RendererOutput = string> { public inlineQueue: { src: string, tokens: Token[] }[]; private tokenizer: _Tokenizer<ParserOutput, RendererOutput>; + private tabSource?: { length: number, remaining: string, ranges: readonly TabRange[], offset: number }; + private enteringTabScope = false; + + /** Return a view, so later code blocks do not copy all remaining ranges. */ + _getTabRanges(src: string): { ranges: readonly TabRange[], offset: number } | undefined { Don't underscore prefix functions. If it is public just remove the underscore. If it is private prefix with #. — Reply to this email directly, [view it on GitHub](#4129?email_source=notifications&email_token=BL7NEWSIHOFIWL6WC2FQ4RL5TCF4PA5CNFSNUABKM5UWIORPF5TWS5BNNB2WEL2QOVWGYUTFOF2WK43UKJSXM2LFO4XTKNBWGY4DSNZSGMY2M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#pullrequestreview-5466897231), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BL7NEWTDJCU6IP3KP2MU6HT5TCF4PAVCNFSNUABDKJSXA33TNF2G64TZHMZDAOJWGU3TSO2JONZXKZJ3GU3TOMJVGIZTIMJQUF3AE). You are receiving this because you were mentioned.