Support for JPEG XL (JXL) images

[winscripter]

Prerequisites

• I have written a descriptive pull-request title
• I have verified that there are no overlapping pull-requests open
• I have verified that I am following the existing coding patterns and practice as demonstrated in the repository. These follow strict Stylecop rules 👮.
• I have provided test coverage for my change (where applicable)

Description

This is a work-in-progress PR whose goal is to introduce decoding and encoding of JPEG XL (*.jxl) images.

Reference software
I use libjxl as reference. See https://github.com/libjxl/libjxl.

Performance
I will begin by applying light optimizations as I implement parts of the JPEG XL codec. Once the codec seems complete enough to handle decoding and encoding of JPEG XL images, I will apply heavier optimizations. Examples include but are not limited to stack allocation, array pooling, and SIMD.

Implementations
The JPEG XL codec lives under src/ImageSharp/Formats/Jxl.

Testing
I will start adding tests whenever the codec is complete enough to handle decoding of JPEG XL images.

Additionally, JPEG XL reference software, libjxl, contains its own tests too, which I might also implement without modification.

Progress

• 🔴 - work didn't start yet
• 🟠 - exists, but incomplete
• 🟡 - either decoding side, encoding side, or tests are complete, but not the other
• 🟢 - complete

---

• 🟡 AC strategy

• 🟢 AC strategy image/row

• 🟢 AC context

• 🔴 AC strategy tests

• 🟢 Adaptive Quantization (encoder)

• 🟠 ANS Entropy

• 🟢 ANS Entropy: Common

• 🟢 ANS Entropy: Common (Tests)

• 🟠 ANS Entropy: Decoder (symbol reader is incomplete)

• 🟠 ANS Entropy: Encoder (SIMD bit-cost calculation only)

• 🔴 ANS Entropy: Tests

• 🟢 Alpha Blending

• 🟡 Bit I/O

• 🟢 Bit I/O: Bit reader

• 🟢 Bit I/O: Bit writer

• 🔴 Bit I/O: tests

• 🟢 Box Content Decoder

• 🟢 Box Content Decoder: Uncompressed boxes

• 🟢 Box Content Decoder: Brotli-compressed boxes

• 🟢 Butteraugli

• 🟢 Butteraugli: Shared methods

• 🟢 Butteraugli: Abstractions

• 🟢 Butteraugli: Default comparator

• 🟢 Butteraugli: Encoder comparator

• 🟡 Cache

• 🟠 Cache: Decoder

• 🔴 Cache: Encoder

• 🟠 Chroma From Luma

• 🟢 Chroma From Luma Abstractions

• 🔴 Chroma From Luma Encoder

• 🟡 Coefficient Order

• 🟢 Coefficient Order: Forward

• 🟢 Coefficient Order: Main

• 🟢 Coefficient Order: Encoder

• 🔴 Coefficient Order: Tests

• 🟢 Compressed DC

• 🟢 Context Map

• 🟢 Context Map: Abstractions

• 🟢 Context Map: Decoder

• 🟢 Context Map: Encoder

• 🟡 Convolution

• 🟢 Convolution: Symmetric

• 🟢 Convolution: Separable

• 🟢 Convolution: Slow

• 🟢 Convolution: SIMD

• 🔴 Convolution: Separable5 Encoder

• 🔴 Convolution: Tests

• 🟠 Decoder: Frame

• 🟢 Decoder: Group

• 🟢 Decoder: Group Border

• 🟠 Decoder: Main

• 🟢 Decoder: Main: Codestream parser

• 🟢 Decoder: Main: Container format parser

• 🟡 Discrete Cosine Transform

• 🟢 Discrete Cosine Transform: DCT scales

• 🟢 Discrete Cosine Transform: Block data wrapper

• 🟠 Discrete Cosine Transform: Block-based

• 🟢 Discrete Cosine Transform: Slow DCT for reference in tests

• 🔴 Discrete Cosine Transform: Tests

• 🟢 Encoder dot detection

• 🟢 Encoder dot dictionary

• 🟢 Entropy coding

• 🟢 Encoder entropy coding

• 🟢 Fields

• 🟢 Fields: Visitor abstractions

• 🟢 Fields: Parser

• 🟢 Fields: Writer

• 🔴 Frame Encoder

• 🟢 Gaborish

• 🟢 Gaborish Encoder

• 🟢 Gaborish Tests

• 🔴 Group Encoder

• 🔴 Heuristics Encoder

• 🟢 Image Bundle

• 🟢 Image Bundle: Decoder/Common

• 🟢 Image Bundle: Encoder

• 🟡 LZ77 compression

• 🟢 LZ77: Fast Lossless Encoder

• 🔴 LZ77: Standard Encoder

• 🟢 Huffman compression

• 🟢 Huffman compression: Shared

• 🟢 Huffman compression: Decoder

• 🟢 Huffman compression: Encoder

• 🟡 Modular

• 🟢 Modular: Transforms

• 🟢 Modular: Transforms: Palette (Inverse)

• 🟢 Modular: Transforms: Palette (Forward)

• 🟢 Modular: Transforms: RCT (Inverse)

• 🟢 Modular: Transforms: RCT (Forward)

• 🟢 Modular: Transforms: Squeeze (Inverse)

• 🟢 Modular: Transforms: Squeeze (Forward)

• 🟢 Modular: Encoding

• 🟢 Modular: Encoding: Context Prediction

• 🟢 Modular: Encoding: MA decoder

• 🟢 Modular: Encoding: MA encoder

• 🟢 Modular: Encoding: Tree Samples

• 🟢 Modular: Encoding: Encoding decoder

• 🟢 Modular: Encoding: Encoding encoder

• 🔴 Modular: Decoder

• 🔴 Modular: Encoder

• 🟢 Modular: Encoder SIMD

• 🔴 Modular: Tests

• 🟢 Patch Dictionary: Decoder

• 🔴 Patch Dictionary: Encoder

• 🟠 Passes State: Decoder

• 🔴 Passes State: Encoder

• 🟢 Passes State: Shared

• 🔴 Encoder Main

• 🔴 Encoder Main

• 🔴 Encoder Internal

• 🔴 Encoder Tests

• 🟢 Encoder: Linear Algebra

• 🟢 Encoder: Linear Algebra Tests

• 🟢 Image Operations

• 🟢 Image Operations

• 🟢 Image Operations: Tests

• 🟢 Common I/O: Frame Header

• 🟢 Common I/O: Metadata

• 🟢 Common I/O: Container format

• 🟠 JPEG to JPEG XL lossless compression

• 🟢 JPEG to JPEG XL lossless compression: JPEG parser/writer

• 🟠 JPEG to JPEG XL lossless compression (decoder)

• 🔴 JPEG to JPEG XL lossless compression (encoder)

• 🟡 Splines

• 🟢 Splines

• 🔴 Splines Tests

• 🟢 Quantizer

• 🟢 Dequantizer matrices

• 🟢 Quantizer encoding

• 🟢 Quantizer weights

• 🟡 Noise

• 🟢 Noise: Shared

• 🟢 Noise: Decoder

• 🟠 Noise: Encoder

• 🟢 Noise: Simulation of Photon Noise

• 🟢 Noise: Simulation of Photon Noise (Tests)

• 🔴 Noise Tests

• 🔴 JPEG XL Testing Tools

• 🟠 Render Pipeline

• 🔴 Render Pipeline: Main

• 🔴 Render Pipeline: Low Memory Render Pipeline

• 🟢 Render Pipeline: Stages Abstractions

• 🔴 Render Pipeline: Stages: Blending

• 🟢 Render Pipeline: Stages: Chroma Upsampling

• 🔴 Render Pipeline: Stages: CMS

• 🟢 Render Pipeline: Stages: EPF

• 🟢 Render Pipeline: Stages: From Linear

• 🟢 Render Pipeline: Stages: Gaborish

• 🟢 Render Pipeline: Stages: Noise

• 🟢 Render Pipeline: Stages: Patches

• 🟢 Render Pipeline: Stages: Splines

• 🟢 Render Pipeline: Stages: Spot color

• 🟢 Render Pipeline: Stages: To Linear

• 🟢 Render Pipeline: Stages: Tone mapping

• 🔴 Render Pipeline: Stages: Upsampling

• 🟢 Render Pipeline: Stages: Write to Output

• 🔴 Render Pipeline: Stages: XYB

• 🟢 Render Pipeline: Stages: Y'Cb'Cr -> RGB

• 🟡 Color Management System (CMS)

• 🟢 CMS: Transfer Functions

• 🟢 CMS: Abstractions/Color Encoding

• 🟢 CMS: Tone Mapping

• 🔴 CMS: Interface

Other completed things:

• 🟢 Frame Dimensions
• 🟢 Field Encodings
• 🟢 Gamma correction (tests, encoder)
• 🟢 Shared math functions
• 🟢 Lehmer Codes
• 🟢 Lehmer Codes: Tests
• 🟢 Luminance helper
• 🟢 Opsin parameters
• 🟢 Passes shared state
• 🟢 SIMD utilities
• 🟢 TOC
• 🟢 XorShift128Plus
• 🟢 XorShift128Plus Tests

[CLAassistant]

All committers have signed the CLA.

[winscripter]

While I'm working on this, I'd like to note something important.

Libjxl is licensed under the BSD 3-Clause license, and since I'm using libjxl code as reference, that means the license must be included.

I'm not really sure what would be the proper way to include the license. I might place the LICENSE.txt file in the Jxl folder or add a README linking to the libjxl repo.

[gfoidl]

This is selecting elements from table as given by indices, so kind of shuffling.
Could then this be written as shuffle which is supported on Vector128, et. al.?

[winscripter]

I wasn't able to use the gather/shuffle on the v128/v256/v512, because it appears that such methods return a Vector<byte> when the input is Vector<byte>, we need a Vector<int> as the resulting vector.

[gfoidl]

Hm, just reading the code and seeing that in L45 int i = -2 this makes me feel it crashes here w/ a index out of range exception. Can you add an assert or comment why it works? I.e. that Debug.Assert(xStart >= 2); holds, etc.

[winscripter]

I'll explain this for row0 but it is the same idea for every other variable accessed like this.

On L32 we can see that row0 is defined as such:

Span<float> row0 = this.GetInputRow(inputRows, c, -2);

which basically says, get a row from the input rows at channel c for row offset -2.

Let's take a look at the GetInputRow method.

public Span<float> GetInputRow(Buffer2D<Memory<float>> inputRows, int c, int offset)
    => inputRows[c, this.Settings.BorderY + offset].Span[RenderPipelineXOffset..];

This returns a Memory<float> stored for channel c at offset this.Settings.BorderY + offset. In this case, this.Settings.BorderY would be equal to 2, because of this constructor in the ConvolveNoiseStage:

public ConvolveNoiseStage(Configuration configuration, int firstC)
    : base(configuration)
{
    this.Settings = RenderPipelineStageConfiguration.CreateSymmetricBorderOnly(2);
    this.firstC = firstC;
}

Specifically, this line:

this.Settings = RenderPipelineStageConfiguration.CreateSymmetricBorderOnly(2);

That means, create RenderPipelineStageConfiguration whose ShiftX, ShiftY = 0 and BorderX, BorderY = input parameter (2). So BorderY=2.

This gives us a Memory<float>, but we also get a Span to it, and:

.Span[RenderPipelineXOffset..]

we advance it by RenderPipelineStageXOffset.

That means it must start at 32 which is what the constant equals to.

Now, let's take a look at reference software, which is in C++. Let's take a look at their implementation of this GetInputRow method.

Note. RowInfo = std::vector<std::vector<float*>>.

// Returns a pointer to the input row of channel `c` with offset `y`.
// `y` must be in [-settings_.border_y, settings_.border_y]. `c` must be such
// that `GetChannelMode(c) != kIgnored`. The returned pointer points to the
// offset-ed row (i.e. kRenderPipelineXOffset has been applied).
float* GetInputRow(const RowInfo& input_rows, size_t c, int offset) const {
  JXL_DASSERT(GetChannelMode(c) != RenderPipelineChannelMode::kIgnored);
  JXL_DASSERT(-offset <= static_cast<int>(settings_.border_y));
  JXL_DASSERT(offset <= static_cast<int>(settings_.border_y));
  return input_rows[c][settings_.border_y + offset] + kRenderPipelineXOffset;
}

When we simplify it:

float* GetInputRow(const RowInfo& input_rows, size_t c, int offset) const {
  return input_rows[c][settings_.border_y + offset] + kRenderPipelineXOffset;
}

It's the same idea... or is it? That's because this C++ method returns a pointer to a float, while a Span would have bounds checks against values below 0. In this C++ version, negative accesses are fine here because the offset is kRenderPipelineXOffset (= 32) and the lowest offset would be 30, so it wouldn't lead to program termination. We should not assume that xStart >= 2.

That means we need to perhaps introduce a new method like this:

public Span<float> GetInputRow(Buffer2D<Memory<float>> inputRows, int c, int offset, int spanOffset)
    => inputRows[c, this.Settings.BorderY + offset].Span[(spanOffset + RenderPipelineXOffset)..];

Then we have to pass -2 as the spanOffset and pass +2 to every Span access offset, meaning the base offset is now 30, the 2 + -2 access would just be 0 and the 2+3 access would be 5, giving us the access to the 35th offset.

With that being said, we'd have to update this:

others += Vector.Create<float>(row0[(x + i)..]);

to:

others += Vector.Create<float>(row0[(x + 2 + i)..]);

Also. xStart cannot be negative.

[gfoidl]

Thanks for the explanation.

[gfoidl]

Use an abstract class here?
Makes it easier for the JIT to perform de-abstraction. Interfaces are harder (as a type can have multiple interface, but exactly one base class (even when it's the object type)).

[winscripter]

Unfortunately the Create method is a static abstract member, it seems that abstract members cannot be static.

[gfoidl]

Please have a unit test for this with different types for T and see what happens 😉.

[gfoidl]

Why suppress and don't fix?

[gfoidl]

Please have a look at all of these loops, I won't comment them to avoid some noise.
They may work by accident, as the input is won't make it happen, but this is a clear bug.

[winscripter]

libjxl uses SIMD through its Google Highway library, which has an API that's extremely similar to .NET. Interestingly, the reference implementation itself contains loops that advance by the SIMD lane count without explicitly handling a potentially incomplete final vector.

For example, this is the C++ reference of the highlighted function:

for (size_t i = 0; i < a.counts.size(); i += Lanes(di)) {
    const auto counts = LoadU(di, &a.counts[i]);
    const bool nz = !AllTrue(di, Eq(counts, kZero));
    total = Add(total, counts);
    if (nz) nz_pos = i;
  }

• a.counts.size() - .counts is a std::vector<T>, so .size() is just number of items in it -> countsSpan.Length
• Lanes(di) - di is the definition for integer vector types, so when it's used it refers to Vector<int>. Lanes() is the number of items a CPU can process in the vector of that type at once -> Vector<int>.Count

There is potential for bug here too, but it seems like that was expected.

I would still apply these fixes, but there are so many instances of this pattern that it's easy to overlook some when fixing them individually.

[gfoidl]

Can this be an inline array? The size is known at compile time, and it's not too big.
So maybe something like

private InlineArray... _contextMap;
public Span<byte> ContextMap => _contextMap;

[JimBobSquarePants]

@winscripter I've finished AVIF. I should be able to switch focus and help sometime next week.

[winscripter]

@JimBobSquarePants Good job! 🎉

For JPEG XL, I've done an incredibly substantial amount of work. Out of 115866 LoC in reference software, only 43966 lines remain to be ported (including unit tests; without unit tests, it's ~18k). Currently in progress of porting simple functions and working on LZ77/AC Strategy encoder.

Your help would be greatly appreciated!

[JimBobSquarePants]

@JimBobSquarePants Good job! 🎉

For JPEG XL, I've done an incredibly substantial amount of work. Out of 115866 LoC in reference software, only 43966 lines remain to be ported (including unit tests; without unit tests, it's ~18k). Currently in progress of porting simple functions and working on LZ77/AC Strategy encoder.

Your help would be greatly appreciated!

You’re a legend!!