datafusion-variant

This crate provides a Variant SQL type and functions for querying semi-structured data using Apache DataFusion, and reading and writing such data to Apache Parquet files.

Note

This crate is under active development and aims to match Apache Spark Variant behavior. Compatibility is incomplete, and APIs may change. See the roadmap for more details.

Please see the main site for documentation, and docs.rs for the Rust API reference.

For example, to read fields from JSON documents:

SELECT event:service::VARCHAR, event:status::BIGINT, event:tags[0]::VARCHAR
FROM (VALUES
    (parse_json('{"service":"api","status":200,"tags":["cached"]}')),
    (parse_json('{"service":"web","status":504,"error":"timeout"}'))
) AS events(event);

+----------------------+---------------------+----------------------+
| events.event:service | events.event:status | events.event:tags[0] |
+----------------------+---------------------+----------------------+
| api                  | 200                 | cached               |
| web                  | 504                 | NULL                 |
+----------------------+---------------------+----------------------+

Features

• A VARIANT SQL type with Spark-style : field access, such as event:tags[0]::VARCHAR
• JSON conversion with parse_json, try_parse_json, and to_json
• Typed extraction with variant_get and try_variant_get, and casts between Variant and SQL types
• Reading and writing Variant columns in Parquet files, including shredded columns
• dfv, a SQL shell with the Variant functions preinstalled

What is Variant?

Variant is a type for storing semi-structured data, similar to JSON. Each value can hold objects, arrays, and primitive values, with different fields and types in each row. Variants are stored in an efficient binary encoding, and fields that many rows share can be "shredded" into typed Parquet columns for faster queries. See the introduction for more background.

Getting started

Add the crate to your project and call register_all on your DataFusion SessionContext to use the VARIANT type, the Variant functions, and : field access in SQL. The getting started guide walks through the setup and the bundled dfv SQL shell, and the cookbook shows common tasks: converting JSON, querying fields, and reading and writing Parquet.

Documentation

• User guide
• SQL function reference
• Rust API reference (docs.rs)
• Contributing

The documentation sources are in docs/.

License

Licensed under the Apache License, Version 2.0.