Crate miniconf

Expand description

§miniconf: serialize/deserialize/access reflection for trees

miniconf enables lightweight (no_std/no alloc) serialization, deserialization, and access within a tree of heterogeneous types by keys.

§Example

See below for an example showing some of the features of the Tree* traits. See also the documentation and doctests of the TreeSchema trait for a detailed description.

Note that the example below focuses on JSON and slash-separated paths while in fact any serde backend (or dyn Any trait objects) and many different Keys/Transcode providers are supported.

use serde::{Deserialize, Serialize};
use miniconf::{SerdeError, json_core, JsonPath, ValueError, KeyError, Tree, TreeSchema, Path, Packed, Shape, leaf};

#[derive(Deserialize, Serialize, Default, Tree)]
pub struct Inner {
    a: i32,
    b: u16,
}

#[derive(Deserialize, Serialize, Default, Tree)]
pub enum Either {
    #[default]
    Bad,
    Good,
    A(i32),
    B(Inner),
    C([Inner; 2]),
}

#[derive(Tree, Default)]
pub struct Settings {
    foo: bool,
    #[tree(with=leaf)]
    enum_: Either,
    #[tree(with=leaf)]
    struct_: Inner,
    #[tree(with=leaf)]
    array: [i32; 2],
    #[tree(with=leaf)]
    option: Option<i32>,

    #[tree(skip)]
    #[allow(unused)]
    skipped: (),

    struct_tree: Inner,
    enum_tree: Either,
    array_tree: [i32; 2],
    array_tree2: [Inner; 2],
    tuple_tree: (i32, Inner),
    option_tree: Option<i32>,
    option_tree2: Option<Inner>,
    array_option_tree: [Option<Inner>; 2],
}

let mut settings = Settings::default();

// Access nodes by field name
json_core::set(&mut settings,"/foo", b"true")?;
assert_eq!(settings.foo, true);
json_core::set(&mut settings, "/enum_", br#""Good""#)?;
json_core::set(&mut settings, "/struct_", br#"{"a": 3, "b": 3}"#)?;
json_core::set(&mut settings, "/array", b"[6, 6]")?;
json_core::set(&mut settings, "/option", b"12")?;
json_core::set(&mut settings, "/option", b"null")?;

// Nodes inside containers
// ... by field name in a struct
json_core::set(&mut settings, "/struct_tree/a", b"4")?;
// ... or by index in an array
json_core::set(&mut settings, "/array_tree/0", b"7")?;
// ... or by index and then struct field name
json_core::set(&mut settings, "/array_tree2/0/a", b"11")?;
// ... or by hierarchical index
json_core::set_by_key(&mut settings, [8, 0, 1], b"8")?;
// ... or by packed index
let packed: Packed = Settings::SCHEMA.transcode([8, 1, 0]).unwrap();
assert_eq!(packed.into_lsb().get(), 0b1_1000_1_0);
json_core::set_by_key(&mut settings, packed, b"9")?;
// ... or by JSON path
json_core::set_by_key(&mut settings, &JsonPath(".array_tree2[1].b"), b"10")?;

// Hiding paths by setting an Option to `None` at runtime
assert_eq!(json_core::set(&mut settings, "/option_tree", b"13"), Err(ValueError::Absent.into()));
settings.option_tree = Some(0);
json_core::set(&mut settings, "/option_tree", b"13")?;
// Hiding a path and descending into the inner `Tree`
settings.option_tree2 = Some(Inner::default());
json_core::set(&mut settings, "/option_tree2/a", b"14")?;
// Hiding items of an array of `Tree`s
settings.array_option_tree[1] = Some(Inner::default());
json_core::set(&mut settings, "/array_option_tree/1/a", b"15")?;

let mut buf = [0; 16];

// Serializing nodes by path
let len = json_core::get(&settings, "/struct_", &mut buf).unwrap();
assert_eq!(&buf[..len], br#"{"a":3,"b":3}"#);

// Tree metadata
const MAX_DEPTH: usize = Settings::SCHEMA.shape().max_depth;
const MAX_LENGTH: usize = Settings::SCHEMA.shape().max_length("/");
assert!(MAX_DEPTH <= 6);
assert!(MAX_LENGTH <= 32);

// Iterating over all leaf paths
for path in Settings::SCHEMA.nodes::<Path<heapless::String<MAX_LENGTH>, '/'>, MAX_DEPTH>() {
    let path = path.unwrap().0;
    // Serialize each
    match json_core::get(&settings, &path, &mut buf) {
        // Full round-trip: deserialize and set again
        Ok(len) => { json_core::set(&mut settings, &path, &buf[..len])?; }
        // Some Options are `None`, some enum variants are absent
        Err(SerdeError::Value(ValueError::Absent)) => {}
        e => { e.unwrap(); }
    }
}

§Settings management

One possible use of miniconf is a backend for run-time settings management in embedded devices.

It was originally designed to work with JSON (serde_json_core) payloads over MQTT (minimq) and provides a MQTT settings management client in the miniconf_mqtt crate and a Python reference implementation to interact with it. Miniconf is agnostic of the serde backend/format, key type/format, and transport/protocol.

§Formats

miniconf can be used with any serde::Serializer/serde::Deserializer backend, and key format.

Explicit support for / as the path hierarchy separator and JSON (serde_json_core) is implemented.

Support for the postcard wire format with any postcard flavor and any Keys type is implemented. Combined with the Packed key representation, this is a very space-efficient serde-by-key API.

Blanket implementations are provided for all TreeSerialize+TreeDeserialize types for all formats.

§Transport

miniconf is also protocol-agnostic. Any means that can receive or emit serialized key-value data can be used to access nodes by path.

The MqttClient in the miniconf_mqtt crate implements settings management over the MQTT protocol with JSON payloads. A Python reference library is provided that interfaces with it. This example discovers the unique prefix of an application listening to messages under the topic quartiq/application/12345 and set its /foo setting to true.

python -m miniconf -d quartiq/application/+ /foo=true

§Derive macros

For structs miniconf offers derive macros for TreeSchema, TreeSerialize, TreeDeserialize, and TreeAny. The macros implements the TreeSchema, TreeSerialize, TreeDeserialize, and TreeAny traits. Fields/variants that form internal nodes (non-leaf) need to implement the respective Tree{Key,Serialize,Deserialize,Any} trait. Leaf fields/items need to support the respective serde (and the desired serde::Serializer/serde::Deserializer backend) or core::any trait.

Structs, enums, arrays, Options, and many other containers can then be cascaded to construct more complex trees.

See also the TreeSchema trait documentation for details.

§Keys and paths

Lookup into the tree is done using a Keys implementation. A blanket implementation through IntoKeys is provided for IntoIterators over Key items. The Key lookup capability is implemented for usize indices and &str names.

Path iteration is supported with arbitrary separator chars between names.

Very compact hierarchical indices encodings can be obtained from the Packed structure. It implements Keys.

§Limitations

enum: The derive macros don’t support enums with record (named fields) variants or tuple variants with more than one (non-skip) field. Only unit, newtype and skipped variants are supported. Without the derive macros, any enum is still however usable as a Leaf node. Note also that netwype variants with a single inline tuple are supported.
The derive macros only support flattening in non-ambiguous situations (single field structs and single variant enums, both modulo skipped fields/variants and unit variants).

§Features

json-core: Enable helper functions for serializing from and into json slices (using the serde_json_core crate).
postcard: Enable helper functions for serializing from and into the postcard compact binary format (using the postcard crate).
derive: Enable the derive macros in miniconf_derive. Enabled by default.

§Reflection

miniconf enables certain kinds of reflective access to heterogeneous trees. Let’s compare it to bevy_reflect which is a comprehensive and mature reflection crate:

bevy_reflect is thoroughly std while miniconf aims at no_std. bevy_reflect uses its Reflect trait to operate on and pass nodes as trait objects. miniconf uses serialized data or Any to access leaf nodes and pure “code” to traverse through internal nodes. The Tree* traits like Reflect thus give access to nodes but unlike Reflect they are all decidedly not object-safe and can not be used as trait objects. This allows miniconf to support non-'static borrowed data (only for TreeAny the leaf nodes need to be 'static) while bevy_reflect requires 'static for Reflect types.

miniconfsupports at least the following reflection features mentioned in the bevy_reflect README:

➕ Derive the traits: miniconf has Tree* derive macros and blanket implementations for arrays and Options. Leaf nodes just need some impls of Serialize/Deserialize/Any where desired.
➕ Interact with fields using their names
➖ “Patch” your types with new values: miniconf only supports limited changes to the tree structure at runtime (Option and custom accessors) while bevy_reflect has powerful dynamic typing tools.
➕ Look up nested fields using “path strings”: In addition to a superset of JSON path style “path strings” miniconf supports hierarchical indices and bit-packed ordered keys.
➕ Iterate over struct fields: miniconf Supports recursive iteration over node keys.
➕ Automatically serialize and deserialize via Serde without explicit serde impls: miniconf supports automatic serializing/deserializing into key-value pairs without an explicit container serde impls.
➕ Trait “reflection”: Together with crosstrait supports building the type registry and enables casting from dyn Any returned by TreeAny to other desired trait objects. Together with erased-serde it can be used to implement node serialization/deserialization using miniconf’s TreeAny without using TreeSerialize/TreeDeserialize similar to bevy_reflect.

Some tangential crates:

serde-reflection: extract schemata from serde impls. The trace example uses serde-reflection to extract a graph and schema for Tree*.
typetag: “derive serde for trait objects” (local traits and impls)
deflect: reflection on trait objects using adjacent DWARF debug info as the type registry
intertrait: inspiration and source of ideas for crosstrait

§Functional Programming, Polymorphism

The type-heterogeneity of miniconf also borders on functional programming features. For that crates like the following may also be relevant:

Modules§

deny: Deny access tools.
json_core: TreeSerialize/TreeDeserialize with “JSON and /”.
leaf: Leaf implementation using serde::{Serialize, Deserialize}
passthrough: Passthrough Tree*
postcard: TreeSerialize/TreeDeserialize with postcard.
str_leaf: TryFrom<&str>/AsRef<str> leaf

Structs§

Chain: Concatenate two Keys of different types
ExactSize: Counting wrapper for iterators with known exact size
Homogeneous: A representative schema item for a homogeneous array
Indices: Indices of usize to identify a node in a TreeSchema
JsonPath: JSON style path notation
JsonPathIter: JSON style path notation iterator
KeysIter: Keys/IntoKeys for Iterators of Key
Leaf: Serialize/Deserialize/Any leaf
Named: A named schema item
NodeIter: Node iterator
Numbered: A numbered schema item
Packed: A bit-packed representation of multiple indices.
Path: Path with named keys separated by a separator char
PathIter: String split/skip wrapper, smaller/simpler than .split(S).skip(1)
Schema: Type of a node: leaf or internal
Shape: Metadata about a TreeSchema namespace.
Short: Track leaf node encounter
Track: Track key depth

Enums§

DescendError: Errors that can occur while visting nodes with crate::Schema::descend.
Internal: An internal node with children
KeyError: Errors that can occur when using the Tree traits.
SerdeError: Compound errors
ValueError: Errors that can occur while accessing a value.

Traits§

IntoKeys: Be converted into a Keys
Key: Convert a key into a node index given an internal node schema
Keys: Capability to yield and look up Keys
Transcode: Look up an IntoKeys in a Schema and transcode it.
TreeAny: Access any node by keys.
TreeDeserialize: Deserialize a leaf node by its keys.
TreeDeserializeOwned: Shorthand for owned deserialization through TreeDeserialize.
TreeSchema: Traversal, iteration of keys in a tree.
TreeSerialize: Serialize a leaf node by its keys.

Type Aliases§

Meta: The metadata type

Derive Macros§

Tree: Derive the TreeSchema, TreeSerialize, TreeDeserialize, and TreeAny traits for a struct or enum.
TreeAny: Derive the TreeAny trait for a struct or enum.
TreeDeserialize: Derive the TreeDeserialize trait for a struct or enum.
TreeSchema: Derive the TreeSchema trait for a struct or enum.
TreeSerialize: Derive the TreeSerialize trait for a struct or enum.

Crate miniconfCopy item path