A new stream abstraction for rust

2 hours ago 2
Expand description

Abstractions for asynchronously working with series of data (“streams” and “sinks”).

See the producer and consumer modules for thorough introductions to the designs. See the ufotofu website for a discussion of our motivating design choices — these crate docs stay focussed on the what, not the why.

§Caveats

Ufotofu adheres to the principles of frugal async rust:

  • The futures returned by async ufotofu methods are !Send, they cannot be run on multi-threaded executors.
  • Dropping any method-returned future before polling it to completion will leave the original object in an undefined state; subsequent method calls may display arbitrary (but always safe) behaviour.
  • Unwinding any panic may leave ufotofu values in an undefined state. Do not attempt to recover from panics when using ufotofu.

§Module Overview

The two central modules are producer and consumer, they define the core abstractions of the crate.

The queues module provides the Queue trait for infallible in-memory queues with bulk push and pop operations, and some types implementing it. These power the buffered producer and consumer implementations of ufotofu.

The channels module provides ordered in-memory communication channels, which implement BulkProducer and BulkConsumer on their endpoints.

The fuzz_testing_tutorial demonstrates the utility types for fuzz testing that ufotofu provides.

§Actual I/O

The ufotofu abstractions are nice and all, but how do you actually do any I/O with ufotofu? The producer::compat::reader::reader_to_bulk_producer function turns any AsyncRead into a BulkProducer, and the consumer::compat::writer::writer_to_bulk_consumer function turns any AsyncWrite into a BulkConsumer (both functions require the compat_futures_io feature to be activated). This way, you can use established crates such as smol or tokio with ufotofu abstractions.

pub use producer::BulkProducer;pub use producer::BulkProducerExt;pub use producer::IntoBulkProducer;pub use producer::IntoProducer;pub use producer::Producer;pub use producer::ProducerExt;pub use consumer::BulkConsumer;pub use consumer::BulkConsumerExt;pub use consumer::Consumer;pub use consumer::ConsumerExt;pub use consumer::IntoBulkConsumer;pub use consumer::IntoConsumer;channelsIn-memory FIFO queue communication primitives.consumerConsumers — values that asynchronously process a sequence of items.fuzz_testing_tutorialA tutorial about fuzz testing ufotofu-related code.preludeA “prelude” for crates using the ufotofu crate.producerProducers — values that asynchronously yield a sequence of items.queuesIn-memory queues for adding buffering to arbitrary producers and consumers.consumeConveniently consume the output of a Producer.ConsumeAtLeastErrorAn error emitted when a consumer is tasked to consume at least some number of items, but it could only consume a lower number of items.ProduceAtLeastErrorAn error emitted when a producer is tasked to produce at least some number of items, but it could only produce a lower number of items.ExpectedFinalErrorAn error emitted when a function expects a final value, but gets a normal item instead.PipeErrorEverything that can go wrong when piping a producer into a consumer.bulk_pipeEfficiently pipes as many items as possible from a BulkProducer into a BulkConsumer, using BulkConsumerExt::bulk_consume. Then closes consumer with the final value emitted by the producer.pipePipes as many items as possible from a Producer into a Consumer. Then closes the consumer with the final value emitted by the producer.
Read Entire Article