Overview

Overview

The sketch below shows the high-level flow of processing within Kolibri.

  1. Define the samples to operate on either via OrderedMultiValues (composed of OrderedValues) and use provided implicit conversions to create IndexedGenerator or create IndexedGenerator directly.
  2. Use batching strategy to split in single batches that are processed through computations defined in RunnableGraph. Each data sample results in a tagged ProcessingMessage[T], which is handled by AggregatingActor (created by RunnableExecutionActor that runs the RunnableGraph).
  3. If the expectation in the AggregatingActor is fulfilled or failed, write partial result if writer is defined, and provide AggregationState[V] upstream. Dependent on flag whether full results shall be passed upstream after execution, this state will either just indicate finished execution or additionally provide the aggregation.

Alt text

The following provides an overview of the distinct steps to define computations and the data structures involved.