Process

Question and objective

What processing do your raw data need so that multiple analyses use the same operational meaning? The objective is to implement the required rules once, transparently, and reuse the results.

Implementation in Bytefabrik

StreamPipes pipelines connect data streams, processors, and sinks. Available processing elements handle tasks such as filtering, calculation, and aggregation. Bytefabrik's AI Pipelines extend this approach with individually generated, verifiable processing logic.

Task	Suitable starting point
Adapt an individual raw event before defining its schema	Script Transformation during connection setup
Connect existing processing elements into a data flow	Pipelines
Develop custom logic from an operational description	AI Pipelines
Investigate stored data for a one-off or exploratory analysis	AI Notebooks

Best practices and considerations

Define the input, expected output, and rules before implementation. Keep a small set of known events available for testing. Then choose the simplest suitable processing approach.

A cleaned stream is not yet a historical dataset: its output must be stored explicitly. Also check whether a rule needs only the current event or must account for previous states.

Start with transformations, then check data quality.