Skip to main content
+49 721 9654-724

Datasets

Question and objective​

What historical data is available for charts and analyses, and how reliable is it? A dataset makes it possible to repeatedly retrieve stored events.

Implementation in Bytefabrik​

Data can be stored during integration or via a suitable pipeline sink. The StreamPipes Base data set management feature provides information on schemas, events, and data volume. Retention functions control how older data is handled according to the respective configuration.

A continuous data stream alone does not yet constitute a usable history. Therefore, after setup, verify that the expected data set actually contains events.

A broker event stream is divided into two usage paths: ongoing processing with live visualization, and a configured sink, historical storage, and later analysis. The diagrams are schematic and do not represent performance data.
Real-time processing and historical analysis use different approaches: Only the configured storage system receives events for later analysis. Open full-size image (new tab)

Best Practices​

  1. Document the purpose, source, time period, units, and asset reference.
  2. Check the stored data for a known event.
  3. Check the timeliness and volume of data over a representative period.
  4. Tailor your storage strategy to your analysis and storage needs.
  5. Note which charts, notebooks, and Insights views use the dataset.

What You Should Keep in Mind​

Raw data and aggregated results serve different purposes. Aggregation reduces the volume of data but may result in the loss of details needed for later root-cause analyses.

A schema change does not automatically correct old events. Check for any impact on consumers and historical comparisons. Deleting data or shortening the retention period can also alter existing analyses. An export should be reviewed for content and future reusability before it is considered a replacement for existing data.

The benefit of a well-maintained data structure lies in the ability to perform repeatable analyses without having to re-collect and reprocess the same data.