Apache StreamPipes: Architecture, Features, and Use Cases
Apache StreamPipes is an open-source platform that enables companies to connect, structure, process in real time, store, and visualize industrial data streams. The platform is designed for both business users, who configure data flows via a graphical user interface, and development teams, who can add their own adapters, processors, and applications.
Since 2023, Apache StreamPipes has been a top-level project of the Apache Software Foundation. This means that the source code, releases, and project decisions are managed according to the rules of an independent open-source community. Bytefabrik initiated StreamPipes and continues to play a key role in its development.
What is Apache StreamPipes?
StreamPipes provides an end-to-end workflow for industrial event data:
- Connect: Adapters read data from machines, control systems, brokers, files, or software systems.
- Structuring: Incoming events are reviewed, described, and harmonized as needed.
- Processing: Pipelines filter, transform, combine, or analyze real-time data streams.
- Benefits: Results are stored, visualized, forwarded to third-party systems, or made available for use in applications.
The building blocks can be configured via a web-based interface. This means that recurring tasks do not require a new, custom-developed integration service for each data source.
The Key Components
Adapters and Data Streams
Adapters establish the connection to the data source. Supported integrations include OPC UA, MQTT, Apache Kafka, and industrial protocols such as S7 or Modbus connected via Apache PLC4X. It’s not just about the transport: When creating an adapter, users can review sample data, describe fields, and define the resulting event schema.
The published data stream then serves as a reusable interface for pipelines, storage, and visualization. This separates source-specific details from further use.
Pipelines and Stream Processing
A pipeline connects one or more data streams to processors and data sinks. Typical steps include:
- Filter values and monitor thresholds;
- Rename, calculate, or add context to fields;
- Standardize units and timestamps;
- Merge events or aggregate them over time;
- Save results, forward them, or display them as a notification.
Pipelines are well-suited for continuous processing of live data. Extensive historical analyses or domain-specific production models often require additional data models and applications.
Historical Data and Visualization
Events can be persisted as data records and then analyzed in charts and dashboards. This allows you to create both real-time views and historical analyses based on the same data set. For production-wide metrics, however, raw signals must first be classified according to their business context—for example, as machine status, production event, or process measurement.
Assets and Technical Context
An asset model maps data sources and resources to real-world structures such as locations, production lines, or machines. This mapping becomes important as soon as installations grow beyond a single pilot project: Users must be able to find data based on the plant context and transfer configurations to comparable machines.
Extensions and APIs
Custom adapters, data processors, and data sinks can be added as extensions. To this end, the project provides tools for Java, Python, and TypeScript, as well as client libraries and a REST interface. This allows proprietary machine protocols or company-specific algorithms to be integrated without modifying the platform core.
Typical Applications
Harnessing Machine and Sensor Data
StreamPipes can connect heterogeneous data sources via a common interface and map their events to a comprehensible schema. This serves as a good starting point for retrofit projects, machine fleets with different generations of control systems, or a phased IIoT implementation.
Live Monitoring and Notifications
Pipelines can detect relevant conditions and send results directly to dashboards, external systems, or notification channels. Examples include threshold violations, unusual event rates, or missing updates from a source.
Industrial Data Preprocessing
Before data is fed into a data lake, historian, or line-of-business application, it can be filtered, normalized, and enriched with metadata. This provides downstream systems with a more stable and better-documented interface.
Pilot Testing of New Data Applications
Using existing adapters, pipelines, and visualizations, teams can quickly evaluate a data source and test initial hypotheses. A successful pilot should then be expanded to include operations, permissions, data models, and reusability.
Foundation for Custom IIoT Applications
Developers can build their own applications on the platform using APIs, stored datasets, and data streams. StreamPipes then handles recurring tasks such as connectivity and event processing, while the application provides domain-specific user guidance.
What StreamPipes Replaces—and What It Doesn't
StreamPipes handles several tasks that would otherwise be spread across individual integration and analytics tools. Nevertheless, the platform is not automatically a replacement for every existing system.
| System or Tool | Differences from Apache StreamPipes |
|---|---|
| MQTT or Kafka broker | Reliably transmits messages; StreamPipes adds adapters, schemas, processing, storage, and user interfaces. |
| PLC, HMI, or SCADA | Controls and operates the system; StreamPipes uses shared data for cross-system processing and analysis. |
| Historian | Specializes in the long-term storage and retrieval of process values; can serve as a data source, target system, or supplemental storage. |
| Node-RED | Suitable for visual integration flows; StreamPipes focuses on the shared lifecycle of industrial data streams, assets, pipelines, and analytics. |
| MES | Plans and documents production execution; provides order and product context or uses processed machine data. |
| BI Platform: Generates reports based on processed data; StreamPipes can handle operational data collection and preprocessing. |
The appropriate role depends on the target architecture. StreamPipes often complements existing brokers, historian, MES, or BI systems rather than completely replacing them.
Deployment and Production Operations
Container-based setups using Docker Compose are available for local testing and smaller installations. For cluster-based environments, the project supports Kubernetes. However, the technical installation is only one part of productive operation.
Before a rollout, teams should clarify the following in particular:
- in which network segment the adapter, broker, processing, and storage are running;
- how certificates, login credentials, roles, and external access are managed;
- which data rates, retention periods, and queries need to be sized;
- how monitoring, backups, updates, and recovery work;
- who is technically responsible for data sources, schemas, pipelines, and extensions;
- how configurations are transferred between development, testing, and production.
A pilot can be launched on a single host. However, when it comes to scaling, the operating model and governance are just as important as additional computing power.
Open Source and Commercial Extensions
Apache StreamPipes is available under the Apache License 2.0. Companies can review the source code, run it themselves, and develop their own extensions. However, “open source” does not automatically mean that implementation and operation are effortless: connectivity, security, data modeling, maintenance, and user empowerment remain real challenges.
Bytefabrik products, when compared, build on this open foundation and supplement it with production-oriented applications, advanced analytics, AI capabilities, and guided support for deployment. The appropriate level depends on the goal:
- Apache StreamPipes:—ideal for teams that want to evaluate, extend, and operate an open IIoT platform on their own.
- IoT Data Hub: is a good fit when a managed data platform with additional operational, governance, and integration capabilities is needed.
- Manufacturing Insights: is ideal when production losses, conditions, alerts, and processes need to be analyzed using predefined analysis workflows.
A Good Place to Start
1. Select a specific data source
Start with a machine, a broker, or an existing test source. Document the protocol, network path, event rate, and required access credentials.
2. Define a measurable outcome
An initial success could be a stable data stream, a persisted dataset, or a dashboard displaying a few relevant metrics. The result should be something that a business user can evaluate quickly.
3. Check the schema and data quality
Check timestamps, units, data types, missing values, and how the system handles connection interruptions. This check prevents technical errors from appearing later as business anomalies.
4. Set up processing as a reusable pipeline
Separate source-specific cleanup from business logic. Name data streams and fields consistently so that other applications can build on them.
5. Plan Production Operations
Define permissions, monitoring, backups, update processes, and responsibilities before deploying the same solution to additional machines or locations.
Evaluation Checklist
- Are the protocols and sources that are actually needed covered?
- Can event schemas, units, and metadata be described clearly?
- Are the existing pipeline elements sufficient, or are custom extensions needed?
- Are real-time processing, history, and visualization supported in the required format?
- Is the deployment compatible with the OT network, security, and operational organization?
- Can configurations and data models be applied to other systems?
- Are there the necessary skills and responsibilities in place for maintenance and further development?
- Do users need a technical platform, or are they already looking for a fully developed production application?