Raw Data & Metadata

Raw data consists of unprocessed or only slightly normalized events from machines, control systems, sensors, files, or connected IT systems. It often contains technical field names, manufacturer-specific codes, and signals that can only be understood within the context of the plant.

Typical raw data includes:

• Timestamp
• Machine or Station ID
• Product or Workpiece Identification
• Status bits, counters, or switch states
• Process variables such as temperature, force, pressure, or displacement
• Message numbers, alarm texts, or HMI states

When it comes to raw data, it is what it is. In most cases, we don’t want to change raw data, because that would usually mean modifying the program directly in the controller. There are cases where this makes sense; for example, during the onboarding of a new system, we often find programming errors in the PLC, such as incorrectly configured signals. In most cases, however, we should accept raw data as is and leave it to the data preprocessing stage to put the data into the correct format.

Raw data is therefore initially stored in as unaltered a form as possible. This makes it possible to conduct late-stage analyses, even if the technical objective changes later on.

For further processing, raw data is mapped to stable fields. Examples include assetId, partId, timestamp, status, cycleTime, errorId, or process parameters such as temperature and pressure.

Raw data is therefore the technical source. The production-related insight only emerges later through logs, rules, and context.

Terms

Data that arrives in the system from the control system as raw data is first converted into a specific format for display in the IT system. There are various options and methods for doing this, each with its own advantages and disadvantages.

To provide a clearer context, we use the following terms:

• A "" signal () is a single data point, e.g., temperature.
• Several signals that we query together form a data structure.
• This data structure typically also requires a timestamp that indicates the time the signal was generated or recorded.
• If we query this data structure from the controller for specific time periods, or if we receive a new message from the controller when values change, the data is available as a continuous data stream consisting of individual data structures.
• A data stream is typically transmitted via a channel (topic) through which all messages of that data stream type are transmitted. This makes it easier to locate specific data later on (e.g., all measurement values from a station).

Modeling Approaches

As mentioned earlier, there are various approaches to structuring data. A particularly simple and generic approach would be to represent each signal along with a timestamp:

One measurement per message

Topic A: Temperature

timestamp_ms: 123
value: 32

Topic B: Pressure

timestamp_ms: 123
value: 580

With this approach, the data structure is always the same. Each message consists of a timestamp and a value. The semantics are primarily determined by the topic; for example, temperature data could be sent via the topic companyA.cnc.temperature and pressure data via the topic companyA.cnc.pressure. The drawback is that there is no logical grouping of signals. In many cases, we read related values from the control system—such as axis measurements (X/Y/Z), IDs associated with a process (e.g., the DMC code of the current part), or other properties. If you want to perform analyses on multiple signals, this creates a synchronization burden for the consumer.

Although this data model may seem simple at first glance (since it is easily transferable and can therefore be quickly rolled out across the entire fleet), it shifts the complexity to the consumer. However, this is something that happens only rarely, since the data user (e.g., production staff) is now tasked with more technical data processing tasks. Data that actually belong together (e.g., humidity and temperature) are split into separate messages, even though they are almost always needed together in an analytical context.

For this reason, we recommend including multiple measurements in a single message.

Multiple Measurement Values in a Single Message

This approach involves logically grouping data into a structure.

For example, if we connect a CNC machine, all the raw data related to the manufacturing process would be contained in a single message:

timestamp_ms: 123
temperature: 50
pressure: 580
processing: true
dmc: ABC

This data structure accurately represents the production process. The control system sets the "processing" value to "true" as soon as processing of a component begins. The measured values can either consist of real-time data that is continuously updated during the process or describe process parameters that are set only at the end of processing. Processing this raw data into a production log is straightforward in a subsequent transformation process, since only a single data stream is required for processing. This model is easy for data users to understand because it can be directly translated into database tables and made available in reporting tools, for example.

We therefore recommend that, for each processing step, you first define the data structures that group signals together from a logical perspective.

In many cases, we create, for example, two data structures per station or machine: one for process data, from which we later generate the production log and the machine state log; and another for message data, which forms the basis for the alarm and event log.

Metadata

In addition to live machine data, metadata is useful for describing the data structure schema alongside the frequently changing raw data. Metadata is not only useful for facilitating automated processing—for example, by modeling specific data types— Metadata makes data more accessible to end users through meaningful descriptions, as it makes the significance of the data immediately apparent.

Examples of metadata include:

• The data type of a signal; for example, the specification that temperature is a floating-point number.
• The semantic meaning of the signal, e.g., whether temperature refers to transmission temperature or ambient temperature
• The unit of measurement on which the signal is based, e.g., degrees Celsius
• The signal's measurement range, e.g., 0–100
• The signal's "Scope"; we typically use Messwert for measured values or Dimension for dimensions, such as assemblies. This information is used during the visual analysis of data—for example, to group measured data by dimension and display temperatures by assembly.
• Additional identifiers, such as descriptions, to provide data users with further information about the data's meaning.

A common question is where metadata is stored. Basically, there are two approaches:

In the first case, all metadata is part of the live data structure.

Procedures and Best Practices

Use of Raw Data in the Bytefabrik Platform