Readers

Readers are helpers that enable easy reading of data from Signals by eliminating the need for having to manually establish a Connection to the Signal, parsing the Data Descriptor info and then read Data Packets in a correct format while still making sure that it is correct when the metadata changes.

Since this is always a chore and prone to mistakes, the Readers automate all this and enable you to handle reading the Signal data as it were a simple memory-stream conveniently already in your desired format. This way you can focus on actually doing your work instead of data juggling and worrying about all the details and options.

Since event handlers and call-backs are currently not applicable for .NET Bindings, some of the topics below cannot be used in .NET applications.

Types of Readers

The openDAQ™ SDK provides multiple types of Readers depending on what do you actually need or want to achieve. Some of the most used ones are listed below:

  • Packet reader just forms a Connection to the Signal and passes the queued packets to the user on request without doing any other processing.

  • Stream reader reads data as it were a stream of values, merging data packets into a continuous data buffer.

  • Tail reader always reads the latest n values output by the signal.

  • Block reader Reads the data in predefined block size and can’t read less than a full block.

  • Multi reader Reads aligned data from multiple signals.

Common Behavior

Initialization

All the Readers provided by openDAQ™ are initialized and behave in a common manner. As a parameter, they receive the Signal to read and the wanted sample-type the data and domain values should be read as. The example of Constructing a reader shows how a Reader for Signal signal is constructed to read value data as double and domain data as Int64.

Example 1. Constructing a reader
// Standard factory signature with signal as input argument
Reader(signal, SampleType::Float64, SampleType::Int64);

// Standard factory signature with input port as input argument
Reader(inputPort, SampleType::Float64, SampleType::Int64);

// In C++ there is also a templated helper
Reader<double, std::int64_t>(signal);

There is also a way to construct a Reader without knowing the sample-types in advance by using SampleType::Undefined.

Reading without knowing the sample-types in advance
Reader(signal, SampleType::Undefined, SampleType::Undefined);

In this case, the user must take extra care to check the actual types before reading and provide correct buffers to the reader read calls otherwise the results are undefined and will probably cause a crash.

When the requested Sample-type for value or domain doesn’t match with the ones produced by the Signal, the reader attempts to convert the read data into the requested Sample type. If the data can’t be converted the Reader goes into an invalid mode and all subsequent read operations will fail. How to resolve the invalid state is explained in Reader invalidation and reuse.

For the purposes of a Reader a conversion exists if it can be performed with an assignment cast.

E.g: The following expression must be valid in C++
Type1 a{};
Type2 b = (Type2) a;

As the Reader receives the Signal as a parameter it must first establish a Connection to it. To perform this, the Readers usually create an internal Input Port to which they connect the passed-in Signal to form a Connection and start to listen to the port events. A Reader can also be created with an existing input port. In this case, it won’t create an internal input port but will instead use the existing one. This enables a Reader to receive Packets and by definition the first packet sent after a new Connection is established is a Data Descriptor Changed event packet containing the descriptors for both value and domain data. These are then sent to the internal data-readers for value and domain to be able to read and convert data. If the data-reader determines that the read operations can’t be performed for a certain reason (e.g. incompatible or inconvertible sample-types) the reader is invalidated.

When re-using an existing Reader the initialization procedure is the same. The only difference is that the Input Port with its Connection is reused and event listening reassigned to the new Reader. After this a check is made if the packet on the top of the connection queue is a Data Descriptor Changed event otherwise it is read directly from the Signal. The old Reader is invalidated after re-use if it wasn’t already.

In typical scenarios, Readers with input ports are frequently employed in modules where ports are attached to the openDAQ component tree within an "Input ports" folder. This is especially common in function blocks or channels.

Reader invalidation and reuse

Once the Reader falls into invalid state, it can’t be used to read data anymore and all attempts will result in an OPENDAQ_ERR_INVALID_DATA error code or the associated exception. The only way to resolve this is to pass the Reader to a new Reader instance with valid sample-types and settings. This enables the new reader to reuse the Connection from the invalidated one and as such, provides the ability to losslessly continue reading.

You can also reuse a valid Reader, for example, if you want to change the read sample-type or change any other configuration that is immutable after creating a Reader. This will make the old reader invalid.

Read calls

All the Readers expose at least two common operations:

  • getAvailableCount() reports how much of something is still left to read. This something can differ between the kinds of readers. In the case of Packet Reader this is the number of packets ready and available in the Connection queue, while for Stream Reader this is the number of samples stored in the available packets.

  • setOnDataAvailable(callback) assigns a callback function to be called when the Reader has available data packets On Data Available event packet. With this callback, the user can automatically read data from reader. More details on how this is handled can be found in the Handling on Data Available section below.

In addition to these two operations, Readers also define their own methods to read data. These read calls usually follow the Read calls function signature where two functions read and readWithDomain are defined.

Example 2. Read calls function signature
ReaderStatusPtr read(void* values, std::size_t* count);
ReaderStatusPtr readWithDomain(void* values, void* domain, std::size_t* count);

The way to use the read calls is to have a memory buffer of a desired size and type pre-allocated. Then you pass it into the call where it will get filled with at maximum count elements. The Reader returns a ReaderStatusPtr object, indicating whether the reading process was successful or if there is an event packet that needs to be handled. Moreover, if the data type has changed, the Reader will include in the status whether the new type is convertible by the Reader or not.

The count / size parameter needs to be set before the call to a desired maximum count and will be modified with the actual amount read after.
The type of the allocated memory buffer must match with the type the Reader is configured to read. There are no run-time checks to enforce this. If the buffer is bigger than the read amount, the rest of the buffer is not modified.

Sample Reader

Sample Reader is an extension of the basic reader that operates on samples, and all openDAQ™ provided Readers except the basic Packet Reader are specializations of it.

The Sample Reader provides another four operations:

  • getValueReadType() / getDomainReadType() reports the sample-type of samples the Reader outputs on read calls. This should be the same as the one passed in on construction except in the case where SampleType::Undefined was used. There it is the Signal’s data type.

  • setValueTransformFunction(callback) / setDomainTransformFunction(callback) enables custom user transformation of raw signal data specific to the programming language or use case. See the chapter Custom conversion of signal data for more info.

If there is a custom transform function assigned the corresponding value or domain SampleType requested at construction is completely ignored and the Reader directly returns whatever data the callback produces. No additional processing is done except to advance the reading position if required.

Handling Data Available

When the Reader is notified about new packets, each packet follows its own logic to determine whether it should trigger the onDataAvailable function or not. Currently, the Stream reader, Packet reader, and Multi reader trigger the callback when any packet arrives at the Reader’s input port. The Tail reader triggers the callback if the total number of samples is greater than the history size. The Block reader will trigger the callback if there is enough available samples for one block.

The user callback signature
void callback()
[optionalCapturedArguments]() -> void {}

Handling a Descriptor changed event

Whenever the Signal information changes, it sends an Event Packet with and id of "SIGNAL_DESCRIPTOR_CHANGED". This event contains new Data Descriptors for both value and domain data. The processing of event packets in our system occurs dynamically through the reader, not immediately upon reception, but rather during the reading process.

To illustrate, consider a scenario with a queue containing 10 packets. One of these is an event packet positioned in the middle, while the remaining packets are data packets, each containing two samples. In a user scenario where reading up to 5 packets is requested, the event packet will not be included in the processing list. However, if the user attempts to read more than 5 samples, the reader will return 5 samples, update the types of internal readers, and provide a reading status. This status will include information about the event packet, and whether the reader can convert new data or not.

auto reader = StreamReader<double, Int>(signal);

// Signal produces 5 samples { 1.0, 2.0, 3.0, 4.0, 5.0 }
auto packet1 = createPacketForSignal(signal, 5);
auto data1 = static_cast<double*>(packet1.getData());
data1[0] = 1.0;
data1[1] = 2.0;
data1[2] = 3.0;
data1[3] = 4.0;
data1[4] = 5.0;

signal.sendPacket(packet1);

// change data type
signal.setDescriptor(setupDescriptor(SampleType::Int64));

auto packet2 = createPacketForSignal(signal, 5);
auto data2 = static_cast<Int*>(packet2.getData());
data2[0] = 6;
data2[1] = 7;
data2[2] = 8;
data2[3] = 9;
data2[4] = 10;

signal.sendPacket(packet1);

SizeT count{10};
double values[10]{};
ReaderStatusPtr status = reader.read(values, &count);
// count = 5, values = { 1.0, 2.0, 3.0, 4.0, 5.0 }
// status.getReadStatus() == ReadStatus::Event

// reading remaining data
count = 5;
reader.read(&values[5], &count);

If the Reader was created with SampleType::Undefined the actual sample-type returned by the getValueSampleType() and getDomainSampleType() gets inferred at the first "DATA_DESCRIPTOR_CHANGED" event where the respective Data Descriptor is available. Until then these calls will return SampleType::Invalid.

In the case of domain the Signal might not even have associated domain data descriptor defined, so it will be inferred at the first readWithDomain() call.

Custom conversion of signal data

Sometimes the Reader can’t auto convert the data with a normal cast for whatever reason. Maybe the conversion is not available during SDK compilation or is specific to the language or use case. For these cases, there are basically three ways to proceed:

  1. Read into an intermediate buffer and then convert:

    • Easy to program

    • Heavy on the memory usage.

  2. Create a whole new reader:

    • Time-consuming even if inherited from an existing implementation.

    • It has to be specialized for every new kind of reader.

    • Fully flexible

  3. Use a transform callback:

    • A simple function that receives raw data and the current Data Descriptor and outputs the transformed values back.

    • It works for any reader and without intermediate buffers.

    • The only catch is that the user must expect this transformation and allocate the buffers correctly.

To use the third option, install a custom callback with the respective domain or value transform setters. The callback signature is shown below where inputBuffer and inputBuffer are passed over the SDK boundary as Int and need to be cast back to void* or the correctly typed pointers. The pointer data type is the same as the one you’d get directly from the Packet getData() and can be read from the passed-in descriptor.

The transform callback signature
bool callback(Int inputBuffer,
              Int outputBuffer,
              SizeT toRead,
              DataDescriptor descriptor)

Packet Reader

Packet reader is the simplest of all the Readers provided by the openDAQ™. It only creates a Connection between the Signal and the Reader and gives the user the option to read Packet after Packet or get all the currently queued ones as a list.

By itself, this does not accomplish much, but it is a great base to build upon if you need some custom specific handling that you can’t achieve using any other provided reader plus you get the Connection queue handling for free, and since there is no other processing being done on packets, it is also as fast as it can be.

Stream Reader

This is the reader that will be useful in most cases. It represents the Connection packet queue to the user as a continuous stream of samples and automatically advances the current read position, handles reading over Packet boundaries and can optionally wait for the requested samples with a time-out.

The read calls follow the common Read calls function signature with an additional parameter specifying the time-out in milliseconds. On construction Stream Reader also requires you to specify how this time-outs should be handled.

There are two options:

  • ReadTimeoutType::Any will return immediately with samples available without waiting for the time-out. If there are none available, it will wait until time-out is exceeded or the next packet arrives. On the next packet it returns immediately even if there is time remaining.

  • ReadTimeoutType::All is the default and always waits for the time-out to be exceeded if the requested number of samples has not been read yet.

Related articles

Tail Reader

This Reader always reads the latest N values output by the signal. On subsequent calls, the samples can overlap and will return already read samples if there isn’t enough of new ones. This is useful if you have some visual control displaying value history, e.g. a scope.

The read calls follow the common Read calls function signature and on construction there is an additional parameter specifying the maximum number of samples in history to keep.

The reader keeps just enough packets in the cache to store at least N samples and removes the oldest packets when new arrive if there are enough samples in the remaining ones.

The Reader will throw an error if trying to read more than N packets except in the case that the cache happens to have enough samples due to having to keep a larger packet to satisfy the history limit.

The following will succeed even if more than history size
History size: 5
Packet sizes: 1 + 3 + 4 (latest to oldest)
Requested samples: 6

Related articles

Block Reader

This reader functions almost exactly the same as the Stream Reader except that it reads the data only in predefined block size and can’t read less than a full block. This is useful in filters and, for example, when calculating FFT.

The block size is defined on construction:

BlockReader(signal, blockSize, valueType, domainType);

Multi Reader

Multi Reader is "just" a Stream Reader that reads multiple signals at once. The catch is that in openDAQ™ Signals can have different starting points, sample rates and clocks. Therefore, the job of a Multi Reader is to align all Signals to the same starting point and on read calls return values for all signals on the same domain point, usually the same time-stamp.

Related articles