Readers

Readers are helpers that enable easy reading of data from Signals by eliminating the need for having to manually establish a Connection to the Signal, parsing the Data Descriptor info and then read Data Packets in a correct format while still making sure that it is correct when the metadata changes.

Since this is always a chore and prone to mistakes, the Readers automate all this and enable you to handle reading the Signal data as it were a simple memory-stream conveniently already in your desired format. This way you can focus on actually doing your work instead of data juggling and worrying about all the details and options.

Types of Readers

The openDAQ™ SDK provides multiple types of Readers depending on what do you actually need or want to achieve. Some of the most used ones are listed below:

  • Packet reader just forms a Connection to the Signal and passes the queued packets to the user on request without doing any other processing.

  • Stream reader reads data as it were a stream of values, merging data packets into a continuous data buffer.

  • Tail reader always reads the latest n values output by the signal.

  • Block reader Reads the data in predefined block size and can’t read less than a full block.

  • Multi reader Reads aligned data from multiple signals. (Not yet implemented)

Common Behavior

Initialization

All the Readers provided by openDAQ™ are initialized and behave in a common manner. As a parameter, they receive the Signal to read and the wanted sample-type the data and domain values should be read as. The example of Constructing a reader shows how a Reader for Signal signal is constructed to read value data as double and domain data as Int64.

Example 1. Constructing a reader
// Standard factory signature
Reader(signal, SampleType::Float64, SampleType::Int64);

// In C++ there is also a templated helper
Reader<double, std::int64_t>(signal);

There is also a way to construct a Reader without knowing the sample-types in advance by using SampleType::Undefined.

Reading without knowing the sample-types in advance
Reader(signal, SampleType::Undefined, SampleType::Undefined);

In this case, the user must take extra care to check the actual types before reading and provide correct buffers to the reader read calls otherwise the results are undefined and will probably cause a crash.

When the requested Sample-type for value or domain doesn’t match with the ones produced by the Signal, the reader attempts to convert the read data into the requested Sample type. If the data can’t be converted the Reader goes into an invalid mode and all subsequent read operations will fail. How to resolve the invalid state is explained in Reader invalidation and reuse.

For the purposes of a Reader a conversion exists if it can be performed with an assignment cast.

E.g: The following expression must be valid in C++
Type1 a{};
Type2 b = (Type2) a;

As the Reader receives the Signal as a parameter it must first establish a Connection to it. To perform this, the Readers usually create an internal Input Port to which they connect the passed-in Signal to form a Connection and start to listen to the port events. This enables a Reader to receive Packets and by definition the first packet sent after a new Connection is established is a Data Descriptor Changed event packet containing the descriptors for both value and domain data. These are then sent to the internal data-readers for value and domain to be able to read and convert data. If the data-reader determines that the read operations can’t be performed for a certain reason (e.g. incompatible or inconvertible sample-types) the reader is invalidated.

When re-using an existing Reader the initialization procedure is the same. The only difference is that the Input Port with its Connection is reused and event listening reassigned to the new Reader. After this a check is made if the packet on the top of the connection queue is a Data Descriptor Changed event otherwise it is read directly from the Signal. The old Reader is invalidated after re-use if it wasn’t already.

Reader invalidation and reuse

Once the Reader falls into invalid state, it can’t be used to read data anymore and all attempts will result in an OPENDAQ_ERR_INVALID_DATA error code or the associated exception. The only way to resolve this is to pass the Reader to a new Reader instance with valid sample-types and settings. This enables the new reader to reuse the Connection from the invalidated one and as such, provides the ability to losslessly continue reading.

You can also reuse a valid Reader, for example, if you want to change the read sample-type or change any other configuration that is immutable after creating a Reader. This will make the old reader invalid.

Read calls

All the Readers expose at least two common operations:

  • getAvailableCount() reports how much of something is still left to read. This something can differ between the kinds of readers. In the case of Packet Reader this is the number of packets ready and available in the Connection queue, while for Stream Reader this is the number of samples stored in the available packets.

  • setOnDescriptorChanged(callback) assigns a callback function to be called when the Reader encounters a Data Descriptor Changed event packet. With this callback, the user can control whether the change is still permissible and perform any needed changes. How this is handled is described in more detail in the Handling a Descriptor Changed Event section below.

In addition to these two operations, Readers also define their own methods to read data. These read calls usually follow the Read calls function signature where two functions read and readWithDomain are defined.

Example 2. Read calls function signature
read(void* values, std::size_t* count);
readWithDomain(void* values, void* domain, std::size_t* count);

The way to use the read calls is to have a memory buffer of a desired size and type pre-allocated. Then you pass it into the call where it will get filled with at maximum count elements.

The count / size parameter needs to be set before the call to a desired maximum count and will be modified with the actual amount read after.
The type of the allocated memory buffer must match with the type the Reader is configured to read. There are no run-time checks to enforce this. If the buffer is bigger than the read amount, the rest of the buffer is not modified.

Sample Reader

Sample Reader is an extension of the basic reader that operates on samples, and all openDAQ™ provided Readers except the basic Packet Reader are specializations of it.

The Sample Reader provides another four operations:

  • getValueReadType() / getDomainReadType() reports the sample-type of samples the Reader outputs on read calls. This should be the same as the one passed in on construction except in the case where SampleType::Undefined was used. There it is the Signal’s data type.

  • setValueTransformFunction(callback) / setDomainTransformFunction(callback) enables custom user transformation of raw signal data specific to the programming language or use case. See the chapter Custom conversion of signal data for more info.

If there is a custom transform function assigned the corresponding value or domain SampleType requested at construction is completely ignored and the Reader directly returns whatever data the callback produces. No additional processing is done except to advance the reading position if required.

Handling a Descriptor changed event

Whenever the Signal information changes, it sends an Event Packet with and id of "SIGNAL_DESCRIPTOR_CHANGED". This event contains new Data Descriptors for both value and domain data. The Reader first forwards the descriptors to their respective internal data-readers to update their information and check if the data can still be converted to the requested sample-types.

If all these internal checks pass, the user callback is called (if installed) with the event’s descriptors to check if the change is still permissible to the user otherwise the Reader is invalidated.

The user callback signature
bool callback(SignalDesciptor valueDescriptor,
              SignalDesciptor domainDescriptor)

If the Reader was created with SampleType::Undefined the actual sample-type returned by the getValueSampleType() and getDomainSampleType() gets inferred at the first "DATA_DESCRIPTOR_CHANGED" event where the respective Data Descriptor is available. Until then these calls will return SampleType::Invalid.

In the case of domain the Signal might not even have associated domain data descriptor defined, so it will be inferred at the first readWithDomain() call.

Custom conversion of signal data

Sometimes the Reader can’t auto convert the data with a normal cast for whatever reason. Maybe the conversion is not available during SDK compilation or is specific to the language or use case. For these cases, there are basically three ways to proceed:

  1. Read into an intermediate buffer and then convert:

    • Easy to program

    • Heavy on the memory usage.

  2. Create a whole new reader:

    • Time-consuming even if inherited from an existing implementation.

    • It has to be specialized for every new kind of reader.

    • Fully flexible

  3. Use a transform callback:

    • A simple function that receives raw data and the current Data Descriptor and outputs the transformed values back.

    • It works for any reader and without intermediate buffers.

    • The only catch is that the user must expect this transformation and allocate the buffers correctly.

To use the third option, install a custom callback with the respective domain or value transform setters. The callback signature is shown below where inputBuffer and inputBuffer are passed over the SDK boundary as Int and need to be cast back to void* or the correctly typed pointers. The pointer data type is the same as the one you’d get directly from the Packet getData() and can be read from the passed-in descriptor.

The transform callback signature
bool callback(Int inputBuffer,
              Int outputBuffer,
              SizeT toRead,
              DataDescriptor descriptor)

Packet Reader

Packet reader is the simplest of all the Readers provided by the openDAQ™. It only creates a Connection between the Signal and the Reader and gives the user the option to read Packet after Packet or get all the currently queued ones as a list.

By itself, this does not accomplish much, but it is a great base to build upon if you need some custom specific handling that you can’t achieve using any other provided reader plus you get the Connection queue handling for free, and since there is no other processing being done on packets, it is also as fast as it can be.

Stream Reader

This is the reader that will be useful in most cases. It represents the Connection packet queue to the user as a continuous stream of samples and automatically advances the current read position, handles reading over Packet boundaries and can optionally wait for the requested samples with a time-out.

The read calls follow the common Read calls function signature with an additional parameter specifying the time-out in milliseconds. On construction Stream Reader also requires you to specify how this time-outs should be handled.

There are two options:

  • ReadTimeoutType::Any will return immediately with samples available without waiting for the time-out. If there are none available, it will wait until time-out is exceeded or the next packet arrives. On the next packet it returns immediately even if there is time remaining.

  • ReadTimeoutType::All is the default and always waits for the time-out to be exceeded if the requested number of samples has not been read yet.

Related articles

Tail Reader

This Reader always reads the latest N values output by the signal. On subsequent calls, the samples can overlap and will return already read samples if there isn’t enough of new ones. This is useful if you have some visual control displaying value history, e.g. a scope.

The read calls follow the common Read calls function signature and on construction there is an additional parameter specifying the maximum number of samples in history to keep.

The reader keeps just enough packets in the cache to store at least N samples and removes the oldest packets when new arrive if there are enough samples in the remaining ones.

The Reader will throw an error if trying to read more than N packets except in the case that the cache happens to have enough samples due to having to keep a larger packet to satisfy the history limit.

The following will succeed even if more than history size
History size: 5
Packet sizes: 1 + 3 + 4 (latest to oldest)
Requested samples: 6

Related articles

Block Reader

This reader functions almost exactly the same as the Stream Reader except that it reads the data only in predefined block size and can’t read less than a full block. This is useful in filters and, for example, when calculating FFT.

The block size is defined on construction:

BlockReader(signal, blockSize, valueType, domainType);

Multi Reader

Multi Reader is "just" a Stream Reader that reads multiple signals at once. The catch is that in openDAQ™ Signals can have different starting points, sample rates and clocks. Therefore, the job of a Multi Reader is to align all Signals to the same starting point and on read calls return values for all signals on the same domain point, usually the same time-stamp.

Related articles