Acquisition and Streaming

Table of Contents

• Introduction
• Acquisition in CVB
• Stream Type
  • Image Stream
  • Point Cloud Stream
  • Composite Stream
• Anatomy of a CVB Stream
  • Flow Set Pool
• Image lifetime
• Cancellation
• Event Interface
• Examples
• Further Reading

<< Configuration GenApi >>

Introduction

Note

This document describes the newest 3rd Generation CVB acquisition stack using GenTL. If using an older acquisition stack using the VIN drivers, refer to the CVB 14.1 Documentation and Migrating from the 2nd Gen acquisition stack.

Once a device has been discovered per Device Discovery, acquisition is the process creating the data stream, and then retrieving the data from it.

Whilst the CVB interface is designed primarily for image streaming from a camera, there are no restrictions on the type of data sent and it is perfectly capable of receiving any binary data (e.g. text or metadata). For the purpose of this documentation, the term Device will be used for the sender side (camera, sensor, etc.), and the term Image used for the data payload (image, point cloud, binary data, etc.).

Acquisition in CVB

In CVB, acquisition begins by creating a Stream object of either Image Stream, Point Cloud Stream or Composite Stream. Once the stream has been created, it can be configured to use custom memory, before being started.

Acquisition can be started on each data stream individually using the Cvb::StreamBase::Start method. This will allocate the Cvb::FlowSetPool (see Flow Set Pool) if it has not been done already, and informs the device to begin streaming data on the chosen stream.

C++C#Python

C++

using namespace Cvb;
 
auto stream = device.Stream<ImageStream>(0);
stream.Start();

C#

using Stemmer.Cvb;
using Stemmer.Cvb.Driver;
 
var stream = device.GetStream<ImageStream>(0);
stream.Start();

Python

import cvb
 
stream = device.stream(cvb.ImageStream, 0)
stream.start()

Stream Type

Inside CVB, all GenICam devices use a Cvb::CompositeStream for publishing data. In addition, CVB provides an Cvb::ImageStream and Cvb::PointCloudStream to simplify operations for common use cases. Which stream type to use depends on the device in question as well as what data needs to be processed.

Image Stream

The Cvb::ImageStream is a specialization of the Cvb::CompositeStream for usage with 1D or 2D images, allowing indexed access to pixel and subpixel data. It returns a Cvb::MultiPartImage that contains only image components of the stream.

When receiving only a single image, the Cvb::MultiPartImage will act as a Cvb::Image, allowing direct access to all image related data without requiring conversion.

C++C#Python

C++

using namespace Cvb;
 
MultiPartImagePtr image;
WaitStatus status;
NodeMapEnumerator nodeMaps;
std::tie(image, status, nodeMaps) = stream->Wait();
 
// Access the image
auto pixelValue = image.GetPixel({x, y});

C#

using Stemmer.Cvb;
using Stemmer.Cvb.Driver;
 
WaitStatus status;
 
using (var image = stream.Wait(out status))
{
  // Access the image
  var pixelValue = image.GetPixel(x, y);
}

Python

import cvb
 
image, status, node_maps = stream.wait()
with image:
  # Access the image
  pixel_value = image.get_pixel(cvb.Point2D(x, y))

When receiving multiple images, the Cvb::MultiPartImage will also act as a container of images, allowing for indexed access to each image received. Accessing the Cvb::MultiPartImage as a Cvb::Image directly will access the information from the first image.

C++C#Python

C++

using namespace Cvb;
 
MultiPartImagePtr multiPartImage;
WaitStatus status;
NodeMapEnumerator nodeMaps;
std::tie(multiPartImage, status, nodeMaps) = stream->Wait();
 
// Access the first image
auto pixelValue1 = get<ImagePtr>(multiPartImage.GetPartAt(0))->GetPixel({x, y});
 
// Access the second image
auto pixelValue2 = get<ImagePtr>(multiPartImage.GetPartAt(1))->GetPixel({x, y});

C#

using Stemmer.Cvb;
using Stemmer.Cvb.Driver;
 
WaitStatus status;
 
using (var multiPartImage = stream.Wait(out status))
{
  // Access the first image
  var pixelValue1 = Image.FromHandle(multiPartImage.Parts[0].Handle).GetPixel(x, y);
 
  // Access the second image
  var pixelValue2 = Image.FromHandle(multiPartImage.Parts[1].Handle).GetPixel(x, y);
}

Python

import cvb
 
multiPartImage, status, node_maps = stream.wait()
with image:
  # Access the first image
  pixel_value_1 = multiPartImage[0].get_pixel(cvb.Point2D(x, y))
 
  # Access the second image
  pixel_value_2 = multiPartImage[1].get_pixel(cvb.Point2D(x, y))

When to Use an Image Stream

It is recommended to use the Cvb::ImageStream whenever possible to simplify access to 1D or 2D images, provided the following is true:

The data stream returns supported 1D or 2D Image PFNC Type (see Supported Image Formats) or only these need to be processed.

For most simple devices, this will always be true. In the case of a device sending complex payloads (e.g. mixed image and point cloud data) the Composite Stream is a better option.

Point Cloud Stream

The Cvb::PointCloudStream is a specialization of the Cvb::CompositeStream that delivers the data received from a 3D acquisition device in the form of a Cvb::PointCloud. This stream type allows an indexed access to the different parts of the point cloud data (typically coordinates in x, y and z; potentially also additional data like confidence, color, or normal vectors). When using Point Cloud Streams to acquire from devices that supply multi-part data, the stream will just provide the first point cloud encountered in the returned composite.

C++C#Python

C++

using namespace Cvb;
 
PointCloudPtr pointCloud;
WaitStatus status;
NodeMapEnumerator nodeMaps;
std::tie(pointCloud, status, nodeMaps) = stream->Wait();
 
// Access the point cloud
auto numPoints = pointCloud->NumPoints();

C#

using Stemmer.Cvb;
using Stemmer.Cvb.Driver;
 
WaitStatus status;
 
using (var pointCloud = stream.Wait(out status))
{
  // Access the point cloud
  auto numPoints = pointCloud.NumPoints;
}

Python

import cvb
 
point_cloud, status, node_maps = stream.wait()
with point_cloud:
  # Access the point cloud
  num_points = point_cloud.num_points

When to Use a Point Cloud Stream

It is recommended to use the Cvb::PointCloudStream whenever possible to simplify access to single and multipart images provided the following is true:

The data stream returns exactly one supported 3D Point Cloud PFNC Type (see Supported Image Formats).
The data stream does not contain any non-3D Point Cloud information.

Composite Stream

If device provides mixed data types that cannot be handled by the Cvb::ImageStream or the Cvb::PointCloudStream, the Cvb::CompositeStream stream type should be used. It allows for any combination of data types and buffers at the cost of increased coding complexity.

C++C#Python

C++

using namespace Cvb;
 
CompositePtr composite;
WaitStatus status;
NodeMapEnumerator nodeMaps;
std::tie(composite, status, nodeMaps) = stream->Wait();
 
for (int i = 0; i < composite->ItemCount(); i++)
{
  auto component = composite->ItemAt(i);
  if (auto image = get_if<ImagePtr>(&component))
  {      
    // Process image
  }
  else if (auto pointCloud = get_if<PointCloudPtr>(&component))
  {      
    // Process point cloud
  }
}

C#

using Stemmer.Cvb;
using Stemmer.Cvb.Driver;
using Stemmer.Cvb.Runtime.InteropServices;
 
WaitStatus status;
using (var composite = stream.Wait(out status))
{
  foreach (var part in composite)
  {
    if (part.GetType() == typeof(Image)) {
      var image = Image.FromHandle(part.Handle, ShareObject.No);
      // Process image
    }
    else if (part.GetType() == typeof(PointCloud)) {
      var pointCloud = PointCloud.FromHandle(part.Handle, ShareObject.No);
      // Process point cloud
    }
  }
}

Python

import cvb
 
composite, status, node_maps = stream.wait()
with composite:
  for part in composite:
    if isinstance(part, cvb.Image):
      # Process image
    elif isinstance(part, cvb.PointCloud):
      # Process point cloud

When to Use Composite Stream

A Cvb::CompositeStream is more complex than a Cvb::ImageStream or Cvb::PointCloudStream and should only be used if the latter are not sufficient.

The data stream sends mixed data types.
The data stream sends unsupported PFNC Types (see Supported Image Formats).

Anatomy of a CVB Stream

Each device is capable of publishing one or more GenICam stream(s) that can each contain one or more GenTL flow(s), each of which contains a single data payload. The CVB acquisition stack supports both multi-flow streams and multi-stream devices or a combination thereof.

Internally, the CVB acquisition engine maps each GenTL flow onto a buffer, termed the CVB Flow. When a frame is received, this collection of CVB Flows is returned as a CVB Flow Set in the form of a Cvb::Composite where each CVB Flow maps to one of the components (e.g. Cvb::MultiPartImage). How many CVB Flow Sets can be in use at the same time and the memory location of the CVB Flow objects is determined by the Cvb::FlowSetPool object.

Note: The details of the CVB Flow and CVB Flow Set are hidden from the user; CVB provides interfaces to allow access to the CVB Flow data in a convenient manner (see Stream Type).

Flow Set Pool

The Cvb::FlowSetPool is a CVB construct that contains the CVB Flow buffers required by the acquisition engine to receive data. It is up to the acquisition engine to determine when and what buffer to use for a given GenTL component. If all buffers are in use, the acquisition engine will drop the incoming data. The count of flow sets in a pool should be well balanced between the minimum required for not dropping frames (camera frame rate) and the maximum possible for host side parallel processing capacity or storage limits. Increasing the amount of buffers will allow for more buffering for fast producer - slow consumer setups at the cost of higher memory usage. Reducing the amount of buffers will reduce the memory usage at the risk of dropped data if the consumer is unable to keep up.

Image lifetime

The CVB acquisition stack manages the lifetime of all image buffers automatically using a reference counting mechanism. This allows the acquisition engine to clear an image buffer when it is no longer being used and requeue it for a future acquisition cycles.

For the purposes of the reference counting mechanism, the returned Cvb::MultiPartImage/Cvb::PointCloud/Cvb::Composite and Cvb::NodeMapEnumerator are considered part of the same buffer. As long as they or any derived items remain in scope, the while buffer will be marked as in use and cannot be used for a future acquisition cycle.

There are a few things to consider when working with CVB to avoid unexpected problems:

• Keep the Cvb::MultiPartImagePtr, Cvb::CompositePtr or Cvb::PointCloudPtr in scope until it is no longer required. Otherwise, the acquisition engine will clear the buffer and the image will be unrecoverable.
  • In .NET, the image and node map enumerator implement the IDisposable interface and should be used in combination with a using statement to ensure correct object cleanup.
  • In Python, the image and node map enumerator implement a Context Manager and should be used in combination with a with statement to ensure correct object cleanup.
• If the image needs to be saved for processing that cannot be done in a single acquisition cycle, the image should be copied to another memory location. This will allow the acquisition engine to requeue the buffer and avoid frame drops when all buffers are unable to be freed.

Cancellation

The Cvb::CancellationToken gives the possibility to cancel individual wait calls instead of the whole acquisition. Restarting the whole acquisition takes much longer, than to simply call wait functions again. A use case for example would be if an external stop signal is received by the application, the fastest reaction to stop the acquisition and to restart it, would be to abort the wait function with a cancellation token. The token itself can be checked if it has been canceled. In addition, on cancellation, the returned Cvb::WaitStatus will have a value of Cvb::WaitStatus::Abort.

When using a cancellation token, the following rules should be observed:

• The cancellation of the acquisition may not occur immediately and if an image is received before the cancellation is processed, a race condition may occur, and the cancellation may not be seen. It is recommended to check the cancellation token even if an image has been received to properly act on it.
• If multiple waits are cancelled at the same time through a single token, they may finish in a random order.

Event Interface

The INotify interface enables notification of application for asynchronous driver events. It allows for the registration of user defined callbacks for asynchronous driver events. All currently supported events can be found at Cvb::DeviceState.

When using it the following rules apply:

• The thread context from which the callback is issued might vary between events and is not necessarily the main application process. It is driver (implementation) dependent whether callbacks are fired in parallel from multiple threads or sequentially from within the same thread context. Even though it is likely that a single callback registered on a single EventID on a single device is not called from multiple threads in parallel, it still valid. Therefore it is the caller's responsibility to ensure that the execution context of the registered notification is correct.
• Multiple callbacks can be registered to a single event. They will be executed in a random order.
• The callbacks are called from the thread context issuing the event. The execution time spent in the callback will possibly (depending on the event) affect image acquisition timing or driver behavior in general. It is recommended to return from a notification as quickly as possible.
• In order to avoid deadlocks it is not recommended to call driver functions from within a callback.

Examples

Example	Description
Single Stream	Simple acquisition of a single stream.
Multiple Streams	Acquisition of a device sending multiple data streams.
Multiple Components	Acquisition of a device sending a single data stream containing multiple components.
Resizing the Flow Set Pool	Customizing the Cvb::FlowSetPool before acquisition.
Using External Memory	Using a custom Cvb::FlowSetPool for acquisition.
Cancelling Acquisition	Using a Cvb::CancellationToken to stop an ongoing acquisition.
Using Event Callbacks	Using a the Cvb::NotifyObservable interface to handle event callbacks.
Point Cloud Acquisition (C++)	Acquiring point clouds coming along with additional data like RGB images.

Further Reading

Getting started with C++
Getting started with .NET
Getting started with Python

CVB Image Data Handling
Pixel Format Conversion - PFNC Formats and Bayer Conversion

Analyze your Composite in C++, .NET and Python
Access to Composites
Access to Simple Point Clouds

VinDevice and VinBuffer Nodemaps
Migrating from the 2nd Gen acquisition stack