CVB++ 15.0
Efficiently write image and point cloud algorithms using Visit

Point cloud and image visit

Writing algorithms on top of Images or PointClouds often comes with hurdles, like having to deal with multiple data layouts (Vpat, Linear or Contiguous, interleaved planes or planar ones) or datatypes. To reduce the overhead of handling these cases, CVB++ brings infrastructure for modern C++, leading to algorithms only having to be written once with the most efficient access (contiguous > linear > vpat) being used automatically.

The typical usage pattern might look like in the following example:

#include <cvb/block.hpp>
struct DivideByTwo { // (1)
template <class T, class ACCESS_TRAIT>
void operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept { // (2)
for(int y = 0; y < block.Height(); ++y) {
for(int x = 0; x < block.Width(); ++x) {
block(x,y) /= 2; // (3)
}
}
}
};
Cvb::Image img = ...;
Cvb::Visit(DivideByTwo{}, img->Plane(0)); // (4)
Non-owning view on a 2d-plane of data.
Definition: decl_block.hpp:23
The Common Vision Blox image.
Definition: decl_image.hpp:45

At the beginning of the snippet a function object (1) is declared: DivideByTwo implements the actual algorithm in operator() (2). Instead of a struct/class with a call operator, a C++ lambda with auto block as parameter can be used in all cases.

The algorithm, in this case dividing every pixel value by 2, gets access to the image plane through the block parameter. The pixel values can be read as well as changed by using the 2D-coordinate access as seen in line (3).

As can be seen in line (2), the Block is templated by the ImagePlane's DataType T and the ACCESS_TRAIT, which represents the vpat / linear / contiguous access. This way no manual reinterpretation of the pixel values is necessary.

The invocation of the algorithm is performed in line (4), through the call to Cvb::Visit, where the first parameter, is the function object and the later parameters, the plane(s) to work on.

Remarks
As the template parameters of the Block have to be known at compile-time and the actual data type and layout of the plane, are only known at run-time, the call operator of DivideByTwo is instantiated for each supported DataType ([u]int8_t, [u]int16_t, [u]int32_t, [u]int64_t, float, double) and access trait (Vpat, LinearAccessData, ArrayAccess) by the Cvb::Visit. This also implies, the call operator must compile for all of those data types.

The template parameters of the Block also allow specializing the algorithm for certain data types T. This just requires partially specializing the call operator for the specified type. SFINAE can of course also be used to select specialized functions.

// algorithm specialization by partial template specialization
struct DivideByTwo {
template <class T, class ACCESS_TRAIT>
void operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept { ... }
template <class ACCESS_TRAIT>
void operator()(Cvb::Block<std::uint8_t, ACCESS_TRAIT> block) const noexcept { ... }
};
// algorithm specialization by SFINAE
struct DivideByTwo {
template <class T, class ACCESS_TRAIT>
std::enable_if_t<!std::is_unsigned<T>, void> operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept { ... }
template <class T, class ACCESS_TRAIT>
std::enable_if_t<std::is_unsigned<T>, void> operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept { ... }
};

Visiting PointClouds with point types

To further simplify development of PointCloud algorithms, multiple planes can be visited simultaneously.

struct NormalizePointCloud {
template <class T, class ACCESS_TRAIT>
void operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept {
for(int y = 0; y < block.Height(); ++y) {
for(int x = 0; x < block.Width(); ++x) {
decltype(auto) point = block(x,y); // (1)
const auto length = std::sqrt(point.X()*point.X() + point.Y()*point.Y() + point.Z()*point.Z());
point.X() /= length; // (2)
point.Y() /= length;
point.Z() /= length;
}
}
}
};
Cvb::PointCloud pcl = ...;
Cvb::Visit(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2)); // (3)
A point cloud object.
Definition: decl_point_cloud.hpp:49
PlanePtr Plane(int index) const
Index based plane access.
Definition: decl_point_cloud.hpp:633

In this example, the three visited PointCloud planes are condensed into a single block parameter. By doing so, the type T of the Block, is no longer the plain Planes' DataType, but instead depending on whether the three planes are interleaved or not, T is a Cvb::LinearValue<PT, N> or Cvb::RefValue<PT, N>. N in this case is 3, as three Planes are visited, PT the Planes' DataType. These two wrapper types allow reading and writing multi-dimensional points directly without caring about the data layout. It is important to use decltype(auto) as seen in line (1) when getting the value from the Block. When assigning a value to point.X() or the subscript operator point[0], it is directly written to the corresponding Plane's memory.

Remarks
When visiting PointCloud Planes, the call operator is only instantiated with float, double and int as possible Plane DataTypes and as PointCloud Planes do not support Vpat access, the ACCESS_TRAIT parameter is only instantiated with LinearAccessData and ArrayAccess, as well as ScatterAccess, if more than one Plane is visited, to support the case when those are not interleaved. This implies, the call operator must be valid for all three data types.

Visiting PointClouds with specified types

As noted above, the function object's call operator is always instantiated for every combination of DataTypes and access traits. This might be unnecessary, if the Plane DataType is already known. In this case VisitAs can be used for a data type specific algorithm that works independent of the underlying data layout.

The two examples from above could look like this, if the type is known:

struct NormalizePointCloud {
template <class T, class ACCESS_TRAIT> // (1)
void operator()(Cvb::Block<T, ACCESS_TRAIT> block) const noexcept { ... }
};
Cvb::PointCloud pcl = ...;
Cvb::VisitAs<float>(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2)); // (2)
struct DivideByTwo {
template <class ACCESS_TRAIT>
void operator()(Cvb::Block<std::uint8_t, ACCESS_TRAIT> block) const noexcept { ... } // (3)
};
Cvb::Image img = ...;
Cvb::VisitAs<std::uint8_t>(DivideByTwo{}, img->Plane(0)); // (4)

In both examples the type of the planes are given via the template parameter to VisitAs in lines (2) and (4). In the PointCloud example, where multiple planes are visited with a single block, this does not allow replacing the T template parameter of the Block in line (1), as this still may either be LinearValue<float, 3> or RefValue<float, 3>. In the Image example, where a single plane is visited by a single block, the type T known to be uint8_t.

Remarks
If the specified type is arithmetic and does not match the Planes' data type, VisitAs will throw a std::domain_error. Visit and VisitAs will both throw a std::domain_error, if the visited planes have differing data types or sizes.

More cases supported by Visit/As

Visiting multiple planes with multiple blocks

In the above examples, it has already been shown that both, Visit and VisitAs, do support visiting either one plane and having a function object with one block, as well as visiting multiple planes with one block. Visit/As both support a generalized variant of the 1 plane, 1 block case: visiting multiple planes with a function object that has the same number of block arguments, as planes are visited:

struct NormalizePointCloud {
template <class T, class ACCESS_TRAIT>
void operator()(Cvb::Block<T, ACCESS_TRAIT> block0,
Cvb::Block<T, ACCESS_TRAIT> block2) const noexcept { ... }
};
Cvb::PointCloud pcl = ...;
Cvb::Visit(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2));
Cvb::VisitAs<float>(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2));

In this case, the block arguments correspond to the visited planes from left-to-right. All three blocks are of the same type T and use the same access trait type ACCESS_TRAIT (the most general supported by all the Planes). The planes may be of differing sizes in this case, the blocks will report the corresponding one. For the VisitAs case, the template parameter T would be known to be float, as in the 1 plane, 1 block case from above.

Visiting with custom types

In addition to specifying the arithmetic plane type with VisitAs, it also supports specifying custom complex types. This can be useful for visiting pointclouds with custom point types, for which arithmetic operators and custom operations are implemented. Supported examples are CVB++'s point types, like Cvb::Point3D. User defined types can be supported as well, as a starting point, everything in the region Point3D in cvb/_detail/block/detail_block_point_helper.hpp may be copyied in user's code and adapted to their custom types. Note, that the get, set and internal_set functions have to reside in the same namespace as the custom type. The ComponentOf and NumComponentsOf code has to live in the same namespace as the CVB++'s API, as outlined here:

namespace CUSTOM_NAMESPACE {
// specialize get, set, internal_set here.
}
namespace Cvb {
CVB_BEGIN_INLINE_NS
using CUSTOM_NAMESPACE::get;
using CUSTOM_NAMESPACE::set;
// the ComponentOf and NumComponentsOf specializations go here
CVB_END_INLINE_NS
}
Root namespace for the Image Manager interface.
Definition: c_barcode.h:24

There are two variants how custom types can be used:

  1. specifying a fully specialized type (e.g. Cvb::Point3D<float>)
  2. specifying an unspecialized template (e.g. Cvb::Point3D<T>, where T will be dispatched according to the plane data type as usual).
struct NormalizePointCloud {
template <class T, class ACCESS_TRAIT>
void operator()(Cvb::Block<Cvb::Point3D<T>, ACCESS_TRAIT> block) const noexcept {
for(int y = 0; y < block.Height(); ++y) {
for(int x = 0; x < block.Width(); ++x) {
auto point = block(x,y); // (1)
point /= point.Length(); // (2)
block.Set(x,y,point); // (3)
}
}
}
};
Cvb::PointCloud pcl = ...;
Cvb::VisitAs<Cvb::Point3D<float>>(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2)); // (4)
Cvb::VisitAs<Cvb::Point3D>(NormalizePointCloud{}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2)); // (5)
Multi-purpose 3D vector class.
Definition: point_3d.hpp:22

The above example shows both variants being used in lines (4) and (5). This version of NormalizePointCloud shows in line (2) how much shorter the algorithm can be by using custom types. One drawback of using fully custom types leads to lines (1) and (3) being slightly different to the above examples: The values of the points are no longer fully in-place. Thus decltype(auto) is not required anymore and changing point has no effect on the actual planes' values. To commit changes to the planes, it is now necessary to call the Block::Set function.

Overview of the Visit/As variants

The following table presents an overview of the variants supported by the Visit/As functions and their corresponding behaviour.

Function PC/Img Template argument # of FunctorArgs # of Planes Type Dispatch Size Checked Vpat plane Linear plane Contiguous plane Non-interleaved planes
Visit PC N/A 1 3* Yes Yes N/A LinearPlaneBlock<LinearValue<T, 3>> ArrayPlaneBlock<LinearValue<T, 3>> ScatterBlock<RefValue<T, 3>, 3>
Visit PC N/A 3 3 Yes No N/A LinearPlaneBlock<T>,… ArrayPlaneBlock<T>,… N/A
Visit Img N/A 1 3 Yes Yes N/A LinearPlaneBlock<LinearValue<T, 3>> ArrayPlaneBlock<LinearValue<T, 3>> ScatterBlock<RefValue<T, 3>, 3>
Visit Img N/A 3 3 Yes No VpatPlaneBlock<T>,… LinearPlaneBlock<T>,… ArrayPlaneBlock<T>,… N/A
VisitAs PC float 1 3 No Yes N/A LinearPlaneBlock<LinearValue<float, 3>> ArrayPlaneBlock<LinearValue<float, 3>> ScatterBlock<RefValue<float, 3>, 3>
VisitAs PC float 3 3 No No N/A LinearPlaneBlock<float>,… ArrayPlaneBlock<float>,… N/A
VisitAs PC Point3D<float> 1 3 No Yes N/A LinearPlaneBlock<Point3D<float>> ArrayPlaneBlock<Point3D<float>> ScatterBlock<Point3D<float>, 3>
VisitAs PC Point3D<float> 3 3 No No N/A LinearPlaneBlock<Point3D<float>>,… ArrayPlaneBlock<Point3D<float>>,… N/A
VisitAs PC template Point3D 1 3 Yes Yes N/A LinearPlaneBlock<Point3D<T>> ArrayPlaneBlock<Point3D<T>> ScatterBlock<Point3D<T>, 3>
VisitAs PC template Point3D 3 3 Yes No N/A LinearPlaneBlock<Point3D<T>>,… ArrayPlaneBlock<Point3D<T>>,… N/A
VisitAs Img uint8_t 1 3 No Yes N/A LinearPlaneBlock<LinearValue<uint8_t, 3>> ArrayPlaneBlock<LinearValue<uint8_t, 3>> ScatterBlock<RefValue<uint8_t, 3>, 3>
VisitAs Img uint8_t 3 3 No No VpatPlaneBlock<uint8_t>,… LinearPlaneBlock<uint8_t>,… ArrayPlaneBlock<uint8_t>,… N/A
VisitAs Img Point3D<std::uint8_t> 1 3 No Yes N/A LinearPlaneBlock<Point3D<uint8_t>> ArrayPlaneBlock<Point3D<uint8_t>> ScatterBlock<Point3D<uint8_t>, 3>
VisitAs Img Point3D<std::uint8_t> 3 3 No No VpatPlaneBlock<Point3D<uint8_t>>,… LinearPlaneBlock<Point3D<uint8_t>>,… ArrayPlaneBlock<Point3D<uint8_t>>,… N/A
VisitAs Img template Point3D 1 3 Yes Yes N/A LinearPlaneBlock<Point3D<T>> ArrayPlaneBlock<Point3D<T>> ScatterBlock<Point3D<T>, 3>
VisitAs Img template Point3D 3 3 Yes No VpatPlaneBlock<Point3D<T>>,… LinearPlaneBlock<Point3D<T>>,… ArrayPlaneBlock<Point3D<T>>,… N/A

*3 is only used as an example here, as the exemplary Point3D is a fit for 3 planes.

The first 5 columns are used to select the relevant line, the other columns describe the resulting behaviour of the used variant. An example to illustrate the interpretation of the table:

Cvb::PointCloud pcl = ...;
Cvb::Visit([](auto block) {..}, *pcl->Plane(0), *pcl->Plane(1), *pcl->Plane(2)); // (1)
Cvb::Image img = ...;
Cvb::VisitAs<Cvb::Point3D>([](auto block0, auto block1, auto block2) {..}, img->Plane(0), img->Plane(1), img->Plane(2)); // (2)

For the Visit call in line (1), the first line in the table is relevant: Function is Visit, 3 pointcloud planes are visited but the lambda only has a single block parameter. In this case, type dispatch has to happen, i.e. the lambda is instantiiated for all supported data types - float, double and int. As the table suggests, all planes must have the same size, otherwise an exception is thrown. The last four columns indicate what types block will be instantiated with. T indicates the element type, the number of planes 3 is just an example. As pointcloud planes are used here, no VpatPlaneBlock instantiations happen. As a single block is used to access all planes, LinearPlaneBlock, ArrayPlaneBlock or ScatterBlock may be used and thus lambda instantiations for all of these happen.

The VisitAs call in line (2) matches the last line, where the function is VisitAs, visiting 3 image planes with the templated type Point3D. The functor has 3 block arguments. Given this information, the table can be used to see that type dispatch has to happen as the T in Point3D<T> has to be set to the element type. In this case, the planes and thus blocks may have differing sizes, thus no exception will be thrown. The last four columns indicate that all blocks have the same type (e.g. VpatPlaneBlock<Point3D<T>>), but no ScatterBlock will be used, as each planes' layout is represented in its own block.

pclImageVisitCustomPlanes

Using Visit and VisitAs is also possible with custom plane types. To enable this, the PlaneTraits struct has to be specialized for the custom plane type. PlaneTraits implements the following concept:

template<>
struct PlaneTraits<CustomPlane> {
using PlaneT = PLANE_T;
using TypeList = DispatchableTypeList<...>;
static constexpr bool HasVpat = ..;
static int GetWidth(const PlaneT &plane);
// Throws if GetRank < 2.
static int GetHeight(const PlaneT &plane);
static Cvb::DataType GetDataType(const PlaneT &plane);
static int GetRank(const PlaneT &);
// only available if HasVpat == true:
static Cvb::Vpat GetVpat(const PlaneT &plane);
// only available if HasVpat == false:
static std::ptrdiff_t GetXInc(const PlaneT &plane);
// only available if HasVpat == false:
static std::ptrdiff_t GetYInc(const PlaneT &plane);
// only available if HasVpat == false:
static std::uint8_t* GetBasePtr(const Plane &plane)
};
Data type description for an image plane.
Definition: data_type.hpp:28
Virtual Pixel Access Table.
Definition: decl_vpat.hpp:24

Two types of planes are supported: those that have at least a linear layout and those that might have a non-linear layout that can be represented by a Vpat. In the latter case, the PlaneTraits<CustomPlane>::HasVpat value is set to true and the GetVpat function provides a Cvb::Vpat. For the linear case, HasVpat is false and GetBasePtr returns the pointer to the first element, GetXInc and GetYInc provide the byte increments for a x and y steps.

The data types, the plane supports and thus are dispatched are specified via the TypeList alias. The Cvb::Plane e.g. has using TypeList = DispatchableTypeList<float, double, int>; leading to instantiations for those three types. The types in the template argument list of DispatchableTypeList have to be representable by Cvb::DataType.

Example specializations can be found for Plane and ImagePlane in the cvb/plane.hpp and cvb/_decl/decl_image_plane.hpp headers.

Note, that the specialization has to reside in the same namespace as CVB++'s functions. So it should be wrapped in

namespace Cvb {
CVB_BEGIN_INLINE_NS
// the specialization goes here
CVB_END_INLINE_NS
} // namespace Cvb