Common Vision Blox 15.0
Metric

Common Vision Blox Foundation Package Tool


C-Style

C++

 .Net API (C#, VB, F#) 

Python
 CVMetric.dll   Cvb::Foundation::Metric   Stemmer.Cvb.Foundation.Metric   cvb.foundation 

Introduction

Many image processing applications require calibrations in order to be able to measure features. This applies to both, 2D and 3D applications. The Metric Tool of Common Vision Blox currently provides functionality for the calibration of 3D sensors. In detail, the calibration of laser triangulation systems can be determined (intrinsic calibration). Furthermore, all pre-calibrated 3D sensors can be transformed into a desired world coordinate system (extrinsic calibration).

Laser Triangulation

The principle of laser triangulation is shown in the figure below. A line laser emits light to provide a visible line on an object. The profile of the laser line is acquired by a camera that is mounted in a fixed angle to the laser. The shape of the object's surface will define the position where the laser line is captured in the camera sensor. From the distinct pixel positions of the laser line in the image a height profile can be calculated with relative height information. However, without knowing the exact positions and orientations of laser and camera it is not possible to refer these pixel positions to real metric coordinates. Therefore, it is necessary to previously calibrate the triangulation system with a calibration target with known coordinates. By continuously moving the object under the laser-camera-system multiple profiles can be generated and accumulated in one height image. We call those images range maps. The coordinate system of a point cloud created from a range map of a laser triangulation system is defined as follows:

laser triangulation setup range map point cloud
lateral position on laser line

number of column in range map

image coordinate u of camera frame

x-coordinate
position in the direction of movement number of row in range map y-coordinate
position in the direction of the laser projection

grey value in range map

image coordinate v of camera frame

z-coordinate

The columns in the range map (x coordinate) correspond to the columns of the raw sensor image (u coordinate). The lines indicate the n'th profile in traverse direction (y direction) and the intensity value represents  the v coordinate of the laser peak within the frame.

In order to get 3D world coordinates, the relationship between range map pixels and metric units have to be determined.

Therefore the relationship between pixel coordinates (in the sensor plane of camera) and coordinates of the laser plane, the so-called intrinsic parameters have to be estimated first.
Second, the relationship between laser plane and world coordinates, the so-called extrinsic parameters have to be determined.

Laser Triangulation

Theory of operation

The calibration of imaging devices can be divided into two parts: the intrinsic and the extrinsic calibration. The theory behind is described in the section Metric conversion.

The section How to calibrate with CVB explains how to estimate the calibration parameters with Metric using a suitable calibration target.

Metric conversion

The metric conversion can be divided into two parts: The intrinsic and the extrinsic calibration.

  • The intrinsic calibration describes the relationship between camera pixel coordinates u and v and laser plane (xz plane).
  • The extrinsic parameters represent the transformation from the laser plane to world coordinates x,y and z.

The figure below gives an overview of the intrinsic and extrinsic calibration parameters. Parameters highlighted in green can be calculated with CVB. The estimation of parameters highlighted in grey is not implemented yet. CVB functions, which calculate the according parameters are marked with a yellow star. The single intrinsic and extrinsic parameters are described in detail in the following sections.

Intrinsic calibration

In order to get highly accurate coordinates lens distortion  (e.g. Brown-Conrad, cubic or polynomial) and laser line distortion (2nd, 3rd, or 4th order polynomial) should be estimated. The calculation of these parameters will be implemented in a future CVB release. For 3D compact sensors with built in lasers usually the correction of lens and laser line distortion is provided and can be used before calibrating with CVB.

The relationship between camera pixel coordinates u and v and laser plane (xz plane) is described by the homography H (3x3 homogenous matrix). The homography can be calculated with the CVB function CVMAQS12CreateIntrinsicCalibratorFromPiece. After applying the homography the x and z coordinates of the laser plane are metric. In order to transform the range map rows to metric y coordinates, the velocity of the conveyer belt and the frame rate of the camera (encoder step) must be known.

If the xz coordinates of the laser plane are highly accurate, however the accuracy of the measurement can be reduced by an inclined laser plane. If the laser plane is not exactly vertical to the moving direction (y axis), additional errors are induced. A rotation of the laser plane about the x axis causes shear in yz and scaling in z. In the figure below the acquisition of a cuboid by a 3D laser triangulation system is shown (side view). The errors induced by an inclined laser plane are drawn schematically.

Inclination of laser plane in YZ

If the laser plane is rotated about the z axis, shear in yx and scaling in x is caused. In the figure below again the acquisition of a cuboid is shown, but from a different view point (view from above). The errors caused by the inclination are drawn in the range map schematically.

Inclination of laser plane in YX

These errors can be corrected by an affine transformation. The affine transformation can be calculated by the CVB function CVMAQS12CreateIntrinsicCalibratorFromPiece or - if homography is already known - by CVMAQS12CalculateCorrectionOfLaserPlaneInclinationFromPiece. Note, that the affine transformation also estimates scaling in y, i.e. the encoder step does not have to be known. After the affine transformation all coordinates are metric.

Extrinsic calibration

For the extrinsic calibration a rigid body transformation (rotation matrix and translation vector) is conducted by the CVB function CVMAQS12CalculateRigidBodyTransformationFromPiece. The extrinsic calibration can only be applied if the range map is already calibrated by the intrinsic parameters, which possibly applies for compact sensors, where the distance and the angle between camera and laser line is fixed. In addition the laser plane xz should be just vertical to the moving direction (y axis). If this cannot be guaranteed, it is recommended to use CVMAQS12CalculateCorrectionOfLaserPlaneInclinationFromPiece in order to estimate an affine transformation correcting errors induced by an inclined laser plane.

Calibration Targets

In order to estimate the intrinsic and extrinsic parameters, a calibration target, where the reference points are precisely known is necessary. CVB currently supports a diamond shaped calibration pattern (AQS12) for both intrinsic and extrinsic calibration. The AQS12 is described in detail in the next section.

AQS12 The AQS12 has a distinct shape with 8 planes and 12 feature points resulting from the intersection points of these planes. The overall size of the target can vary, however, the proportions of the pattern are important for the internal algorithm, as well as the order of the points located on the pattern.

AQS12 - calibration pattern

The dimensions of the calibration target should be similar to the example shown in the figures below. In any case it should meet the following requirements:

  • The base plane has to contain enough area: The number of pixels on the base plane has to be (significantly) larger than the number of pixels on the roof.
  • Points 1-6 and 7-12 respectively have to lie on the same plane. Base plane and roof have to be parallel. The shape of the object may not be changed, i.e. parallel lines must remain parallel, and the object has to be symmetric.
  • Calibration points 1, 8 and 12 projected to the base plane must lie on a line, same for 4, 9 and 11.
  • In order to get accurate calibration results, the surfaces should be non-specular.
  • Check, if the slopes of the calibration target can be acquired under the triangulation angle needed for your application.

Note, that the x-axis in the range map represents the x-axis of the camera frame. The y-axis is the traverse direction. The z-axis should be perpendicular to the base plane and represents the height of the object.

In addition the following specifications should be considered during the acquisition:

  • The calibration target has to be moved at constant speed or synchronized with the camera by an encoder, so that the distances between successive profiles are always equal.
  • The image used for the calibration should contain only points belonging to the calibration target. In case the acquired range map shows more than the pattern, use functions CVMAQS12CreateIntrinsicCalibratorFromPieceRect, CVMAQS12CalculateCorrectionOfLaserPlaneInclinationFromPieceRect and CVMAQS12CalculateRigidBodyTransformationFromPieceRect, where an area of interest can be specified.
  • The axis on which the points 7, 1, 4 and 10 are located, should be (more or less) parallel to the motion direction. Since CVB release 13.3, the target may be rotated clockwise about 90° (point 7 right, point 10 left). In this case points 7, 1, 4 and 10 have to be more or less parallel to the x axis.
  • The z-axis of the calibration target should be perpendicular to the motion plane.
  • The range map acquired from the AQS12 shape should definitely include the base plane (i. e. the points on the base plane should be neither 0 nor whatever value the actually used 3D sensor defines for invalid points). The distance from the AQS12 corner points to the base plane are relevant piece of information that will be used by calibration algorithm.

Example dimensions

Top view

CalItemDesign1

Side view (long side)

CalItemDesign3

Side view (short side)

CalItemDesign2

The world coordinates of the reference points for the example shown in the figure above are listed in the table below:

Point X [mm] Y [mm] Z [mm]
    right-handed left-handed  
1 0 110 40 60
2 10 96.667 53.333 60
3 10 53.333 96.667 60
4 0 40 110 60
5 -10 53.333 96.667 60
6 -10 96.667 53.333 60
7 0 150 0 30
8 30 110 40 30
9 30 40 110 30
10 0 0 150 30
11 -30 40 110 30
12 -30 110 40 30
Height 60      

The first column indicates the x, second  the y and third the z coordinate. Mind the order of the points (see Figure above). The single value in the last line represents the height distance between the roof and the base plane. Note, that this value is optional. With this value the height (z value) of the base plane is computed. If it is missing, the height of the base plane is supposed to be zero.

Note, that the vertical walls of the calibration target are not required. The calibration target may also have only the base and the top plane and 6 faces (see figure below).

Calibration pattern without walls

CVB 13.3 provides functions, which return the segmented label image of the AQS12 (CVMAQS12CreateFaceSegmentationImageFromRangeMap, CVMAQS12CreateFaceSegmentationImageFromRangeMapRect, CVMAQS12CreateFaceSegmentationImageFromDensePointCloud, and CVMAQS12CreateFaceSegmentationImageFromDensePointCloudRect) or the extracted points (CVMAQS12ExtractProjectedPointsFromRangeMap, CVMAQS12ExtractProjectedPointsFromRangeMapRect, CVMAQS12ExtractProjectedPointsFromDensePointCloud, and CVMAQS12ExtractProjectedPointsFromDensePointCloudRect). This can be useful, if the user wants to check intermediate results, like the face segmentation or the point extraction.

Note, that the world coordinate system is defined by the given reference coordinates (see table above). By default a calibrated point cloud should be in a right-handed trihedron. But, in some cases, the user likes to do further image processing on a rectified point cloud. Then it is recommended to transform the point cloud to a left-handed trihedron, as the image coordinates are left-handed, too. If he rectifies a right-handed point cloud, the y coordinates will be mirrored in the range map. The figure below shows the calibrated and rectified point clouds in a right- and left-handed trihedron. Please note that in the 3D images in the center column the axis indicators are not located at the actual origin positions for better visibility! The actual origin position - in accordance with the table above - is the point right below corner #7 for the left-handed coordinate system and the point below corner #10 for the right-handed coordinate system.

How to calibrate with CVB

For the estimation of the calibration parameters with CVB, a suitable calibration target with precisely known dimensions is needed. Until now only the diamond shaped AQS12 is supported (see section above). The CVB calibration functions need as input the accurate coordinates of the feature points (reference) and a range map of the AQS12 (and possibly known intrinsic parameters). First the faces of the AQS12 are segmented on the range map using the kmeans as clustering algorithm. After that a plane is fitted to each face and the intersection point between three neighboring planes is computed. With the intersection points and the precise reference points an affine transformation is estimated. In addition the residuals are returned in each calibration function. The residuals are a good indicator for the accuracy which can be achieved by the calibration. Note that, the residuals relate only to the area of the AQS12. Acquisitions far from the target may be more erroneous, as residual errors propagate with an increasing distance to the calibration target.

The two figures below show the metric functions calculating the intrinsic and extrinsic calibration. In the boxes the internal calculation is described. Several steps can also be calculated separately since CVB release 13.3 (see functions marked with a star). With them, the user can check intermediate results like face segmentation and point extraction in case of an unsuccessful calibration.

In section Calibration Setup typical acquisition scenarios and their solution with CVB are described.

Note, that the calibration functions of release 13.3 and higher expect two object handles which have to be created before. One is needed for the segmentation and one for the intrinsic calibration:

Calibration Setup - Use Cases

There are several possibilities to acquire accurate x,y,z coordinates of objects with 3D sensors. The individual calibration workflow you should follow depends on the device chosen and the setup of the camera and the laser. E.g. Area 3D compact sensors already provide a point cloud in metric units. In this case only an extrinsic calibration which moves the points to a world coordinate system is required. Other than with 3D compact sensors with built in laser which usually provide high precision intrinsic calibration parameters (lens and laser line correction and homography). But here in most cases, it cannot be guaranteed, that the laser plane is exactly vertical to the direction of movement. Therefore the estimation of an affine matrix (which corrects errors induced by an inclined laser plane) is necessary. Laser triangulation systems with individual components (camera and laser separated) need both: intrinsic calibration parameters (like homography) and the correction of the laser plane inclination.

The following table shows three typical use cases. They are described in detail below.

Use Case Prerequisites Calibration with CVB and Typical Setup

Intrinsic calibration parameters

(homography, lens correction,...)

encoder step Laser plane is exactly vertical aligned to the direction of movement
1 NO NO NO

CVMAQS12CreateIntrinsicCalibratorFromPiece

-> homography

->  affine transformation (rotation, translation, scaling and shear) corrects the laser plane inclination

Typical Setup:

  • Modular setup
2 YES NO NO

CVMAQS12CalculateCorrectionOfLaserPlaneInclinationFromPiece<p>->  affine transformation (rotation, translation, scaling and shear) corrects the laser plane inclination

Typical Setup:

  • Compact sensors
  • Modular setup, where intrinsic calibration is given (e.g. zigzag calibration)
3 YES YES YES

[CVMCalculateRigidBodyTransformation]

CVMAQS12CalculateRigidBodyTransformationFromPiece

->  transformation matrix (only rotation and translation)

Typical Setup:

  • Area 3D compact sensors
  • Compact sensors, where the laser plane is exactly vertical to the direction of movement and the encoder step is known.

Use Case 1:

  • Device: 2D/3D camera and laser are individual components.
  • Intrinsic Calibration: Calibration parameters are unknown for this setup.
  • Setup: Laser is mounted in a way that the laser plane is NOT vertically aligned to the direction of movement.
  • Y-scale: The resolution of two consecutive profiles does not have to be known (e.g. from the encoder step).

Workflow:

If all calibration parameters are unknown, use CVMAQS12CreateIntrinsicCalibratorFromPiece. This function estimates homography and an affine transformation calculating scale in moving direction and correcting possible errors induced by an inclined laser plane. All coordinates (x, y and z) are metric after the transformation.

Use Case 2:

  • Device: Laser triangulation compact sensor (with built in laser)
  • Intrinsic Calibration: Calibration parameters are provided by the manufacturer  (lens and laser line correction, homography, etc...).
  • Setup: Sensor is mounted in a way that the laser plane is NOT vertically aligned to the direction of movement.
  • Y-scale: The resolution of two consecutive profiles does not have to be known (e.g. from the encoder step).

Workflow:

Applying the intrinsic calibration parameters leads to metric x and z coordinates. In order to correct errors induced by the inclined laser plane, an affine transformation should be calculated using CVMAQS12CalculateCorrectionOfLaserPlaneInclinationFromPiece. With the affine transformation also a scale factor in y is included. All coordinates (x, y and z) are metric after the transformation.

Use Case 3:

  • Device: Area 3D compact sensor, which provides metric coordinates in x,y,z.

Workflow:

Area 3D compact sensors provide point clouds with metric x,y and z coordinates. They usually have to be transformed to a given world coordinate system, which can be done via a rigid body transformation. The target coordinate system is given by the reference points of the AQS12. CVB function CVMAQS12CalculateRigidBodyTransformationFromPiece estimates a rigid body transformation including only rotation and translation.

Use Case 3b: Note, that this workflow is only recommended, if the laser plane is exactly vertical to the direction of movement! If you cannot ensure that, please apply use case 2!

  • Device: Laser triangulation compact sensor (with built in laser)
  • Intrinsic Calibration: Calibration parameters are provided by the manufacturer (lens and laser line correction, homography, etc...).
  • Setup: Sensor is mounted in a way that the laser plane is vertically aligned to the direction of movement.
  • Y-scale: The resolution of two consecutive profiles is known, e.g. from the encoder step.

Workflow:

In this case after applying the given calibration parameters the range map values are already transformed to metric x, y and z coordinates. As the laser plane is just vertical to the moving direction, shear in y is not expected. Therefore  a simple rigid body transformation can be done with::CVMAQS12CalculateRigidBodyTransformationFromPiece. The rigid body transformation estimates a rotation matrix and translation vector and moves the target to the desired coordinate system given by the reference points of the AQS12.

Validation of calibration

Sometimes the calibration is not successful or the results are not as accurate as hoped for. There a several reasons for that. The figure below shows the user, what he can do in order to improve his results.

If the calibration ran without an error, he should check the residuals first. If the residuals are smaller than the desired accuracy, the calibration was successful. But attention, the residuals relate only to the area of the AQS12! Acquisitions far from the target may be more erroneous, as residual errors propagate with an increasing distance to the calibration target. Therefore it is very important, that the area where objects will be measured in future is (more or less) completely covered by  the calibration piece.

If the residuals are too high, you should check the following:

  1. Check, if the extraction of the AQS12 intersection points and the face segmentation is correct. Therefore you can use the CVB functions CVMAQS12ExtractProjectedPointsFromRangeMap, CVMAQS12ExtractProjectedPointsFromDensePointCloud, CVMAQS12CreateFaceSegmentationImageFromRangeMap, and CVMAQS12CreateFaceSegmentationImageFromDensePointCloud (since CVB release 13.3). The function CVMAQS12CreateFaceSegmentationImageFromRangeMap or CVMAQS12CreateFaceSegmentationImageFromDensePointCloud return an image with labels for the segmented faces of the range map. If the labeling is completely wrong, this might be due to background, which is not eliminated during the segmentation. In this case use CVB functions, where you can specify an AOI (see <function name>[Rect]()). If the labeling is more or less correct, but many falsely classified pixels remain, you should reconsider your setup. Maybe due to a flat inclination, the target is occluded at some points. Or the acquisition is too noisy to segment faces correctly.
  2. Check, if the given reference points are correct. It is very important, that the sequence of the points is correct (see Calibration Targets). Note, that the AQS12 is symmetric in principle. But if the target is not accurately produced and smaller deviations from the symmetry are present, it is fundamental, that you do not swap opposing points.
  3. Last you have to be really really sure, that your target and the measured reference points are precise enough. Your calibration results will never get better, than the accuracy of your reference points. The AQS12 has to be of hard and temperature-stable material (like aluminum). If you have precisely measured the reference points of the AQS12, but your target deforms with variation in temperature, the calibration will not lead to satisfying results.