Stereo calibration with partially detected checkerboard patterns

Hello, I have a particular application in which I need to perform a stereo calibration, but taking each calibration image is "very expensive". For this reason, I want to be able to get the most value out of each image, even those in which only part of the calibration pattern is visible.

• Is there a correct workflow to perform stereo calibration using partially detected checkerboard patterns?

This is what I tried:

The (mono) camera calibrator can use partially detected targets to calculate camera intrinsics using estimateCameraParameters for one camera.

The stereo camera calibrator (again, using estimateCameraParameters) instead drops image pairs in which the target is only partially visible (thus "wasting" the information contained within those discarded pairs).

Instead, I tried using stereo camera calibrator with fixed intrinsics (which uses estimateStereoBaseline), providing the intrinsics calculated by the mono camera calibrator on the full image set. This however seems to lead to worse results, with much higher mean reprojection errors.

• Is there a difference in the calibration algorithms used by estimateCameraParameters and estimateStereoBaseline that could justify these worse results?

This was also tested using a dataset that only contains fully-visible targets (see code below), which again confirmed worse mean reprojection errors when using fixed intrinsics.

My next step has been to try to modify the source code of estimateCameraParameters to allow the function calibrateTwoCameras to use the full image dataset (including imagePoints that contain NaNs). I know that the call to refine() will not work if I pass imagePoints that contain NaNs (It generates the following error: "Error using lsqncommon: Objective function is returning Inf or NaN values at initial point. lsqcurvefit cannot continue."), but I thought about using the full dataset only for each call to calibrateOneCamera within the calibrateTwoCameras function. After that, I updated the pairsUsed variable to also remove all the imagePoints that contain NaNs before imagePoints is passed to refine(). However, when I compare the output of my edited estimateCameraParameters code (which uses the full dataset when calling calibrateOneCamera) to the output of the default estimateCameraParameters code (which drops any image pair with partially detected targets), the calibrations are identical. This seems to indicate that refine() recalculates and rewrites the camera intrinsics initially stored in stereoParams, pretty much ignoring differences in the intrinsics calculated by calibrateOneCamera.

• Is there any additional documentation for the refine() function?

I appreciate any feedback or pointers to the right direction.

Thanks,

Massimo

%% Init
clear
close all

% Specify calibration images.
leftImages = imageDatastore(fullfile(toolboxdir('vision'), 'visiondata', ...
    'calibration', 'stereo', 'left'));
rightImages = imageDatastore(fullfile(toolboxdir('vision'), 'visiondata', ...
    'calibration', 'stereo', 'right'));

squareSize = 108; % in millimeters
I = readimage(leftImages,1);
imageSize = [size(I, 1), size(I, 2)];

% Detect the checkerboards.
[imagePoints, boardSize] = ...
    detectCheckerboardPoints(leftImages.Files, rightImages.Files);

% Specify world coordinates of checkerboard keypoints.
worldPoints = generateCheckerboardPoints(boardSize, squareSize);

%% Stereo Calibration without prior intrinsics using estimateCameraParameters

% Calibrate the stereo camera system.
[params, imgused, estimationErrors] = estimateCameraParameters(imagePoints, worldPoints, ...
    'ImageSize', imageSize);

%% Stereo calibration with prior intrinsics, using estimateStereoBaseline

% Mono calibrations
[leftparams, leftimageUsed, leftestimationErrors] = ...
    estimateCameraParameters(imagePoints(:,:,:,1), worldPoints, ...
    'ImageSize', imageSize);
[rightparams, rightimageUsed, rightestimationErrors] = ...
    estimateCameraParameters(imagePoints(:,:,:,2), worldPoints, ...
    'ImageSize', imageSize);

% Stereo calibration
[StereoParams, StereoPairsUsed, StereoEstimationErrors] = ...
    estimateStereoBaseline(imagePoints, worldPoints, ...
    leftparams, rightparams, 'WorldUnits', 'millimeters');

%% Print results

disp("Stereo calibration without prior intrinsics")

Stereo calibration without prior intrinsics

disp("MRE of camera 1: " + num2str(params.CameraParameters1.MeanReprojectionError))

MRE of camera 1: 0.056842

disp("MRE of camera 2: " + num2str(params.CameraParameters2.MeanReprojectionError))

MRE of camera 2: 0.057644

disp("Average MRE: " + num2str(params.MeanReprojectionError))

Average MRE: 0.057243

disp(" ")

figure;
showReprojectionErrors(params);
title('Mean Reprojection Error per Image (without intrinsics)')

disp("Mono calibrations")

Mono calibrations

disp("MRE of camera 1: " + num2str(leftparams.MeanReprojectionError))

MRE of camera 1: 0.04465

disp("MRE of camera 2: " + num2str(rightparams.MeanReprojectionError))

MRE of camera 2: 0.044418

disp(" ")

disp("Stereo calibration with prior intrinsics")

Stereo calibration with prior intrinsics

disp("MRE of camera 1: " + num2str(StereoParams.CameraParameters1.MeanReprojectionError))

MRE of camera 1: 0.071023

disp("MRE of camera 2: " + num2str(StereoParams.CameraParameters2.MeanReprojectionError))

MRE of camera 2: 0.072068

disp("Average MRE: " + num2str(StereoParams.MeanReprojectionError))

Average MRE: 0.071545

disp(" ")

figure;
showReprojectionErrors(StereoParams);
title('Mean Reprojection Error per Image (with intrinsics)')

Answer 1

[Aishwarya]

Hi Massimo,

As per my understanding, you want to perform stereo calibration using partially detected checkerboard patterns.

After reviewing the information provided, I would like to offer few insights and address the questions asked:

Is there a correct workflow to perform stereo calibration using partially detected checkerboard patterns?

• There is no official workflow to perform stereo calibration using partially detected checkerboard patterns.
• Having fully detected checkerboard patter will result in better estimation of calibration parameters. Stereo calibration relies on matching keypoints between left and right images to estimate relative pose of two cameras. If the keypoints are partially detected in one image of the pair, it would be challenging to match them with keypoints in the other image, which could lead to inaccurate calibration results.
• Here is the code snippet from “estimateCameraParameters” function that checks for images where both cameras have successfully detected the pattern:

    % Account for possible mismatched pairs
    pairsUsed = imagesUsed1 & imagesUsed2;
    cameraParameters1 = vision.internal.calibration.removeUnusedExtrinsics(...
    cameraParameters1, pairsUsed, imagesUsed1);
    cameraParameters2 = vision.internal.calibration.removeUnusedExtrinsics(...
    cameraParameters2, pairsUsed, imagesUsed2);

Is there a difference in the calibration algorithms used by estimateCameraParameters and estimateStereoBaseline that could justify these worse results?

• The reason for higher mean reprojection error observed when using “estimateStereoBaseline” function is because it only estimates extrinsic parameters with fixed intrinsic parameters. Whereas in the “estimateCameraParameters” function, the intrinsic parameters are optimized during the calibration process.
• The “estimateStereoBaseline” function is specifically designed for estimating the relative pose between two cameras in a stereo setup.

Is there any additional documentation for the refine() function?

• The MathWorks documentation for “refine” function used inside the “estimateCameraParameters” function is given below: https://www.mathworks.com/help/map/ref/wmslayer.refine.html
• After reviewing the “refine” function, it is not recalculating or rewriting camera intrisics stored in “stereoParams” function. You can also investigate the source code “refine” function using the following command:

        open("refine")

Please refer to below MathWorks documentations for more information about the functions used:

• “estimateStereoBaseline” function: https://www.mathworks.com/help/vision/ref/estimatestereobaseline.html
• “estimateCameraParameters” function: https://www.mathworks.com/help/vision/ref/estimatecameraparameters.html
• “stereoParams” function: https://www.mathworks.com/help/vision/ref/stereoparameters.html

I hope this helps!

Best Regards,

Aishwarya