# Create 360° Bird's-Eye-View Image Around a Vehicle

This example shows how to create a 360° bird's-eye-view image around a vehicle for use in a surround view monitoring system.

### Overview

Surround view monitoring is an important safety feature provided by advanced driver-assistance systems (ADAS). These monitoring systems reduce blind spots and help drivers understand the relative position of their vehicle with respect to the surroundings, making tight parking maneuvers easier and safer. A typical surround view monitoring system consists of four fisheye cameras, with a 180° field of view, mounted on the four sides of the vehicle. A display in the vehicle shows the driver the front, left, right, rear, and bird's-eye view of the vehicle. While the four views from the four cameras are trivial to display, creating a bird's-eye view of the vehicle surroundings requires intrinsic and extrinsic camera calibration and image stitching to combine the multiple camera views.

In this example, you first calibrate the multi-camera system to estimate the camera parameters. You then use the calibrated cameras to create a bird's-eye-view image of the surroundings by stitching together images from multiple cameras.

### Calibrate the Multi-Camera System

First, calibrate the multi-camera system by estimating the camera intrinsic and extrinsic parameters by constructing a monoCamera object for each camera in the multi-camera system. For illustration purposes, this example uses images taken from eight directions by a single camera with a 78˚ field of view, covering 360˚ around the vehicle. The setup mimics a multi-camera system mounted on the roof of a vehicle.

#### Estimate Monocular Camera Intrinsics

Camera calibration is an essential step in the process of generating a bird's-eye view. It estimates the camera intrinsic parameters, which are required for estimating camera extrinsics, removing distortion in images, measuring real-world distances, and finally generating the bird's-eye-view image.

In this example, the camera was calibrated using a checkerboard calibration pattern in the Single Camera Calibrator App and the camera parameters were exported to cameraParams.mat. Load these estimated camera intrinsic parameters.

Since this example mimics eight cameras, copy the loaded intrinsics eight times. If you are using eight different cameras, calibrate each camera separately and store their intrinsic parameters in a cell array named intrinsics.

numCameras = 8;
intrinsics = cell(numCameras, 1);

intrinsics(:) = {ld.cameraParams.Intrinsics};

#### Estimate Monocular Camera Extrinsics

In this step, you estimate the extrinsics of each camera to define its position in the vehicle coordinate system. Estimating the extrinsics involves capturing the calibration pattern from the eight cameras in a specific orientation with respect to the road and the vehicle. In this example, you use the horizontal orientation of the calibration pattern. For details on the camera extrinsics estimation process and pattern orientation, see Calibrate a Monocular Camera.

Place the calibration pattern in the horizontal orientation parallel to the ground, and at an appropriate height such that all the corner points of the pattern are visible. Measure the height after placing the calibration pattern and the size of a square in the checkerboard. In this example, the pattern was placed horizontally at a height of 62.5 cm to make the pattern visible to the camera. The size of a square in the checkerboard pattern was measured to be 29 mm.

% Measurements in meters
patternOriginHeight = 0.625;
squareSize = 29e-3;

The following figure illustrates the proper orientation of the calibration pattern for cameras along the four principal directions, with respect to the vehicle axes. However, for generating the bird's-eye view, this example uses four additional cameras oriented along directions that are different from the principal directions. To estimate extrinsics for those cameras, choose and assign the preferred orientation among the four principal directions. For example, if you are capturing from a front-facing camera, align the X- and Y- axes of the pattern as shown in the following figure.

The variable patternPositions stores the preferred orientation choices for all the eight cameras. These choices define the relative orientation between the pattern axes and the vehicle axes for estimateMonoCameraParameters function. Display the images arranged by their camera positions relative to the vehicle.

patternPositions = ["front", "left" , "left" , "back" ,...
"back" , "right", "right", "front"];
extrinsicsCalibrationImages = cell(1, numCameras);
for i = 1:numCameras
filename = "extrinsicsCalibrationImage" +  string(i) + ".jpg";
end
helperVisualizeScene(extrinsicsCalibrationImages, patternPositions)

To estimate the extrinsic parameters of one monocular camera, follow these steps:

1. Remove distortion in the image.

2. Detect the corners of the checkerboard square in the image.

3. Generate the world points of the checkerboard.

4. Use estimateMonoCameraParameters function to estimate the extrinsic parameters.

5. Use the extrinsic parameters to create a monoCamera object, assuming that the location of the sensor location at vehicle coordinate system's origin.

In this example, the setup uses a single camera that was rotated manually around a camera stand. Although the camera's focal center had moved during this motion, for simplicity, this example assumes that the sensor remained at the same location (at origin). However, distances between cameras on a real vehicle can be measured and entered in the sensor location property of monoCamera.

monoCams = cell(1, numCameras);
for i = 1:numCameras
% Undistort the image.
undistortedImage = undistortImage(extrinsicsCalibrationImages{i}, intrinsics{i});

% Detect checkerboard points.
[imagePoints, boardSize] = detectCheckerboardPoints(undistortedImage,...
"PartialDetections", false);

% Generate world points of the checkerboard.
worldPoints = generateCheckerboardPoints(boardSize, squareSize);

% Estimate extrinsic parameters of the monocular camera.
[pitch, yaw, roll, height] = estimateMonoCameraParameters(intrinsics{i}, ...
imagePoints, worldPoints, patternOriginHeight,...
"PatternPosition", patternPositions(i));

% Create a monoCamera object, assuming the camera is at origin.
monoCams{i} = monoCamera(intrinsics{i}, height, ...
"Pitch", pitch, ...
"Yaw"  , yaw, ...
"Roll" , roll, ...
"SensorLocation", [0, 0]);
end

### Create 360° Bird's-Eye-View Image

Use the monoCamera objects configured using the estimated camera parameters to generate individual bird's-eye-view images from the eight cameras. Stitch them to create the 360° bird's-eye-view image.

Capture the scene from the cameras and load the images in the MATLAB workspace.

sceneImages = cell(1, numCameras);
for i = 1:numCameras
filename = "sceneImage" + string(i) + ".jpg";
end
helperVisualizeScene(sceneImages)

#### Transform Images to Bird's-Eye View

Specify the rectangular area around the vehicle that you want to transform into a bird's-eye view and the output image size. In this example, the farthest objects in captured images are about 4.5 m away.

Create a square output view that covers 4.5 m radius around the vehicle.

distFromVehicle = 4.5; % in meters
outView = [-distFromVehicle, distFromVehicle, ... % [xmin, xmax,
-distFromVehicle, distFromVehicle];    %  ymin, ymax]
outImageSize = [640, NaN];

To create the bird's-eye-view image from each monoCamera object, follow these steps.

1. Remove distortion in the image.

2. Create a birdsEyeView object.

3. Transform the undistorted image to a bird's-eye-view image using the transformImage function.

bevImgs = cell(1, numCameras);
birdsEye = cell(1, numCameras);
for i = 1:numCameras
undistortedImage = undistortImage(sceneImages{i}, monoCams{i}.Intrinsics);
birdsEye{i} = birdsEyeView(monoCams{i}, outView, outImageSize);
bevImgs{i}  = transformImage(birdsEye{i}, undistortedImage);
end
helperVisualizeScene(bevImgs)

Test the accuracy of the extrinsics estimation process by using the helperBlendImages function which blends the eight bird's-eye-view images. Then display the image.

tiled360DegreesBirdsEyeView = zeros(640, 640, 3);
for i = 1:numCameras
tiled360DegreesBirdsEyeView = helperBlendImages(tiled360DegreesBirdsEyeView, bevImgs{i});
end
figure
imshow(tiled360DegreesBirdsEyeView)

For this example, the initial results from the extrinsics estimation process contain some misalignments. However, those can be attributed to the wrong assumption that the camera was located at the origin of the vehicle coordinate system. Correcting the misalignment requires image registration.

### Register and Stitch Bird's-Eye-View Images

First, match the features. Compare and visualize the results of using matchFeatures with matchFeaturesInRadius, which enables you to restrict the search boundary using geometric constraints. Constrained feature matching can improve results when patterns are repetitive, such as on roads, where pavement markings and road signs are standard. In factory settings, you can design a more elaborate configuration of the calibration patterns and textured background that further improves the calibration and registration process. The Feature Based Panoramic Image Stitching example explains in detail how to register multiple images and stitch them to create a panorama. The results show that constrained feature matching using matchFeaturesInRadius matches only the corresponding feature pairs in the two images and discards any features corresponding to unrelated repititive patterns.

% The last two images of the scene best demonstrate the advantage of
% constrained feature matching as they have many repetitive pavement
% markings.
I = bevImgs{7};
J = bevImgs{8};

% Extract features from the two images.
grayImage = rgb2gray(I);
pointsPrev = detectKAZEFeatures(grayImage);
[featuresPrev, pointsPrev] = extractFeatures(grayImage, pointsPrev);

grayImage = rgb2gray(J);
points = detectKAZEFeatures(grayImage);
[features, points] = extractFeatures(grayImage, points);

% Match features using the two methods.
indexPairs1 = matchFeaturesInRadius(featuresPrev, features, points.Location, ...
pointsPrev.Location, 15, ...
"MatchThreshold", 10, "MaxRatio", 0.6);

indexPairs2 = matchFeatures(featuresPrev, features, "MatchThreshold", 10, ...
"MaxRatio", 0.6);

% Visualize the matched features.
tiledlayout(1,2)
nexttile
showMatchedFeatures(I, J, pointsPrev(indexPairs1(:,1)), points(indexPairs1(:,2)))
title(sprintf('%d pairs matched\n with spatial constraints', size(indexPairs1, 1)))

nexttile
showMatchedFeatures(I, J, pointsPrev(indexPairs2(:,1)), points(indexPairs2(:,2)))
title(sprintf('%d pairs matched\n without spatial constraints', size(indexPairs2,1)))

The functions helperRegisterImages and helperStitchImages have been written based on the Feature Based Panoramic Image Stitching example using matchFeaturesInRadius. Note that traditional panoramic stitching is not enough for this application as each image is registered with respect to the previous image alone. Consequently, the last image might not align accurately with the first image, resulting in a poorly aligned 360° surround view image.

This drawback in the registration process can be overcome by registering the images in batches:

1. Register and stitch the first four images to generate the image of left side of the vehicle.

2. Register and stitch the last four images to generate the image of right side of the vehicle.

3. Register and stitch the left side and right side to get the complete 360° of the bird's-eye-view image of the scene.

Note the use of larger matching radius for stitching images in step 3 compared to steps 1 and 2. This is because of the change in the relative positions of the images during the first two registration steps.

% Combine the first four images to get the stitched leftSideview and the
% spatial reference object Rleft.
leftImgs = bevImgs(1:4);
[leftSideView, Rleft] = helperStitchImages(leftImgs, tforms);

% Combine the last four images to get the stitched rightSideView.
rightImgs = bevImgs(5:8);
rightSideView = helperStitchImages(rightImgs, tforms);

% Combine the two side views to get the 360° bird's-eye-view in
% surroundView and the spatial reference object Rsurround
imgs = {leftSideView, rightSideView};
[surroundView, Rsurround] = helperStitchImages(imgs, tforms);
figure
imshow(surroundView)

### Measure Distances in the 360° Bird's-Eye-View

One advantage in using bird's-eye-view images to measure distances is that the distances can be computed across the image owing to the planar nature of the ground. You can measure various distances that are useful for ADAS applications such as drawing proximity range guidelines and ego vehicle boundaries. Distance measurement involves transforming world points in the vehicle coordinate system to the bird's-eye-view image, which you can do using the vehicleToImage function. However, note that each of the eight bird's-eye-view images have undergone two geometric transformations during the image registration process. Thus, in addition to using the vehicleToImage function, you must apply these transformations to the image points. The helperVehicleToBirdsEyeView function applies these transformations. The points are projected to the first bird's-eye-view image, as this image has undergone the least number of transformations during the registration process.

#### Draw Proximity Range Guidelines

Circular parking range guidelines around the vehicle can assist drivers maneuvering in tight parking spots. Draw circular guidelines at 2, 3, and 4 meters on the 360° bird's-eye-view image:

1. Transform the vehicle center and a point in the circular guideline in the vehicle coordinate system, to the 360° bird's-eye-view image using helperVehicleToBirdsEyeView function.

2. Calculate the radius of the circular guideline in pixels by finding the distance between the two transformed points.

3. Draw the guidelines using the insertShape function and label the guidelines using the insertText function.

proximityRange = [2, 3, 4]; % in meters
colors         = ["red", "yellow", "green"];
refBirdsEye    = birdsEye{1};
Rout           = {Rleft, Rsurround};
vehicleCenter  = [0, 0];
vehicleCenterInImage = helperVehicleToBirdsEyeView(refBirdsEye, vehicleCenter, Rout);

for i = 1:length(proximityRange)

% Estimate the radius of the circular guidelines in pixels given its
circlePoint         = [0, proximityRange(i)];
circlePointInImage  = helperVehicleToBirdsEyeView(refBirdsEye, circlePoint, Rout);

% Compute radius using euclidean norm.
proximityRangeInPixels  = norm(circlePointInImage - vehicleCenterInImage, 2);

surroundView = insertShape(surroundView, "Circle", [vehicleCenterInImage, proximityRangeInPixels], ...
"LineWidth", 1, ...
"Color", colors(i));

labelText = string(proximityRange(i)) + " m";
surroundView = insertText(surroundView, circlePointInImage, labelText,...
"TextColor", "White", ...
"FontSize", 14, ...
"BoxOpacity", 0);
end

imshow(surroundView)

#### Draw Ego Vehicle Boundary

Boundary lines for a vehicle help the driver understand the relative position of the vehicle with respect to the surroundings. Draw the ego vehicle's boundary using a similar procedure as that of drawing proximity guidelines. The helperGetVehicleBoundaryOnBEV function returns the corner points of the vehicle boundary on the 360° bird's-eye-view image given the vehicle position and size. Show the guidelines on the scene using the showShape function.

vehicleCenter = [0, 0];
vehicleSize   = [5.6, 2.4]; % length-by-width in meters
[polygonPoints, vehicleLength, vehicleWidth] = helperGetVehicleBoundaryOnBEV(refBirdsEye, ...
vehicleCenter, ...
vehicleSize, ...
Rout);
showShape("polygon", polygonPoints, "Label", "Ego Vehicle")

Additionally, you can also overlay a simulated vehicle on the scene for visually pleasing results.

% Read the picture of the simulation vehicle.
egoVehicle = imread("vehicle.png", "BackgroundColor", [0 0 0]);

% Bring the simulation vehicle into the vehicle coordinate system.
egoVehicle = imresize(egoVehicle, [vehicleLength, vehicleWidth]);
vehicle    = zeros(size(surroundView), "uint8");
xIdx       = polygonPoints(1,1) + (1:vehicleWidth);
yIdx       = polygonPoints(1,2) + (1:vehicleLength);
vehicle(yIdx, xIdx, :) = egoVehicle;

% Overlay the simulation vehicle on the 360° bird's-eye-view image.
sceneBirdsEyeView = helperOverlayImage(vehicle, surroundView);

Finally, let's eliminate black borders in the image by selecting smaller range from the vehicle's coordinate system's origin.

distFromVehicle = 4.25; % in meters
[x, y, h, w] = helperGetImageBoundaryOnBEV(refBirdsEye, distFromVehicle, Rout);
croppedSceneBirdsEyeView = imcrop(sceneBirdsEyeView, [x, y, h, w]);
imshow(croppedSceneBirdsEyeView)

### Conclusion

The procedure shown in this example can be extended to build a production grade surround view monitoring system. This requires accurate camera calibration to estimate the monocular camera positions with minimal errors and tuning the registration hyperparameters. The example can also be modified to use fisheye cameras.

### Supporting Functions

#### helperVisualizeScene Function

The helperVisualizeScene function displays the images arranged by their camera positions relative to the vehicle on a 3-by-3 tiled chart layout and optionally shows the title text for each of the tiles.

function helperVisualizeScene(images, varargin)

numImages = numel(images);
if nargin == 2
titleText = varargin{1};
else
titleText = strings(numImages,1);
end

% Define index locations to simulate the camera positions relative to the vehicle.
cameraPosInFigureWindow = [1, 4, 7, 8, 9, 6, 3, 2];
egoVehiclePosInFigureWindow = 5;

figure
t = tiledlayout(3, 3, "TileSpacing", "compact", "Padding", "compact");
for i = 1:numImages
nexttile(cameraPosInFigureWindow(i))
imshow(images{i})
title(titleText(i))
end

% Visualize the vehicle on the scene.
egoVehicle = imread("vehicle.png", "BackgroundColor", [1 1 1]);
nexttile(egoVehiclePosInFigureWindow)
imshow(egoVehicle)
if nargin == 2
title("Ego Vehicle")
title(t, "Pattern positions");
end
end

#### helperBlendImagesFunction

The helperBlendImages function performs alpha blending to the given two input images, I1 and I2, with alpha values that are proportional to the center seam of each image. The output Iout is a linear combination of the input images:

${\mathit{I}}_{\mathrm{out}}=\text{\hspace{0.17em}}\alpha {\mathit{I}}_{1}+\left(1-\alpha \right){\mathit{I}}_{2}$

function outputImage = helperBlendImages(I1, I2)
arguments
I1 uint8
I2 uint8
end
% Identify the image regions in the two images by masking out the black
% regions.
mask1 = sum(I1, 3) ~= 0;
mask2 = sum(I2, 3) ~= 0;

% Compute alpha values that are proportional to the center seam of the two
% images.

I1 = double(I1);
I2 = double(I2);
outputImage = alpha1.*I1 + alpha2.*I2;
outputImage = uint8(outputImage);
end

#### helperRegisterImages Function

The helperRegisterImages function registers a cell array of images sequentially using the searching radius for matchFeaturesInRadius and returns the transformations, tforms.

params.MatchThreshold = 10;
params.MaxRatio       = 0.6;
params.Confidence     = 99.9;
params.MaxDistance    = 2;
params.MaxNumTrials   = 2000;

numImages = numel(images);

% Store points and features for all images.
features = cell(1,numImages);
points   = cell(1,numImages);
for i = 1:numImages
grayImage = rgb2gray(images{i});
points{i} = detectKAZEFeatures(grayImage);
[features{i}, points{i}] = extractFeatures(grayImage, points{i});
end

% Initialize all the transforms to the identity matrix.
tforms(numImages) =  affine2d(eye(3));

% Set the seed for reproducibility.
rng(0);

% Find the relative transformations between each image pair.
for i = 2:numImages
% Find correspondences between images{i} and images{i-1} using
% constrained feature matching.
indexPairs = matchFeaturesInRadius(features{i-1}, features{i}, points{i}.Location, ...
"MatchThreshold", params.MatchThreshold, ...
"MaxRatio", params.MaxRatio);

%  Estimate the transformation between images{i} and images{i-1}.
matchedPointsPrev = points{i-1}(indexPairs(:,1), :);
matchedPoints     = points{i}(indexPairs(:,2), :);
tforms(i) = estimateGeometricTransform2D(matchedPoints, matchedPointsPrev,"similarity",...
"Confidence" , params.Confidence, ...
"MaxDistance", params.MaxDistance, ...
"MaxNumTrials",params.MaxNumTrials);

% Compute the transformation that maps images{i} to the stitched
% image as T(i)*T(i-1)*...*T(1).
tforms(i).T = tforms(i).T*tforms(i-1).T;
end
end

#### helperStitchImages Function

The helperStitchImages function applies the transforms tforms to the input images and blends them to produce the outputImage. It additionally returns the outputView, which you can use to transform any point from the first image in the given image sequence to the output image.

function [outputImage, outputView] = helperStitchImages(images, tforms)

numImages = numel(images);
imageSize = zeros(numImages,2);
xlim = zeros(numImages,2);
ylim = zeros(numImages,2);

% Compute the output limits for each transform.
for i = 1:numel(images)
imageSize(i,:) = size(images{i}, 1:2);
[xlim(i,:), ylim(i,:)] = outputLimits(tforms(i), ...
[1 imageSize(i,2)], ...
[1 imageSize(i,1)]);
end

% Find the minimum and maximum output limits.
maxImageSize = max(imageSize);

xMin = min([1; xlim(:)]);
xMax = max([maxImageSize(2); xlim(:)]);

yMin = min([1; ylim(:)]);
yMax = max([maxImageSize(1); ylim(:)]);

% Width and height of panorama.
width  = round(xMax - xMin);
height = round(yMax - yMin);

% Initialize the "empty" panorama.
outputImage = zeros([height width 3], "like", images{1});

% Create a 2-D spatial reference object defining the size of the panorama.
xLimits = [xMin xMax];
yLimits = [yMin yMax];
outputView = imref2d([height width], xLimits, yLimits);

% Step 7 - Stitch the images.
for i = 1:numel(tforms)
% Apply transformation.
warpedImage = imwarp(images{i}, tforms(i), "OutputView", outputView);

% Blend the images.
outputImage = helperBlendImages(warpedImage, outputImage);
end
end

#### helperVehicleToBirdsEyeView Function

The helperVehicleToBirdsEyeView function transforms the given world points in vehicle coordinate system to points in the 360° bird's-eye-view image.

function tranformedImagePoints = helperVehicleToBirdsEyeView(birdsEye, vehiclePoints, Rout)

% Transform the 3D worldPoints in vehicle coordinate system to 2D
% imagePoints in the Bird's-Eye-View (BEV) image.
imagePoints = vehicleToImage(birdsEye, vehiclePoints);

% Transform these imagePoints from single BEV image to 360° bird's-eye-view images.
[xPoints, yPoints] = worldToIntrinsic(Rout{1}, imagePoints(:,1), imagePoints(:,2));
[xPoints, yPoints] = worldToIntrinsic(Rout{2}, xPoints, yPoints);

tranformedImagePoints = [xPoints, yPoints];
end

#### helperGetImageBoundaryOnBEV Function

The helperGetImageBoundaryOnBEV function returns the position and size of a bounding box in the bird's-eye-view image that defines a square area that covers distFromVehicle meters around the vehicle.

function [x, y, h, w] = helperGetImageBoundaryOnBEV(birdsEye, distFromVehicle, Rout)

% Define three corner points of the vehicle's bounding box.
vehiclePoints = [ distFromVehicle, distFromVehicle;
-distFromVehicle, distFromVehicle;
-distFromVehicle,-distFromVehicle];

% Flip the x and y axes between the world and vehicle coordinate
% systems.
vehiclePoints = fliplr(vehiclePoints);

imagePoints = helperVehicleToBirdsEyeView(birdsEye, vehiclePoints, Rout);
x = imagePoints(1,1);
y = imagePoints(1,2);
h = abs(imagePoints(1,1) - imagePoints(2,1));
w = abs(imagePoints(2,2) - imagePoints(3,2));
end

#### helperGetVehicleBoundaryOnBEV Function

The helperGetVehicleBoundaryOnBEV function returns the corner points of a vehicle boundary given its position and size.

function [polygonPoints, vehicleLength, vehicleWidth] = ...
helperGetVehicleBoundaryOnBEV(birdsEye, vehicleCenter, vehicleSize, Rout)

x = vehicleCenter(1);
y = vehicleCenter(2);

% Find half length and half width in a and b respectively.
a = vehicleSize(1)/2;
b = vehicleSize(2)/2;

% Define vehiclePoints in the world coordinate system.
vehiclePoints = [ x+b, y+a;
x+b, y-a;
x-b, y-a;
x-b, y+a ];

% Flip the x and y axes between the world and vehicle coordinate
% systems.
vehiclePoints      = fliplr(vehiclePoints);
imagePoints = helperVehicleToBirdsEyeView(birdsEye, vehiclePoints, Rout);

xPoints = ceil(imagePoints(:,1));
yPoints = ceil(imagePoints(:,2));

vehicleLength = abs(yPoints(1) - yPoints(2));
vehicleWidth  = abs(xPoints(2) - xPoints(3));

% x,y points as points on a polygon for showShape.
polygonPoints = [xPoints   , yPoints;...
% Close the polygon by setting the last point as first point.
xPoints(1), yPoints(1)];
end

#### helperOverlayImage Function

The helperOverlayImage function overlays the topImage on the bottomImage and returns the result in outputImage.

function outputImage = helperOverlayImage(topImage, bottomImage)

blender = vision.AlphaBlender("Operation", "Binary mask", ...
mask = sum(topImage, 3) ~= 0;
outputImage = step(blender, bottomImage, topImage, mask);
end