Multi-Sensor Labeler

R2026b

Label ground truth data in images, videos, and point clouds across multi-sensor scenes

Since R2026b

Description

The Multi-Sensor Labeler app enables you to label ground truth data in videos, image sequences, and lidar point clouds from multiple sensors capturing the same scene.

Using the app, you can:

• Simultaneously label multiple time-overlapped signals representing the same scene.

• Define rectangular, rotated rectangular, point, line, polygon, projected cuboid, cuboid, pixel, Semantic Point, and scene label definitions. Use these labels to interactively label your ground truth data.

• Use Semantic Point tools including brush, lasso and cluster selection to label individual points in point cloud data.

• Use built-in AI-assisted detection and tracking algorithms to label ground truth data automatically.

• Write, import, and use custom automation algorithms to automatically label ground truth data.

• Label data in burst mode, which enables you to draw labels on static objects and apply those labels across multiple frames simultaneously.

• Use the Projected View option to view point cloud labels in top, front, and side views simultaneously.

• Use the XY View, YZ View, and XZ View options to get the top view, front view, and side view of a point cloud scene, respectively.

• Use the Camera View option to create and reuse custom views of the point cloud data.

• Use the Auto Align option to rotate and best fit a cuboid to a cluster.

• Remove the ground plane from point cloud scenes for clearer labeling.

• Evaluate the performance of your label automation algorithms by using a visual summary.

• Export the ground truth labels as a groundTruthMultiSensor object. You can use this object for system verification or for creating training data for object detection, semantic segmentation or instance segmentation networks.

• Display time-synchronized signals, such as CAN bus data, by using the lidar.connector.Connector class.

The app supports videos, image sequences, and a variety of point cloud formats including PLY, PCD, LAS, LAZ, and E57 files, and Velodyne, Hesai, and Ouster PCAP files. The app also supports ROS (.bag) and ROS 2 (.db3, .mcap) bag files, which can contain both image and point cloud signals.

To learn more about this app, see Get Started with Multi-Sensor Labeler.

The Multi-Sensor Labeler app consolidates and replaces the Ground Truth Labeler (Automated Driving Toolbox) and Lidar Labeler apps into a single, unified labeling environment. If you are transitioning from these apps, see Migrate from Ground Truth Labeler and Lidar Labeler Apps to Multi-Sensor Labeler App.

Open the Multi-Sensor Labeler App

• MATLAB^® Toolstrip: On the Apps tab, under Image Processing and Computer Vision, click the app icon.

• MATLAB command prompt: Enter multiSensorLabeler.

Examples

• Get Started with Multi-Sensor Labeler
• Load and View Multi-Sensor Data
• Create Labels and Label Multi-Sensor Data
• Automate Labeling for Multi-Sensor Data
• Export Multi-Sensor Ground Truth and Create Training Data

Programmatic Use

multiSensorLabeler opens a new session of the app, enabling you to label ground truth data.

multiSensorLabeler(videoFileName) opens the app and loads the input video. The video file must have an extension supported by VideoReader.

Example: multiSensorLabeler("caltech_cordova1.avi")

multiSensorLabeler(imageSeqFolder) opens the app and loads the image sequence from the input folder. An image sequence is an ordered set of images that resembles a video.

imageSeqFolder must be a string scalar or character vector that specifies the folder containing the image files. The image files must have extensions supported by imformats and are loaded in the order returned by the dir function.

multiSensorLabeler(imageSeqFolder,timestamps) opens the app and loads a sequence of images with their corresponding timestamps. timestamps must be a duration vector of the same length as the number of images in the sequence.

For example, load a sequence of road images and their corresponding timestamps into the app.

imageDir = fullfile(toolboxdir("driving"),"core","drivingdata","roadSequence");
load(fullfile(imageDir,"timeStamps.mat"))
multiSensorLabeler(imageDir,timeStamps)

multiSensorLabeler(ptCloudSeqFolder) opens the app and loads the point cloud sequence from the folder ptCloudSeqFolder, where ptCloudSeqFolder is a string scalar or character vector specifying a folder that contains point cloud files. The point cloud files must have extensions supported by pcformats, and are loaded in the order returned by the dir function.

multiSensorLabeler(ptCloudSeqFolder,timestamps) opens the app and loads a point cloud sequence with corresponding timestamps. timestamps must be a duration vector of the same length as the number of point cloud files in the sequence.

Example: multiSensorLabeler("D:\data",seconds(0:9))

multiSensorLabeler(pcapFile,sensorModel,calibrationFile) opens the app and loads point cloud data from a PCAP file recorded by a Velodyne or Hesai lidar sensor. pcapFile is a string scalar or character vector specifying the path to the PCAP file. sensorModel is a string scalar or character vector specifying the sensor model (for example, "VLP16" or "Pandar128E3X"). calibrationFile is a string scalar or character vector specifying the path to the sensor calibration file.

For supported Velodyne sensor models, see velodyneFileReader. For supported Hesai sensor models, see hesaiFileReader.

Example: multiSensorLabeler("vlp16.pcap","VLP16","VLP-16.xml") multiSensorLabeler("busy_road.pcap","Pandar128E3X","Pandar128_calibration.csv")

multiSensorLabeler(pcapFile,calibrationFile) opens the app and loads point cloud data from a PCAP file recorded by an Ouster lidar sensor. pcapFile is a string scalar or character vector specifying the path to the PCAP file. calibrationFile is a string scalar or character vector specifying the path to the sensor metadata JSON file.

For supported Ouster sensor models, see ousterFileReader.

Example: multiSensorLabeler("OS-DOME-128.pcap","OS-DOME-128.json")

multiSensorLabeler(sessionFile) opens the app and loads a saved app session from the session file, sessionFile. The sessionFile input contains the path to and name of the .prj project file saved from the Multi-Sensor Labeler app.

You can also use this syntax to load legacy session files from the Ground Truth Labeler app (.prj or .mat project files) and the Lidar Labeler app (.mat session files). The app automatically converts these into Multi-Sensor Labeler app sessions.

Note

Support for loading legacy Ground Truth Labeler and Lidar Labeler session files will be removed in a future release. To preserve your work, open the legacy sessions in the Multi-Sensor Labeler app and save them as a project file.

multiSensorLabeler(gTruth) opens the app and loads a ground truth object. The gTruth input is a groundTruthMultiSensor object. The ground truth object data source must be an image sequence, video, point cloud sequence, or a custom data source.

You can also use this syntax to load a groundTruthMultisignal (Automated Driving Toolbox) object exported from the Ground Truth Labeler app or a groundTruthLidar object exported from the Lidar Labeler app.

Note

Support for loading groundTruthMultisignal (Automated Driving Toolbox) and groundTruthLidar objects exported from Ground Truth Labeler and Lidar Labeler apps will be removed in a future release. To preserve your work, load the groundTruthMultisignal and groundTruthLidar objects in the Multi-Sensor Labeler app and export them as groundTruthMultiSensor objects.

multiSensorLabeler(___,"ConnectorTargetHandle",connector) opens the app and loads both of these components:

• A signal, depending on the input argument combination you specify

• An external analysis or visualization tool that is time-synchronized with the specified signal

The connector input is a handle to a lidar.connector.Connector class that implements the external tool. You can also use this syntax with a connector that inherits from driving.connector.Connector (Automated Driving Toolbox) supported by the Ground Truth Labeler app.

Note

Support for loading connectors that inherit from driving.connector.Connector (Automated Driving Toolbox) will be removed in a future release. Update your connectors to inherit from lidar.connector.Connector instead.

For example, this syntax opens the app with a video signal and a synchronized lidar visualization tool.

multiSensorLabeler("01_city_c2s_fcw_10s.mp4","ConnectorTargetHandle",@LidarDisplay);

When you have an external tool connected to a signal in the app, consider these tips.

• If you remove the signal that is connected to the tool, the app disconnects the tool and closes it.

• The signal connected to the tool must be the main signal, that is, the signal whose timestamps are used in the playback controls at the bottom of the app. If you change the main signal, the app disconnects the tool and closes it.

• If you start a new app session, the app disconnects the tool and closes it.

Limitations

• Semantic Point ROI labels do not support sublabels or attributes.

• Pixel ROI labels do not support sublabels or attributes.

• Cuboid ROI labels do not support sublabels.

• The Label Summary window does not support sublabels or attributes.

More About

ROI Labels, Sublabels, and Attributes

On the left side of the app, the ROI Label Definitions pane contains the region of interest (ROI) label definitions that you can mark on the frames. You can create label definitions using the Add Label option from the app toolstrip. Alternatively, you can create label definitions programmatically by using a labelDefinitionCreatorMultiSensor object and then import these label definitions into an app session.

The app supports the definition of ROI labels, sublabels, and attributes.

ROI Labels

An ROI label is a label that corresponds to a region of interest (ROI) in a signal frame. The table describes the supported label types.

ROI Label	Description	Example
Rectangle/Cuboid	Draw rectangular or cuboidal ROI labels around objects, depending on the signal type. In image signals, draw rectangular ROI labels (2-D bounding boxes). In point cloud signals, draw cuboidal ROI labels (3-D bounding boxes).	Vehicles, pedestrians, road signs Rectangle:
		Cuboid:
Rotated Rectangle	Draw rotated rectangular ROI labels around objects.	Rotated Rectangle:
Point	Draw point labels in an image signal to identify objects.
Line	Draw linear ROI labels to represent lines. To draw a polyline ROI, use two or more points.	Lane boundaries, guard rails, road curbs
Projected Cuboid	Draw cuboidal ROI labels (3-D bounding boxes) on image signals.
Polygon	Draw polygon ROI labels. You can label distinct instances of the same class. For more information on drawing polygon ROI labels for instance and semantic segmentation networks, see Label Objects Using Polygons for Instance Segmentation (Computer Vision Toolbox).	Vehicles, road surface, trees, pavement
Pixel	Assign labels to pixels for semantic segmentation. You can label pixels manually using polygons, brushes, or flood fill. For more on pixel labeling, see Label Pixels for Semantic Segmentation (Computer Vision Toolbox).	Vehicles, road surface, trees, pavement
Semantic Point ROI	Assign labels to individual points in point cloud data for semantic segmentation. You can label points using brush, lasso, smart lasso, cluster selection, and planar region selection tools.

ROI Sublabels

An ROI sublabel is an ROI label that belongs to a parent label. Use ROI sublabels to provide a greater level of detail about the ROIs in your labeled ground truth data. For example, a vehicle label might contain headlight, licensePlate, and wheel sublabels. You can create sublabels only for axis-aligned and rotated rectangular, polyline, and projected cuboid labels. For more details about sublabels, see Use Sublabels and Attributes to Label Ground Truth Data (Computer Vision Toolbox).

You can view and edit label hierarchies, sublabels, and attributes for the selected ROI label in the Label, Sublabel, Attributes tree table on the right side of the app.

Show or Hide Labels and Sublabels

You can show or hide the labels or sublabels in a labeled ground truth data by using the icon on the ROI Label Definitions pane. The appears only after you define a label or sublabel. By default, the app displays all the labels and the sublabels.

To hide a label or sublabel, click on the icon alongside the label or sublabel name. The app hides the corresponding label or sublabel and displays the icon.

ROI Attributes

An ROI attribute specifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes might include the type or color of a vehicle. The table describes the supported attribute types.

Attribute Type	Sample Attribute Definition	Sample Default Values
List
Numeric
String
Logical

For more details about attributes, see Use Sublabels and Attributes to Label Ground Truth Data (Computer Vision Toolbox).

Tips

• To avoid having to relabel ground truth with new labels, organize the labeling scheme you want to use before marking your ground truth.

• You can copy and paste labels between signals that are of the same type.

• Remove the ground plane to clearly view the created object labels in point cloud signals.

• Use the rotate, translate, expand, and shrink options to edit cuboids after drawing them.

• Use the Camera View option to save a view of the point cloud data from the current angle and direction.

• Use burst mode to label static objects across multiple point cloud frames simultaneously.

Algorithms

The Multi-Sensor Labeler app provides semi-automated and fully automated labeling options for both image and point cloud signals. Semi-automated algorithms require initial guidance — such as drawing labels on key frames — before propagating labels across the remaining frames. Fully automated algorithms process data and generate labels without manual input. Custom automation algorithms require implementing your own algorithms using the template interfaces provided by the app to enable automated labeling. This table summarize the automated labeling features of the Multi-Sensor Labeler app for images and point clouds:

Automation Type	Image Signals	Point Cloud Signals
Semi-Automated Algorithms (Built-In)	Point Tracker, Temporal Interpolator	Burst Mode, Lidar Object Tracker, Point Cloud Temporal Interpolator
Fully Automated Algorithms (Built-In)	ACF People Detector ACF Vehicle Detector, Lane Boundary Detector (requires Automated Driving Toolbox™)	Pretrained PointPillars, SalsaNext, and SqueezeSegV2 Models
Custom Algorithms (Examples)	Automate Vehicle Labels and Distance Attributes Using YOLOv2 in Multi-Sensor Labeler	Automate Point Cloud Labeling Using SNAP Model

Image Labeling Automation Algorithms

Algorithm	Type	Label Types	Description
ACF People Detector	Fully Automated	Rectangle	Detect people using aggregate channel features.
ACF Vehicle Detector	Fully Automated	Rectangle	Detect vehicles using aggregate channel features.
Lane Boundary Detector	Fully Automated	Line	Detect lane boundaries in highway scenes.
Point Tracker	Semi-Automated	Rectangle	Track a labeled object across frames using the KLT algorithm. Draw an initial label before running.
Temporal Interpolator	Semi-Automated	Rectangle	Estimate labels between keyframes by interpolation. Draw labels on at least two frames before running.

Point Cloud Labeling Automation Algorithms

Algorithm	Type	Label Types	Description
Automation Using Pretrained Models	Fully Automated	Cuboid, Semantic Point ROI	Detect objects or segment points using pretrained deep learning models such as PointPillars, SalsaNext, and SqueezeSegV2. You can also import custom networks.
Lidar Object Tracker	Semi-Automated	Cuboid	Track cuboid labels across point cloud frames. Draw an initial cuboid on the object to track before running.
Point Cloud Temporal Interpolator	Semi-Automated	Cuboid	Estimate cuboid labels between keyframes by interpolation. Draw cuboids on at least two frames before running.
Burst Mode	Semi-Automated	Cuboid, Semantic Point ROI	Merge point clouds across a time range and label static objects once. Labels apply across all timestamps in the range.

Version History

Introduced in R2026b

R2026b: Multi-Sensor Labeler replaces groundTruthMultisignal and groundTruthLidar objects with groundTruthMultiSensor object

The Multi-Sensor Labeler app uses the groundTruthMultiSensor object to store exported labels. If you are transitioning from the Ground Truth Labeler or Lidar Labeler apps, you can load existing groundTruthMultisignal (Automated Driving Toolbox) and groundTruthLidar objects into the Multi-Sensor Labeler app. After loading, re-export your labels as a groundTruthMultiSensor object. Support for loading groundTruthMultisignal and groundTruthLidar objects will be removed in a future release.

R2026b: Multi-Sensor Labeler uses project files instead of MAT files for sessions

The Multi-Sensor Labeler app saves labeling sessions as project files (.prj). If you have existing MAT-file sessions from the Ground Truth Labeler or Lidar Labeler apps, you can load them into the Multi-Sensor Labeler app. After loading, save the session as a project file to take advantage of the latest features. Support for loading legacy MAT-file sessions will be removed in a future release.

R2026b: Multi-Sensor Labeler uses lidar.connector.Connector instead of driving.connector.Connector

The Multi-Sensor Labeler app uses the lidar.connector.Connector class to connect external analysis or visualization tools. If you have existing connectors that inherit from driving.connector.Connector (Automated Driving Toolbox) (used with the Ground Truth Labeler app), update them to inherit from lidar.connector.Connector instead.