groundTruthMultisignal

Ground truth label data for multiple signals

Since R2020a

Description

The groundTruthMultisignal object contains information about the ground truth data source, label definitions, and marked label annotations for multiple signals. The source of the signals can be a video, image sequence, lidar point cloud, or any other custom format containing multiple signals. You can export or import a groundTruthMultisignal object from the Ground Truth Labeler app.

To create training data for deep learning applications from arrays of groundTruthMultisignal objects, use the gatherLabelData function.

Creation

To export a groundTruthMultisignal object from the Ground Truth Labeler app, on the app toolstrip, select Export Labels > To Workspace. The app exports the object to the MATLAB^® workspace. To create a groundTruthMultisignal object programmatically, use the groundTruthMultisignal function (described here).

Syntax

gTruth = groundTruthMultisignal(dataSources,labelDefs,roiData,sceneData)

Description

gTruth = groundTruthMultisignal(dataSources,labelDefs,roiData,sceneData) returns an object containing ground truth labels that can be imported into the Ground Truth Labeler app.

• dataSources specifies the sources of the ground truth data and sets the DataSource property.

• labelDefs specifies the label, sublabel, and attribute definitions of the ground truth data and sets the LabelDefinitions property.

• roiData specifies the identifying information, position, and timestamps for the marked region of interest (ROI) labels and sets the ROILabelData property.

• sceneData specifies the identifying information and timestamps for marked scene labels and sets the SceneLabelData property.

Properties

DataSource — Sources of ground truth data

vector of MultiSignalSource objects

Sources of ground truth data, specified as a vector of MultiSignalSource objects. These objects contain information that describe the sources from which ground truth data was labeled. This table describes the type of MultiSignalSource objects that you can specify in this vector.

MultiSignalSource Object Type	Data Source	Class Reference
VideoSource	Video file	vision.labeler.loading.VideoSource
ImageSequenceSource	Image sequence folder	vision.labeler.loading.ImageSequenceSource
VelodyneLidarSource	Velodyne® packet capture (PCAP) file	vision.labeler.loading.VelodyneLidarSource
RosbagSource	Rosbag file	vision.labeler.loading.RosbagSource
PointCloudSequenceSource	Point cloud sequence folder	vision.labeler.loading.PointCloudSequenceSource
CustomImageSource	Custom image format	vision.labeler.loading.CustomImageSource

To specify additional data sources, create a new type of MultiSignalSource object by using the vision.labeler.loading.MultiSignalSource class.

LabelDefinitions — Label definitions

table

Label definitions, specified as a table. To create this table, use one of these options.

• In the Ground Truth Labeler app, create label definitions, and then export them as part of a groundTruthMultisignal object.

• Use a labelDefinitionCreatorMultisignal object to generate a label definitions table. If you save this table to a MAT-file, you can then load the label definitions into a Ground Truth Labeler app session by selecting Open > Label Definitions from the app toolstrip.

• Create the label definitions table at the MATLAB command line.

This table describes the required and optional columns of the table specified in the LabelDefinitions property.

Column	Description	Required or Optional
Name	Strings or character vectors specifying the name of each label definition.	Required
SignalType	SignalType enumerations that specify the signal type supported for each label definition. Valid values are Image for image signals such as videos or image sequences, PointCloud for lidar signals, or Time for scene label definitions. If a label definition supports multiple signal types, then the label definition has a separate row for each signal type. For example, consider a label definition named car. In the Ground Truth Labeler app, you draw this label as a rectangle in image signals and a cuboid in lidar point cloud signals. In the LabelDefinitions table, car appears twice and has these Name, SignalType, and LabelType values. Name SignalType LabelType 'car' Image Rectangle 'car' PointCloud Cuboid	Required
LabelType	labelType enumerations that specify the type of each label definition. For ROI label definitions with a SignalType of Image, valid LabelType enumerations are: labelType.Rectangle labelType.RotatedRectangle labelType.Line labelType.PixelLabel labelType.Polygon labelType.ProjectedCuboid For ROI label definitions with a SignalType of PointCloud, the only valid LabelType enumeration is labelType.Cuboid. For scene label definitions, the only valid LabelType enumeration is labelType.Scene. You can also add ground truth data that is not an ROI or scene label to a groundTruthMultisignal object. In the label definitions table, specify a label definition whose labelType is Custom.	Required
Group	Strings or character vectors specifying the group to which each label definition belongs.	Optional If you create label definitions at the MATLAB command line, you do not need to include a Group column. If you export label definitions from the Ground Truth Labeler app or create them using a labelDefinitionCreatorMultisignal object, the label definitions table includes this column, even if you did not specify groups. The app assigns each label definition a Group value of 'None'.
Description	Strings or character vectors that describe each label definition.	Optional If you create label definitions at the MATLAB command line, you do not need to include a Description column. If you export label definitions from the Ground Truth Labeler app or create them using a labelDefinitionCreatorMultisignal object, the label definitions table includes this column, even if you did not specify descriptions. The Description for these label definitions is an empty character vector.
LabelColor	1-by-3 row vectors of RGB triplets that specify the colors of the label definitions. Values are in the range [0, 1]. The color yellow (RGB triplet [1 1 0]) is reserved for the color of selected labels in the Ground Truth Labeler app.	Optional When you define labels in the Ground Truth Labeler app, you must specify a color. Therefore, an exported label definitions table always includes this column. When you create label definitions using the labelDefinitionCreatorMultisignal object without specifying colors, the returned label definition table includes this column, but all column values are empty.
PixelLabelID	Scalars, column vectors, M-by-3 matrices of integer-valued label IDs. PixelLabelID specifies the pixel label values used to represent a label definition. Pixel label ID values must be between 0 and 255.	Optional When you define pixel labels in the Ground Truth Labeler app or the labelDefinitionCreatorMultisignal object, the generated label definitions table includes this column. When creating a label definitions table at the MATLAB command line, if you set LabelType to labelType.PixelLabel for any label, then this column is required.
Hierarchy	Structures containing sublabel and attribute data for each label definition. For an example of the Hierarchy format, see Export and Explore Ground Truth Labels for Multiple Signals.	Optional When you define sublabels or attributes in the Ground Truth Labeler app or the labelDefinitionCreatorMultisignal object, the generated label definitions table includes this column.

ROILabelData — ROI label data

ROILabelData object

ROI label data across all signals, specified as an ROILabelData object.

For Rectangle, RotatedRectangle, Cuboid, ProjectedCuboid, Polygon, and Line label types, ground truth data that is not a floating-point array has a data type of single.

SceneLabelData — Scene label data

SceneLabelData object

Scene label data across all signals, specified as a SceneLabelData object.

Object Functions

selectLabelsByLabelName	Select multisignal ground truth by label name
selectLabelsByLabelType	Select multisignal ground truth by label type
selectLabelsByGroupName	Select multisignal ground truth by label group name
selectLabelsBySignalName	Select multisignal ground truth by signal name
selectLabelsBySignalType	Select multisignal ground truth labels by signal type
gatherLabelData	Gather synchronized label data from ground truth
writeFrames	Write signal frames for ground truth data to disk
changeFilePaths	Change file paths in multisignal ground truth data

Examples

Create Ground Truth from Multiple Signals

Create ground truth data for a video signal and a lidar point cloud sequence signal that captures the same driving scene. Specify the signal sources, label definitions, and ROI and scene label data.

Create the video data source from an MP4 file.

sourceName = '01_city_c2s_fcw_10s.mp4';
sourceParams = [];
vidSource = vision.labeler.loading.VideoSource;
vidSource.loadSource(sourceName,sourceParams);

Create the point cloud sequence source from a folder of point cloud data (PCD) files.

pcSeqFolder = fullfile(toolboxdir('driving'),'drivingdata','lidarSequence');
addpath(pcSeqFolder)
load timestamps.mat
rmpath(pcSeqFolder)

lidarSourceData = load(fullfile(pcSeqFolder,'timestamps.mat'));

sourceName = pcSeqFolder;
sourceParams = struct;
sourceParams.Timestamps = timestamps;

pcseqSource = vision.labeler.loading.PointCloudSequenceSource;
pcseqSource.loadSource(sourceName,sourceParams);

Combine the signal sources into an array.

dataSource = [vidSource pcseqSource]

dataSource = 

  1x2 heterogeneous MultiSignalSource (VideoSource, PointCloudSequenceSource) array with properties:

    SourceName
    SourceParams
    SignalName
    SignalType
    Timestamp
    NumSignals

Create a table of label definitions for the ground truth data by using a labelDefinitionCreatorMultisignal object.

• The Car label definition appears twice. Even though Car is defined as a rectangle, you can draw rectangles only for image signals, such as videos. The labelDefinitionCreatorMultisignal object creates an additional row for lidar point cloud signals. In these signal types, you can draw Car labels as cuboids only.

• The label definitions have no descriptions and no assigned colors, so the Description and LabelColor columns are empty.

• The label definitions have no assigned groups, so for all label definitions, the corresponding cell in the Group column is set to 'None'.

• Road is a pixel label definition, so the table includes a PixelLabelID column.

• No label definitions have sublabels or attributes, so the table does not include a Hierarchy column for storing such information.

ldc = labelDefinitionCreatorMultisignal;
addLabel(ldc,'Car','Rectangle');
addLabel(ldc,'Truck','ProjectedCuboid');
addLabel(ldc,'Lane','Line');
addLabel(ldc,'Road','PixelLabel');
addLabel(ldc,'Sunny','Scene');
labelDefs = create(ldc)

labelDefs =

  7x7 table

      Name       SignalType       LabelType        Group      Description    LabelColor    PixelLabelID
    _________    __________    _______________    ________    ___________    __________    ____________

    {'Car'  }    Image         Rectangle          {'None'}       {' '}       {0x0 char}    {0x0 double}
    {'Car'  }    PointCloud    Cuboid             {'None'}       {' '}       {0x0 char}    {0x0 double}
    {'Truck'}    Image         ProjectedCuboid    {'None'}       {' '}       {0x0 char}    {0x0 double}
    {'Lane' }    Image         Line               {'None'}       {' '}       {0x0 char}    {0x0 double}
    {'Lane' }    PointCloud    Line               {'None'}       {' '}       {0x0 char}    {0x0 double}
    {'Road' }    Image         PixelLabel         {'None'}       {' '}       {0x0 char}    {[       1]}
    {'Sunny'}    Time          Scene              {'None'}       {' '}       {0x0 char}    {0x0 double}

Create ROI label data for the first frame of the video.

numVideoFrames = numel(vidSource.Timestamp{1});
carData = cell(numVideoFrames,1);
laneData = cell(numVideoFrames,1);
truckData = cell(numVideoFrames,1);
carData{1} = [304 212 37 33];
laneData{1} = [70 458; 311 261];
truckData{1} = [309,215,33,24,330,211,33,24];
videoData = timetable(vidSource.Timestamp{1},carData,laneData, ...
                      'VariableNames',{'Car','Lane'});

Create ROI label data for the first point cloud in the sequence.

numPCFrames = numel(pcseqSource.Timestamp{1});
carData = cell(numPCFrames, 1);
carData{1} = [27.35 18.32 -0.11 4.25 4.75 3.45 0 0 0];
lidarData = timetable(pcseqSource.Timestamp{1},carData,'VariableNames',{'Car'});

Combine the ROI label data for both sources.

signalNames = [dataSource.SignalName];
roiData = vision.labeler.labeldata.ROILabelData(signalNames,{videoData,lidarData})

roiData = 

  ROILabelData with properties:

    video_01_city_c2s_fcw_10s: [204x2 timetable]
                lidarSequence: [34x1 timetable]

Create scene label data for the first 10 seconds of the driving scene.

sunnyData = seconds([0 10]);
labelNames = ["Sunny"];
sceneData = vision.labeler.labeldata.SceneLabelData(labelNames,{sunnyData})

sceneData = 

  SceneLabelData with properties:

    Sunny: [0 sec    10 sec]

Create a ground truth object from the signal sources, label definitions, and ROI and scene label data. You can import this object into the Ground Truth Labeler app for manual labeling or to run a labeling automation algorithm on it. You can also extract training data from this object for deep learning models by using the gatherLabelData function.

gTruth = groundTruthMultisignal(dataSource,labelDefs,roiData,sceneData)

gTruth = 

  groundTruthMultisignal with properties:

          DataSource: [1x2 vision.labeler.loading.MultiSignalSource]
    LabelDefinitions: [7x7 table]
        ROILabelData: [1x1 vision.labeler.labeldata.ROILabelData]
      SceneLabelData: [1x1 vision.labeler.labeldata.SceneLabelData]

Tips

• groundTruthMultisignal objects with video-based data sources rely on the video reading capabilities of your operating system. A groundTruthMultisignal object created using video data sources remains consistent only for the same platform that was used to create it. To create a platform-independent groundTruthMultisignal object, convert the videos into sequences of images and include the associated timestamps with the image sequences.

• To create a groundTruthMultisignal object containing ROI label data but no scene label data, specify the SceneLabelData property as an empty array. To create this array, at the MATLAB command prompt, enter this code.

  sceneData = vision.labeler.labeldata.SceneLabelData.empty

Version History

Introduced in R2020a