conforming video labeller groundtruth object to work with the split label function

Hi all and thank you so much in advance for reponding to my questions!

Your direction enables my knowledge to expand and I really appreciate this!

I obtain the groundtruth object from the video labelling app and what is throwing me off is that the groundtruth is not exported as a table from the video labelling app, it is exported as groundtruth object and I am lacking eperience manipulating that groundtruth object type

What i am trying to do is process the image data store step with the ground truth object but the split label function is not compartible with that object type!

and Matlab is constantly arguing!

I tried another way but that too is creating more confusion when I get to the bag of words for category classification and evaluating confusion matrix

Though the code works It's a challenge to get pass the bag of word for video category classification and move onto the specificity, accuracy and recall

I am stuck and positive that there is another way to do this conversion,

questions:

1. Is there a way to convert the groundtruth object with a function so that it is compartible with the split label function? This I need to understand please!
2. Is there a bag of word for video method?, I googled but my seach was futile!

Can someone guide me in reference to where I went wrong and the functions that must be deployed in order to move forward, please? and thank you once more!

Please see my code

%% Turning Of PNG Warnings
[~, warnid] = lastwarn;  %get identifier of warning
warning('off', warnid);  %turn warning off
clc
close all
clear 

%% Reading the video
% Video Datasource
Violence = '/Users/mmgp/Documents/MATLAB/Actions'
mov1 = fullfile(Violence, 'Actions.avi')

%Displaying a visual representation of the video file
%implay(mov1);

%% Extracting/View Frames Properties Of Video File
%assigning the name of sample avi file to a variable
filename = VideoReader(mov1);
disp(filename);

%% Filename 1 Beat1.avi Reading Video File
% Defining Output folder as 'Violence'
Shooting = '/Users/mmgp/Documents/MATLAB/Actions/Shooting/';

% % if Shooting Folder Does not exist 
%      if ~exist(Shooting, 'dir')
% % Make directory & execute as indicated in opfolder variable
%      mkdir(Shooting);
%      end
% 
%  % Getting The Number Of Frames
%     numFrames = filename.NumberOfFrames;
%  % Setting Current Status Of Number Of Frames Written To Zero
%     numFramesWritten = 0;
%  % Deploying For Loop To Traverse & Process From Frame '1' To 'last' Frame 
%     for t = 1 : numFrames
%         currFrame = read(filename, t);    % Reading Individual Frames
%         opBaseFileName = sprintf('%1.1d.jpg', t);
%         opFullFileName = fullfile(Shooting, opBaseFileName);
%         imwrite(currFrame, opFullFileName, 'jpg');   % Saving Frames As 'jpg' Files
%         % Indicating Current Progress Of Files/Frames Written
%         progIndication = sprintf('Wrote frame %1d of %1d.', t, numFrames);
%         disp(progIndication);
%         numFramesWritten = numFramesWritten + 1;
%     end      %End Of This 'For' Loop
% progIndication = sprintf('Wrote %d frames to folder "%s"',numFramesWritten, Shooting);
% disp(progIndication);

%% Loading Data
load('net.mat');
load('Shooting.mat')

%% Accessing Classification Labels For Processing  
ShootData1 = [Shooting.LabelData]; % Accessing The Labelling Data
% load('ShootData1.mat','ShootData1')
% data = load('Shooting.mat'); 
% values = data.Shooting.LabelData;
% % Use cellfun to identify which elements within the timetable are empty
% idx = all(~cellfun(@isempty,values{:,:}),2);
% % variable 'out' will contain only those rows which have all non-empty values
% outputdata = values(idx,:);

%% Converting To Cell Array For Access Efficiency
%SData2Table = table2cell(outputdata);
SData2Table = table2cell(ShootData1);
SDataFrames = (1:150)';

%% Specify The Location Of The Image
countNumSdataRec= size(SData2Table,1);% contunting the number of rows
for i = 1:countNumSdataRec
    FrameNames{i,1} = fullfile(Violence,'Shooting',sprintf('%i.jpg',SDataFrames(i)));% creating an image name 
end

SDataTable = [FrameNames,SData2Table]; %Linking the Frame Name With Beating Data

%% Checking For Empty Fields Within SDataTable
%find all cells that don't contain data
% countNumEmptyFields=cellfun('isempty',SData2Table);%faster than L=cellfun(@isempty,data);
% %remove rows with any empty cell
% emptyrows=any(countNumEmptyFields,2);
% SData2Table(emptyrows,:)=[];
% 
% countNumEmptyFields=false(size(SData2Table));%pre-allocating result
% for n=1:numel(SData2Table)
%    countNumEmptyFields(n)=isempty(SData2Table{n});
% end

% load('SDataTable.mat','SDataTable')

%% Splitting Inputs Into Training and Testing Sets
[MyTrain,MyTest] = randomSplit2TrainAndTest(SDataTable,0.7,0);
size(MyTrain);

%% Specifying Input Size Of The First Network Layer
net.Layers(1);
net.Layers(end);
numel(net.Layers(end).ClassNames);
% 
inputSize = net.Layers(1).InputSize;

%% Replacing Final Layer/Last 3/ Configure For Classes 
% Finetuning these 3 layers for new classification
% Extracting all Layers except the last 3
layersTransfer = net.Layers(1:end-3);
% Stipulating Amount Of Classes 
numClasses = size(SDataTable,2)% dimension 2 taking The Size Of The Network;
 
% Adding Newly Edited Layers & Properties
Tlayers = [layersTransfer
  fullyConnectedLayer(numClasses,'Name','fullyConn',...
  'WeightLearnRateFactor',20,'BiasLearnRateFactor',20)
  softmaxLayer('Name','softmax')
  classificationLayer('Name','classoutput','Classes', 'auto')];

%% Specifying Option Features
% Lower the InitialLearnRate to reduce the rate at which
% network parameters are changed. This is beneficial when fine-tuning a
% pre-trained network and prevents the network from changing too rapidly.
% (SGDM) stochastic gradient descent groups the full dataset into disjoint mini-batches This reaches 
% convergence faster as it updates the network's weight value more 
% frequently and increases the computationl speed
options = trainingOptions('sgdm',... 
    'Momentum',0.9,... 
    'InitialLearnRate', 1e-4,... 
    'LearnRateSchedule', 'piecewise', ... 
    'LearnRateDropFactor', 0.1, ... 
    'Shuffle','every-epoch', ... 
    'LearnRateDropPeriod', 14, ... 
    'L2Regularization', 1e-4, ... 
    'MaxEpochs',5,... 
    'MiniBatchSize',20,...
    'Verbose',true); %'Plots','training-progress', ...

%% Combining All Data Variables For Structure 2 Table Conversion1
ImageLocation = MyTrain(:,1); % Selecting Image Location Of 1st data column
Shoot = MyTrain(:,2); % Selecting 2nd data column Activity ROI Data 
ShootNeg = MyTrain(:,3); % Selecting 3rd data Column Activity ROI Negative Data
Object  = MyTrain(:,4); % Selecting 4th data Column Object ROI Data
ObjectNeg  = MyTrain(:,5); % Selecting 5th data Column Object Negative ROI Data
Victim  = MyTrain(:,6); % Selecting 6th data Column Victim ROI Data
VictimNeg  = MyTrain(:,7); % Selecting 7ths data Column Victim Negative ROI Data

TrainingData = table(ImageLocation,Shoot,ShootNeg,Object,ObjectNeg,Victim,VictimNeg);

%% Training The R-CNN Detector 
rcnn = trainRCNNObjectDetector(TrainingData, Tlayers, options, 'NegativeOverlapRange', [0 0.3]);
RegionProps = rcnn.RegionProposalFcn;
RCNNnetwork = rcnn.Network;
RCNNlayers = RCNNnetwork.Layers;

%% Computing Optical Flow (OFl) Using Lucas-Kanade Algorithm
% % Returning (OFl) object estimating trajectory, acceleration & motion 
% % from previous video frame to the current one 
% % 
% % CURRENTTIME: Timestamp specified as **1**
% % seconds from the start of the video file. This is between zero 
% % and the video duration as a 
% % numeric scalar. 
% 
% vidReader = VideoReader('Actions.avi','CurrentTime',1);
% 
% % *** Creating (OFl) Object
% %     (OFl) Estimation & Properties
% %     NOISEThreshold REDUCTION: 
% %     The higher the number, the less small movements impact (OFl) 
% %     calculation
% % 
% % *** SMOOTHNESS-
% %     Modifying value to identify increased motion between 
% %     consecutive frames. 1 is typical value for 'Smoothness' 
%     
% %     Estimating (OFl) From Consecutive Image Frames
% %     LK Provides A Better Combination of Object Trajectory, Accuracy
% %     & Acceleration
% opticFlow = opticalFlowLK('NoiseThreshold',0.003);
% 
% %***Creating window for visualizing (OFl) vectors.
% h = figure;
% movegui(h);
% hViewPanel = uipanel(h,'Position',[0 0 1 1],'Title','Plot of (OFl) Vectors');
% hPlot = axes(hViewPanel);
% 
% %***Reading & Displaying image frames/ convert to grayscale
% % % Display (OFl) velocity vectors as a quiver plot with arrows 
% % % components denoting (u,v) at the points (x,y)
% % % Plot flow controls Volume of the vectors displayed
%    
%       while hasFrame(vidReader)
%         frameRGB = readFrame(vidReader);
%         frameGray = rgb2gray(frameRGB);
%         % Computing OFl
%         flow = estimateFlow(opticFlow, frameGray); 
%         % Displaying video frame with flow vectors
%         imshow(frameRGB) 
%         hold on
%         plot(flow, 'DecimationFactor', [5 5], 'ScaleFactor', 7)
%         drawnow
%         hold on 
%       end
% 
% %  Reseting Internal (OFl) Object State
% reset(opticFlow)


%% Displaying The Strongest Detection Result.
% Set the Non-Maximum Suppression to True to eliminate overlapping bounding
% boxes based on their scores. Set this to false for custom selection
% When False all bounding boxes are returned



[score, idx] = max(score);
bbox = bbox(idx, :); 
annotation = sprintf('%s: (Confidence = %f)', label(idx), score);
annotation1 = sprintf('%s: (Confidence = %f)', label(idx), selectscore);
detectedImg = insertObjectAnnotation(img, 'rectangle', bbox, annotation);
detectedImg1 = insertObjectAnnotation(img, 'rectangle', bbox,annotation,'Color','g');
detectedImg2 = insertObjectAnnotation(img, 'rectangle', selectbbox,annotation1,'Color','r');
figure
imshow(detectedImg);
figure
imshow(detectedImg1);
figure
imshow(detectedImg2);

%%

bag = bagOfFeatures(MyTrain);