Best Way to Quickly Label Pixels in Video?

I'm trying to speed up a very tedious process right now by developing a Matlab tool in App Designer, but I'm really struggling with handling videos. The Video Labeler app ALMOST accomplishes what I would like, but I want to make a more customizable app. I'm going through thousands of frames by hand and recording them in Excel for every frame.

My Experience Level

Coming from a mechanical engineer background, I've dealt with statistics, neural networks, machine learning, and optimization problems. Now I'm delving into video/image processing and designing apps it seems.

What I'm Looking For

What would be the best method of going about a fast pixel labeler for videos that assigns labels off of keybinds? Should I plot images into an interactive graph? Is there a way to do an interactive overlay? Just pointing me to reference material is plenty, I obviously don't want someone to write the code for me.

The gist of it is this:

Load in a video

Every time I left click a spot in the video, it saves the pixel location with a label of "Left Click" and moves to the next frame

Every time I right click a spot in the video, it saves the pixel location of a click with the label of "Right Click" and moves to the next frame

END RESULT

The idea is that at the end, I will have a table that looks like

Frame Label X_Pos Y_Pos

1 L 800 45

2 L 880 32

3 R 588 40

...

Answer 1

[Ameer Hamza]

Check the attached app. It uses simple axes() instead of UIAxes(), because UIAxes() does not support any mouse click callback yet. The app just demonstrates the concept and uses disp() to print the mouse-click type, x_pos and y_pos on the command window. You can adapt it according to your requirements.

[Grant Anderson]

Wow Ameer! Thank you so much! I will dissect this but I'm already seeing some very useful components.

[Ameer Hamza]

I am glad to know that my answer is helpful. If you have any confusion, then you can leave a comment here.