How can I move my mouse according to a centroid value?

I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.

The gestures I have are number-based:

1 - the mouse will move according to the centroid of my hand.

2 - left click

3 - right click

4 - double click

5- scroll

I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?

I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!

close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
 
%% Create video input object. 
cam = videoinput('macvideo');   %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000;  %change to Inf when needed
cam.FrameGrabInterval = 1;
 
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
set(cam,'ReturnedColorspace','rgb')
 
%% Count Number of Fingers
runloop = true;
while runloop
    img1 = getsnapshot(cam);
    img1 = rgb2gray(img1);
    img2 = imcomplement(imbinarize(img1));
    img3 = imfill(img2, 'holes');               %filling all holes
    img4 = bwareaopen(img3, 10000);             %removing objects lesser than 10K size
    
    SE = strel('disk', 10);                     %defining structural elements
    SE1 = strel('disk', 50);
    SE2 = strel('disk', 60);
    img4e = imerode(img4, SE1);                 %image erosion
    img4d = imdilate(img4e, SE2);               %image dilation
    imgfo = img4-img4d;                         %getting fingers
    imgfo(imgfo==1)=0;
    imgfo = logical(imgfo);
    imgfo = bwareaopen(imgfo,5000);
    CC = bwconncomp(imgfo);                     %connected component analysis
    nof = CC.NumObjects;                        %getting number of fingers
    imgfog = uint8(255.*imgfo);
    nofs = num2str(nof);
    imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
    step(videoplayer, imgfogrgb);
    runloop = isOpen(videoplayer);              %checking if videoplayer is on or off
    set(gcf, 'Position',  [100, 100, 500, 400])
    movegui('northeast')
    pause(2);
  
%     preview(cam);
%     movegui('northeast');
    
%% Begin Move Movements
while nof==1 %move
     st = regionprops(imgfo, 'BoundingBox', 'Centroid');
     imshow(imgfo)
     hold on
     for k = 1 : length(st) 
     bbox = st(k).BoundingBox;
     cbox = st(k).Centroid;
     rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
     plot(cbox(1),cbox(2), '-m+')
     a = text(cbox(1)+15, cbox(2), strcat('X:  ', num2str(round(cbox(1))), '    Y: ', num2str(round(cbox(2)))));
     set(a, 'FontSize', 12, 'Color', 'white');
     %hold on;
     end
     X = cbox(1);
     Y = cbox(2);
     set(0,'PointerLocation',[X Y]);
     centroids = cat(1,st.Centroid);
     %mouse.mouseMove(1600-(centroids(1,1)*(5/2)),(centroids(1,2)*(5/2)-180)); 

[Walter Roberson]

set figure or axes CurrentPoint rather than root PointerLocation

https://www.mathworks.com/matlabcentral/answers/429274-get-axes-currentpoint-stack-on-not-updated-current-point#answer_347137