How can I move my mouse according to a centroid value?

2 views (last 30 days)
Priyana Kumar
Priyana Kumar on 14 Jan 2021
Commented: Walter Roberson on 14 Jan 2021
I have been trying to complete a gesture recognition mouse control program in MATLAB and want to do this without using color detection. So far I have been able to structure the code how I best see fit but I am unable to get the mouse to move with the centroid of my hand.
The gestures I have are number-based:
1 - the mouse will move according to the centroid of my hand.
2 - left click
3 - right click
4 - double click
5- scroll
I have gesture 1 as the start of a while loop with gestures 2-5 inside the loop. Is there any way I can move the mouse according to my centroid value?
I have tried using color detection, which works well, but it isnt an option for my final project. So far, the finger detection has worked for gestures 2-5 but not for 1. Any help is greatly appreciated!
close all
%% Import Java Robot
import java.awt.Robot;
import java.awt.event.*
mouse = Robot;
%% Create video input object.
cam = videoinput('macvideo'); %OR ('winvideo');
cam.Name = 'FaceTime HD Camera'; %change your camera name if needed
cam.TriggerRepeat = 10000; %change to Inf when needed
cam.FrameGrabInterval = 1;
picture = getsnapshot(cam);
frameSize = size(picture);
videoplayer = vision.VideoPlayer('Name', 'Hand Gesture Recognition', 'Position', [100 100 [frameSize(2), frameSize(1)]]);
set(cam,'ReturnedColorspace','rgb')
%% Count Number of Fingers
runloop = true;
while runloop
img1 = getsnapshot(cam);
img1 = rgb2gray(img1);
img2 = imcomplement(imbinarize(img1));
img3 = imfill(img2, 'holes'); %filling all holes
img4 = bwareaopen(img3, 10000); %removing objects lesser than 10K size
SE = strel('disk', 10); %defining structural elements
SE1 = strel('disk', 50);
SE2 = strel('disk', 60);
img4e = imerode(img4, SE1); %image erosion
img4d = imdilate(img4e, SE2); %image dilation
imgfo = img4-img4d; %getting fingers
imgfo(imgfo==1)=0;
imgfo = logical(imgfo);
imgfo = bwareaopen(imgfo,5000);
CC = bwconncomp(imgfo); %connected component analysis
nof = CC.NumObjects; %getting number of fingers
imgfog = uint8(255.*imgfo);
nofs = num2str(nof);
imgfogrgb = insertText(imgfog, [0,0], nofs, 'FontSize', 30, 'BoxColor', 'white', 'BoxOpacity', 1, 'TextColor', 'black');
step(videoplayer, imgfogrgb);
runloop = isOpen(videoplayer); %checking if videoplayer is on or off
set(gcf, 'Position', [100, 100, 500, 400])
movegui('northeast')
pause(2);
% preview(cam);
% movegui('northeast');
%% Begin Move Movements
while nof==1 %move
st = regionprops(imgfo, 'BoundingBox', 'Centroid');
imshow(imgfo)
hold on
for k = 1 : length(st)
bbox = st(k).BoundingBox;
cbox = st(k).Centroid;
rectangle('Position',bbox,'EdgeColor','r','LineWidth',1 )
plot(cbox(1),cbox(2), '-m+')
a = text(cbox(1)+15, cbox(2), strcat('X: ', num2str(round(cbox(1))), ' Y: ', num2str(round(cbox(2)))));
set(a, 'FontSize', 12, 'Color', 'white');
%hold on;
end
X = cbox(1);
Y = cbox(2);
set(0,'PointerLocation',[X Y]);
centroids = cat(1,st.Centroid);
%mouse.mouseMove(1600-(centroids(1,1)*(5/2)),(centroids(1,2)*(5/2)-180));

Answers (0)

Products


Release

R2020b

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by