how to fix particular connected component in multiple frames

hi everyone.

i have binary image consisting of 8 to 10 connected components. actually i want to check weather the center of these 8 to 10 connnected component is changing postion in next frame or not but for that first i have to fix/label the connected components in all frames for making sure that the connected component in first frame was the same as in the next frame (like tracking but not propere tracking because i just want to find change in position of multiple connected compnents in multiiple frames).

How i can fix the particular connected components in multiple frames. eiather by assinging ID or label them but dont know how to code that in matlab. so because of begginer in matlab i need someone's help in codding.

thanx in advance

[Rik]

Have a read here and here. It will greatly improve your chances of getting an answer. Flagging is meant to attract attention from site admins (or users with high reputation, who also have some editing privileges).

What I tend to do is label the areas on one frame (e.g. with bwlabel). For the next frame I use the component as a mask on the previous frame and extract the most common value (making sure it isn't 0) and set that as the label for the second frame.

[QuestionsAccount]

Rik will you please help me in achiving my goal.

i just wanted to fix the connected components in multiple video frames (that is the conneccted signed as 1 in 1st frame will remain 1 in 2nd and upto last frame and similartly connected component marked as 2 in 1st frame will remain 2 in 2nd frame as well and upto last frame and so on for all other connected component).

one idea is to fix the CC on the bases of number of pixls it contains (but i m facing difculties in coding that idea i need help in cooding).

your help is highly appricated.

[Walter Roberson]

If you feel that your question is unclear then as the person who asked the question, you should clarify the question instead of flagging it.

[Rik]

I outlined an idea. Have you tried it? Can you share an example file and the code you tried to solve it with?

[jonas]

If their size and shape is more or less constant, the you could also crop the bounding box of the component in the first frame, then compare them one by one to each consecutive frame by xcorr2. Never tried it though.

[QuestionsAccount]

attach is the image of connected components that i want to fix in multiple video frames (that is the conneccted component marked as 1 in 1st frame will remain 1 in 2nd and upto last frame and similartly connected component marked as 2 in 1st frame will remain 2 in 2nd frame and upto last frame and so on for all other connected components).

one idea is to fix the CC on the bases of number of pixls it contains (but i m facing difculties in coding that idea i need help in cooding).

part of my code is attach in which i m trying to do the above mention work (although its hard coding but i didn't any built in function for that)

[jonas]

If you are looking for actual code then you should include a second image (or more) where the blobs have changed position. They appear to be people, so I assume both shape and position are variable.

I wrote something similar for my own project last night using Rik's idea and it worked well. Loop over components from binary image t+1 and use as masks on labeled image t. The most common value (other than zero) is then assigned as label to labeled image t+1. Don't know how to deal with components merging yet..

[QuestionsAccount]

some other images are also attached which shows change in position.

actually first of all i want to fix/lock the connected components in multiple video frames (that is the conneccted component marked as 1 in 1st frame will remain 1 in 2nd and upto last frame and similartly connected component marked as 2 in 1st frame will remain 2 in 2nd frame and upto last frame and so on for all other connected components).

one idea is to fix the CC on the bases of number of pixls it contains (but i m facing difculties in coding that idea i need help in cooding).

my first goal is to fix the CC. please help me in codding.thanx

[Rik]

Is this really the time resolution? Those two frames are very far apart. It will be very difficult to match the blobs, and my idea will not work.

[jonas]

• That sounds exactly like what Rik described
• Are those two consecutive images? Because the objects have moved so far that some are not even overlapping..
• Please upload the actual files and not screenshots

[Image Analyst]

I don't know why you say "like tracking but not propere tracking". It certainly is tracking. And it is inherently an ambiguous task if the objects move very far from one frame to another, plus change shape (people turning or in different phases of stepping). Let's say you had 10 blobs at random locations in an image, like a shotgun blast. Now let's say you have 15 blobs at entirely different locations with different shapes in the next image. Can you tell which are new, and which just moved and changed shape while moving? No, or at least not reliably. I think you need to increase your frame rate so things don't move so much and don't change shape so much. Then you at least have a fighting chance.

[QuestionsAccount]

no above attached screenshots are not consective frames, i just upload as an example for showing that some CC are changing position and some are not.

actually intially i didn't want change detection but here my first goal and question is how can i fix/lock the connected components in multiple video frames (that is the conneccted component marked as 1 in 1st frame will remain 1 in 2nd and upto last frame and similartly connected component marked as 2 in 1st frame will remain 2 in 2nd frame and upto last frame and so on for all other connected components).

extremly sorry to say but i did'nt understand Rik idea (may be because of begginer in matlab codding).plz explain it in detail for better understanding.

please help me in codding the above mention work.thanx

attched are some consective frames for example

[jonas]

If you post actual images and not screenshot then people are more likely to write code for you... I dont want to crop your desktop and binarize before even getting started.

[QuestionsAccount]

These are the actual images. I preprocessed original images and after than i got binary image and i want to fix/ lock these CC that i showed in screenshots (means these screenshots are results of preprossed images) if your are talking about original images then i will attach that as well..

Answer 1

[jonas]

Here's something I stitched together that works OK for the images you provided. I had to start by cropping them to the right size.. Note that this method will fail as soon as two blobs merge, disappears or move too much in a frame. All in all, not very robust.

files = dir('C:\Users\pics\*.png');
rect = ([360 130 700 520]);
for i = 1:numel(files)
    BW{i} = imcrop(imbinarize(rgb2gray(imread(fullfile(files(i).folder,files(i).name)))),rect); %very long line
end

%% real action starts here
%build a colormap
cmap = repmat([perms([1 0 0]);perms([1 1 0]);[0,0,0]],2,1);

%start with a black image
im_old = zeros(size(BW{1}(:,:,1)));
k = 1
for j = 1:numel(files)
    %empty new image
    im_new = zeros(size(BW{1}(:,:,1)));
    %load next frame
    im_j = BW{j};
    %remove small components
    im_j = bwareaopen(im_j,20);
    %find all blobs
    S = bwconncomp(im_j,4);
    %check if empty and continue to next frame if so
    if S.NumObjects < 1
        continue
    end
    %loop over objects
    for i = 1:S.NumObjects
        %apply mask of object 1 on previous image
        overlapping = im_old(S.PixelIdxList{i});
        %remove any non-overlapping pixels
        overlapping(overlapping==0) = [];
        if isempty(overlapping) %if no overlapping, assign new label
            im_new(S.PixelIdxList{i}) = k;
            k = k+1;
        else 
            %if overlapping, assign most common label 
            im_new(S.PixelIdxList{i}) = mode(overlapping);
        end
    end 
    %replace old with new
    im_old = im_new;
    %store image as rgb 
    out{j} = label2rgb(im_new,cmap);
end

[QuestionsAccount]

jonas thank you very much for your help/effort.

error:

Index exceeds matrix dimensions.

Error in label2rgb (line 81)

if isequal(zerocolor,cmap(i,:))

Error in angle_all (line 42)

out{j} = label2rgb(im_new,cmap);

This was the error i received on running this code. how i can reslove it?

Also i have 3 qestion regarding this code

• what was the logic you used in this code?
• what this line do? "cmap = repmat([perms([1 0 0]);perms([1 1 0]);[0,0,0]],2,1);"
• what these lines do? "

overlapping = im_old(S.PixelIdxList{i});

%remove any non-overlapping pixels

overlapping(overlapping==0) = [];

[jonas]

One bullet point for each of your questions

• The logic is explained in the comments several times, originally by Rik.
• Creates the color map which is used to color the patches. The reason your code is failing is probably because there are not enough colors in cmap. I assume you ran it with more pictures than me. Replace by this line, and increase number of colors (200) if error persists

cmap = jet(200)
cmap = cmap(randperm(size(cmap,1)),:)

• It takes the mask of connected component i of image t and applies to labelled image i-1. If the blob is stationary between image t-1 and t, then the variable overlapping will exclusively contain the labels from image t-1, which are then assigned to blob i of image t. If the blob has moved, then it also contains a bunch of zeros, which are removed. If it contains only zeros, then it is assumed to be a new blob. If it contains more than a single unique label, then the most common value is passed to blob i of image t.

[QuestionsAccount]

sorry jonas i am replying after long time because of some personal urgent work i did'nt find time to work. now i m back and ready to focus on my work again.

i tried your above code but i encounter with error mention below.

" Index exceeds matrix dimensions.

Error in angle_all (line 33)

overlapping = im_old(S.PixelIdxList{i}); "

how to resolve this error please suggest.

also please guide me how to visulise it that every CC in each frame is fixed.like in attach two images i show manually.

Your help is highly appricated.thanx

[Image Analyst]

What does this show

mask = S.PixelIdxList{i};
whos mask
whos im_old

[QuestionsAccount]

still same error. plz find the attach screenshot

[jonas]

I guess the images may have different size, or there is a bug in the code. Would be easier to verify if you actually show the output of mask and im_old, as Image Analyst asked you to.

You could also upload the actual images and NOT screenshots of your desktop, so that we can run the code in the same way you would.

[Image Analyst]

QuestionsAccount : did you realize that the screenshot you showed actually clipped off the part of the stuff we need to see? We were expecting you to copy and paste the stuff in the command window, not give a screenshot. I cannot scroll your screenshot up to see what was in your command window. I knew my code would not prevent the error of course. But I was expecting it to show that the two images were of different sizes. This forehead-slapping revelation would then lead you to find the reason why, and fix it.

Also, replace the first chunk of your code with this more robust version:

folder = 'C:\users\Pic' % Wherever...
files = dir(fullfile(folder, '*.png'));
rect = ([360 130 700 520]);
for k = 1:numel(files)
	fullFileName = fullfile(files(k).folder, files(k).name);
	grayImage = imread(fullFileName);
	[rows, columns, numberOfColorChannels] = size(grayImage);
	fprintf('%s is %d rows by %d columns by %d color channels.\n', files(k).name, rows, columns, numberOfColorChannels);
	% Make gray scale if it's not already.
	if numberOfColorChannels == 3
		grayImage = rgb2gray(grayImage);
	end
	if islogical(grayImage)
		grayImage = uint8(255 .* grayImage);
	end
	BW{k} = imcrop(imbinarize(grayImage),rect);
end

Try this:

v = S.PixelIdxList{k}
whos v

and you'll see that v = S.PixelIdxList{k} is a vector, not an image. Doing

overlapping = im_old(S.PixelIdxList{k});

makes overlapping a vector not an image. This will be of varying size depending on how many pixels are in the PixelIdxList.

[QuestionsAccount]

images are of the same size but objects are changing size in each frame. (objects size is not same in my case).

attach is the actually images and in screenshot i showed the desired output (i draw different color of bounding boxes on each silheoute and i want to fix the silheoute with that same color of bouding box in full video. like in attach screenshot i draw blue BB on 1st silheoute and i want that in 2nd frame the same color of bounding box (blue) will draw on that same 1st silheoute and no other silheoute with that color of BB and if 1st silheoute disapear the blue BB will also disapear with that silheoute and same will happed with the other silheoutes as well. In other words i want to fix every silheoute with some color of bouding box in full video.

[Image Analyst]

Come on, make it easy for us to help you. You forgot to attach angle_all.m with the changes I suggested. I'll check back later for it.

By the way, this code I suggested works just fine:

folder = pwd
files = dir(fullfile(folder, 'frame_*.jpg'));
rect = ([360 130 700 520]);
for k = 1:numel(files)
	fullFileName = fullfile(files(k).folder, files(k).name);
	grayImage = imread(fullFileName);
	[rows, columns, numberOfColorChannels] = size(grayImage);
	fprintf('%s is %d rows by %d columns by %d color channels.\n', files(k).name, rows, columns, numberOfColorChannels);
	% Make gray scale if it's not already.
	if numberOfColorChannels == 3
		grayImage = rgb2gray(grayImage);
	end
	if islogical(grayImage)
		grayImage = uint8(255 .* grayImage);
	end
	BW{k} = imcrop(imbinarize(grayImage),rect);
end

And I can't write a complete turnkey tracking app for you that keeps unique colors like you want. I've described many times in many posts how difficult it is, with all the ambiguities etc. Like

1. objects occluding each other or merging together or splitting apart
2. objects leaving the field of view, perhaps reappearing later
3. objects entering the field of view
4. objects changing shape from frame to frame
5. objects changing size, color, or some other unique identifying attribute from frame to frame

One thing you'd have to do is make a feature vector with all the attributes describing all the things that are needed to distinguish one blob from another like size, shape, color, speed, etc. and then try to match up feature vectors for the blobs from one frame to the next to decide which blob was present in the prior frame, and if it was, which one it was. This is a lot of work that I just can't donate to you. It would take months or probably years. I'm sure there are companies out there that have teams of people who have spent years on tracking. Mathworks is one but their tracking stuff is just part of the low level stuff you'd need. There is still a whole bunch of stuff you'd need to add beyond what they did, and I'm sure there are companies that have done all that. You just need to find them and buy their software. That will be MUCH, MUCH easier and faster than writing it from scratch on your own.

Answer 2

[Image Analyst]

The Computer Vision Toolbox has lots of tracking capabilities:

https://www.mathworks.com/help/vision/tracking-and-motion-estimation.html?s_tid=CRUX_lftnav

It could save you a lot of time if you bought that toolbox. Do you already have it? If not, buy it.

Answer 3

[Image Analyst]

See Mathworks ball tracking demo: https://www.mathworks.com/matlabcentral/fileexchange/39851-algorithm-development-with-matlab?s_tid=prof_contriblnk

[QuestionsAccount]

Image Analyst thanx alote for your response. but i didn't want any tracking algorithm.

my goal and question was just to fix/lock the connected components in multiple video frames (that is the conneccted component marked as 1 in 1st frame will remain 1 in 2nd and upto last frame and similartly connected component marked as 2 in 1st frame will remain 2 in 2nd frame and upto last frame and so on for all other connected components).

please help me in codding of how can i fix/lock the connected components in multiple video frames. so that CC 1 will remain 1 in all the frames likewise CC 10 will remain 10 in all the frames and so on. thanx

[Image Analyst]

Sorry I can't. Because that would be tracking and you are steadfastly and adamantly against tracking. So you're stuck with whatever default ID label the connected components algorithm assigns. We can't change the ID labels to what they were on a prior frame because that's a tracking process and you won't allow tracking. Good luck though.

[QuestionsAccount]

tracking is something else and my goal is something else although i know this process is also involve in tracking but my purpose is not to track these CC i just want to fix/lock them for some other purpose. fixing/ locking CC in multiple frames can be used for lots of many purposes otherthen tracking. if i want to track then i asked directly for tracking.

[Rik]

No. This is tracking. You can claim it isn't, and your end goal might not be actually to track something, but that is what is required to achieve what you describe.

Why don't you try it instead of ignoring his advice? Image Analyst has probably forgotten more about image analysis with Matlab than I have learned. You're getting free help from an expert, you would do well not to ignore it.

[Image Analyst]

You'd have to make a list of features of all the blobs. Then I'd be tempted to try K-Nearest Neighbors, knnsearch() to match blobs from one feature set to the prior feature set. Sorry, I don't have a demo for this already. But it's simple so you can try it yourself easily I'd think. You need the Statistics and Machine Learning Toolbox to use that function. However it will match each blob in set 2 with a blob in set 1 regardless if it's correct or not. I mean the blob in set 1 might have left the scene and the blob in set 2 may have just entered the scene but it will say that some blob in set 1 matches the blob in set 2. In that case, perhaps (but maybe not) some blob in set 1 may have two blobs that match it in set 2. Or it may have none. So you'll need to inspect the number of matches and if there are zero, or two or more, you're going to have to make some decisions about what to do. But if all goes well and you have the same blobs in both sets (none left and none entered), and the features didn't change much from set 1 to set 2, then each blob will have exactly one matching blob in the other set.

[Image Analyst]

You may want to watch this webinar tomorrow:

Join us for the fourth of seven bi-weekly free seminars focusing on the best paper research presented at EI 2020. Talks will be followed by live discussion: seminars will be recorded.*

Best Paper Imaging and Multimedia Analytics in a Web and Mobile World 2020 Conference

LambdaNet: A Fully Convolutional Architecture for Directional Change Detection

Bryan Blakeslee & Andreas Savakis (Rochester Institute of Technology)

Change detection in image pairs has traditionally been a binary process, reporting either "Change" or "No Change." In this paper, we present LambdaNet, a novel deep architecture for performing pixel-level directional change detection based on a four-class classification scheme. LambdaNet successfully incorporates the notion of "directional change" and identifies differences between two images as "Additive Change" when a new object appears, "Subtractive Change" when an object is removed, "Exchange" when different objects are present in the same location, and "No Change".

About the Speaker

Bryan Blakeslee is a recent graduate of the Rochester Institute of Technology’s computer engineering program. His areas of interest are deep learning and embedded systems.

Join us on Wednesday, 26 August 2020

10:00 - 10:45 EDT / 15:00 - 15:45 BST / 16:00 - 16:45 CET