Tracking and detecting shapes

Hi there, I have a problem when i try to detect white and black squares in a cheekerboard, inside squares there are red or blue points which represents objects like in this pic:

After i try to detect white and black squares i get this result:

Result is good i can detect white and black squares but it also detects red and blue's circles as a square, something that i don't want to happen :/, is there any way to ignore these circles cause i want them to be only part of squares not a "square" like this:

This is an edited image, but that's the result that i want to get.