There are several options in the vision.CascadeObjectDetector that you can tweak. If you know how large you expect the faces to be in your images, you can set MinSize and MaxSize to eliminate the false detections that are too small or too big to be a face. You can also try increasing MergeThreshold, or set an ROI (region of interest) to exclude the parts of the image where you do not expect to see any faces at all.
Alternatively, you can also try using a different model: Frontal Face (LBP) instead of the default Frontal Face (CART). You can also try detecting the upper bodies, using the Upper Body model, and then detect the faces inside the resulting bounding boxes.