Erase many layers of a plot, leave the background

I use the rectangle function to draw colored bounding boxes on top of an image containing text. I would like to be able to remove the bounding boxes and draw new ones without having to recreate the entire plot. Is there a way to do that? I saw a post about doing this with a single layer (<http://www.mathworks.com/matlabcentral/answers/1444-clear-one-plot-in-multiple-hold-figure)>, but that doesn't seem scalable to many layers.

Example code:

    imshow(I);
    c = colormap(jet);
    for i = 1:x
        % Plot the bounding boxes
        for j = 1:y
            rectangle('Position', pos, 'EdgeColor', c(j, :));
        end
        % Clear boxes, leave background image
    end