Hi Matt,
At a general level - and speaking from too much personal experience - expectations about geometrical operations such as the convex hull are often not met when we're dealing with a discrete grid instead of a continuous domain. Since bwconvhull is inherently working on inputs and outputs on a discrete grid, it is really only an approximation of a true convex hull, and I wouldn't expect it to be exactly idempotent.
With approximations, different approximation techniques usually have their own pros and cons, and one can consider whether one approximation technique is better than another, and under what conditions.
Your function treats pixels as points. It computes the convex hull of all the pixel centers and then uses inpolygon to "rasterize" the resulting shape. A disadvantage of that approach is that it will make skinny, single-pixel lines disappear competely because the convex hull of the pixel centers is a degenerate polygon with no area. On the following input, your function produces no foreground pixels at all in the result.
W = zeros(30,30);
W(15,10:20) = 1;
I don't think we could ship this method without somehow addressing that issue.
The function bwconvhull treats pixels as squares having unit area. It computes the convex hull of the set of all the pixel corners and then uses roipoly to rasterize the result. This method behaves better for skinny shapes, but it behaves less well in terms of idempotence, as you have clearly demonstrated.
Oversimplifying a bit, it comes down to tie-breaking rules related to polygons that go exactly through a pixel center. Is such a pixel inside or outside the polygon? Each choice results in its own set of nonideal behaviors.
It is worth asking, though, whether the method in bwconvhull can be improved with respect to idempotence while avoiding the degeneracy I mentioned above. We can take a look at this.
