We don't distribute the source code for the undocumented helper function that rmoutliers uses. [Note the extra s in the name; the function is rmoutliers nor rmoutlier.]
For your second question, if you're asking how rmoutliers determines if something is or is not an outlier, see the description of the method input argument on the documentation page. The second of the Topics pages listed in the See Also section on that page may also be of interest to you.
For your third question, it is possible to write a program without using rmoutliers. The function was only introduced in release R2018b and I've been writing programs in MATLAB since 2000. It is even possible to write a program to remove outliers without using rmoutliers, if you're willing to code the outlier detection yourself (or use isoutlier.)
