maybe I can help you, because I came across this post when looking for "how to open a mex file". Now I understood that is a compiled function (thanks Jan), but I still wanted to double check the underslying .m file where this function was created from.
My mex file was created by using the Matlab coder. I created it from a .m file. Matlab generated automatically a folder named "codegen", and inside this there is a folder named "mex". In this folder, I can find the folder that has the same name as my original Matlab file. Inside this folder, I see the "html" folder and there, finally, I find the report.mldatx. In this report, I can double-check the original .m function associated to the compiled mex file.
Hope this helps!