The error message might occur if you try to store matrix in the mat file which has size greater than the maximum size supported by that particular version of MAT file.
There are several subsequent versions of MAT-files that support an increasing maximum size of variables that can be stored in the MAT files. Refer the below link for more information:
https://www.mathworks.com/help/matlab/import_export/mat-file-versions.html
Modify the version of your MAT file as per your need to resolve this issue.
