Ways to get around using 7.3 version of mat files

I am building a set of tools that will regularly deal with datasets larger than 2 GB and I want users to be able to save their current state including all loaded data to make getting back into the tool as easy as can be. All of the users data and setup is contained in a class object. I initially got around this by using the 7.3 version of the mat file, but the file size and additional time required for these really hurts the efficiency. I was trying to circumvent this, but I am running into some issues. Below are my attempts and the issues I've run into:

1. Since the 2GB is a per variable limit, I was planning to split the data into multiple variables inside my saveobj method, but as far as I can tell saveobj must return a single variable that will be saved in the mat file. This make sense, and I can see why the question I am about to ask is nonsense, but I'll ask anyway. Can saveobj return multiple variables that saved separately in the mat file?
2. Since I am guessing the answer to #1 is no, I was thinking I could create a separate function within my class that could save the data into multiple mat files. I could then call this separate function with saveobj to split the data and save it. I would then need to know what those extra files were named in the loadobj function so that I could restore the data. My first thought here was to use the mat file name that was used in the call to saveobj and just append something to it. Is there a way to get the mat file name being saved to/loaded from within the saveobj and loadobj functions?

If you have other ideas, I am open to those as well.

Here is a simple code example:

classdef LargeData

  properties
    Data
  end

  methods

    function filenames = saveDataToDisk(obj,filename)
      % Split into 2 sections for demonstrative purposes
      numSplits = 2;
      numRows = size(obj.Data,1);
      numRowsPerSplit = ceil(numRows/numSplits);
      filenames = cell(numSplits,1);
      for iS = 1:numSplits
        startIdx = 1+(iS-1)*numRowsPerSplit;
        endIdx = min(numRows,startIdx + numRowsPerSplit - 1);
        if startIdx < endIdx
          filenames{iS} = sprintf('%s_%1',filename,iS);
          thisSplitData = obj.Data(startIdx:endIdx,:);
          save(filenames{iS},'thisSplitData','-mat');
        end
      end
    end % saveDataToDisk

    function sobj = saveobj(obj)
      % Assume the data is large, real implementation has protection for
      % small data to avoid complexity
      
      % Save the data to separate files
      % HOW CAN I DETERMINE THE MAT FILE BEING SAVED TO SO THAT I CAN
      % KEEP THE DATA WITH THE MAT FILE THAT THE USER SAVED TO?
      sobj.dataFiles = obj.saveDataToDisk(tempname);
      % Remove the data so that a v7 mat file can be used
      sobj.origObj.Data = [];
    end % saveobj

  end

  methods (Static)

    function obj = loadobj(s)
      if isstruct(s)
        obj = LargeData;
        if isfield(s,'dataFiles')
          for iF = 1:numel(s.dataFiles)
            newData = load(s.dataFiles{iF},'-mat','thisSplitData');
            if iF == 1
              data = newData.thisSplitData;
            else
              data = [data;newData.thisSplitData];
            end
          end
          obj.Data = data;
        end
      else
        obj = s;
      end % isstruct
    end % loadobj

  end % Static methods

end

[dpb]

I don't suppose these are all base data class variables are they, but structs and/or tables, etc., etc., etc., ...? So that just using fread,fwrite is out without writing serializers...

I'm too old of an old fogey to have gotten into writing classes in any language so I've no familiarity at all with them in MATLAB other than at the absolute most trivial level, but...it appears that one can write your own save/load object functions for your class from root principles...this would not be for the faint of heart, no doubt, but you did ask! :)

[Matt Butts]

Are you referring to the saveobj and loadobj methods or something else? I am already working within the saveobj and loadobj, but if there are other options, I'm all for exploring them. Any links to more about this?

[Jan]

"I initially got around this by using the 7.3 version of the mat file, but the file size and additional time required for these really hurts the efficiency." - what exactly is the problem? Do you use the 7.3 format with or without compression? Is the file size the problem, e.g. because you work on a network drive over a 100mbit Ethernet connection or a NAS drive which is heavily loaded? Then the MAT format is not the problem, but the disk or network.

If the processing time for the compression is the problem, use the '-nocompression' flag. Of course this increases the file size.

Writing the data to a set of MAT files (in which format version? -v6?) is not a magic solution: The same amount of data must be read, even if it is divided into chunks.

Would saving the variables in uncompressed binary files solve the problem?

[Walter Roberson]

The overhead for structured data types is much higher in -v7.3 files.

[Matt Butts]

The speed is the biggest issue, but all of my trials have suggested that I can get a speed and file size benefit from the v7 format. Using my actual implementation, here are the results I have gotten:

• Using v7.3 and no compression: 571 s to save 1 file totaling 4.47 GB
• Using v7.3 and compression: 573 s to save 1 file totaling 2.59 GB
• Using v7 and compression: 44s to save 7 files totaling 102 MB

This really seems to suggest that the v7 mat file is the right solution. I just need to figure out how to keep the extra files stored with the filename that was input to the save(...) command.

[Jan Kappen]

@Matt Butts

have you found a solution? I'm facing the same issue and -v7.3 is just not usable when many class instances shall be saved. It's a real nightmare and I had to spend weeks of optimizations so far :/