Large data mex file crashes generating 2e6 x 100 matrix

The main objective is take a vector(N,1), add random errors from a matrix (N,S), and output an (N,S) matrix such that I can run a simulated estimator for a larger S. Matlab is taking a lot of memory to hold errors for S = 100 simulations, so I decided to try doing this simple arithmetic in a C-mex file where the mex file loads a large error.txt file, adds it to the vector, and outputs the matrix. It works in theory (300x100 matrix) but when I ramp up to N = 2e6, let's say, Matlab crashes when I try the mex function.

I've also tried to compile for larger data arrays (tag: -largeArrayDims) and switched my int variables to mwSize, though it still crashed. I also just generated a hypothetical 2e6 x 100 matrix to see if the memory on my machine was okay, and it is. So I'm just not sure why this happens as it doesn't seem so large yet, and any thoughts would be greatly appreciated! My code is below, and I do this for two input vectors and output matrices. ~ Thank you!

/* Include files */

#include "math.h"

#include "mex.h"

#include "matrix.h"

#include <stdio.h>

/* Checks whether inputs are matrices */

#define IS_REAL_2D_FULL_DOUBLE(P) (!mxIsComplex(P) && \

mxGetNumberOfDimensions(P) == 2 && !mxIsSparse(P) && mxIsDouble(P))

#define IS_REAL_SCALAR(P) (IS_REAL_2D_FULL_DOUBLE(P) && 

mxGetNumberOfElements(P) == 1)

/*************************************************/

/* MAIN */
/*************************************************/

void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{

  /* Define output matrix */

  #define fillAMat      plhs[0]

  #define fillBMat      plhs[1]

  /* Define input matrices */

  #define Amat   prhs[0]

  #define Bmat  prhs[1]

  mwSize a, b, i;

  /* Check the number of inputs is correct (Aoffer,Baccept,QbarA) */

  if(nrhs < 2)

    mexErrMsgTxt("Too few input arguments.");

  else if(nrhs > 2)

    mexErrMsgTxt("Too many input arguments.");

  if(!IS_REAL_2D_FULL_DOUBLE(Amat))

    mexErrMsgTxt("dataVec must be real 2D full double array.");

  if(!IS_REAL_2D_FULL_DOUBLE(Bmat))

    mexErrMsgTxt("dataMat must be real 2D full double array.");

  /* Dimensions and pointers for inputs */

  mwSize A = mxGetM(Amat);

  mwSize S = 100;

  mwSize N = S*A;

  double *Amat_pt = mxGetPr(Amat);

  double *Bmat_pt = mxGetPr(Bmat);

  /* Output matrix */

  fillAMat = mxCreateDoubleMatrix(A,S,mxREAL);

  double *fillA_pt = mxGetPr(fillAMat);

  fillBMat = mxCreateDoubleMatrix(A,S,mxREAL);

  double *fillB_pt = mxGetPr(fillBMat);

  /* Assignment test */
/*  printMatEl(A_pt,Bset_pt,A,B); */

/***************************************************************/
/* Begin re-allocating        */
/***************************************************************/
  
  FILE *infileA, *infileB;

  infileA = fopen("Error.txt", "r");
  
  if (infileA == NULL) {

    fprintf(stderr, "Can't open input file A in.list!\n");

    exit(1);
  }

  float seconds;

  char line[N];
  
  i = 0; a = 0; b = 0;

  while(fgets(line, N, infileA) != NULL)

  {

    if (a == A)

    {

      a = 0;

      b++;

    }

    sscanf(line, "%f", &seconds);

/*    error[i] = seconds; */
    
    fillA_pt[a + A*b] = Amat_pt[a] + seconds;

    i++; a++;
  }

  fclose(infileA);

  infileB = fopen("Error.txt", "r");

  if (infileB == NULL) {

    fprintf(stderr, "Can't open input file B in.list!\n");

    exit(1);

  }

  i = 0; a = 0; b = 0;

  while(fgets(line, N, infileB) != NULL)

  {

    if (a == A)

    {

      a = 0;

      b++;

    }

    sscanf(line, "%f", &seconds);

/*    printf("%f\n",seconds); */
/*    error[i] = seconds; */
    
    fillB_pt[a + A*b] = Bmat_pt[a] + seconds;

    i++; a++;

  }

  fclose(infileB);

  free(line);

}

Answer 1

[Titus Edelhofer]

Hi,

I'm not sure where the crash comes from. But what I think is strange is the size of variable line. Why is it N, i.e., A * S? Shouldn't it be just the size to be able to hold one floating point number? In this case something like 40 should be more than enough ...?

Otherwise: on this page debugging of mex files is described, maybe when you see the line where it crashes that might help.

Titus

Answer 2

[Titus Edelhofer]

Hi,

another thought: I'm not convinced that using a mex file will help you save memory. If you read the matrix S from your file into memory, it will have about 1.5 GB. This is regardless of doing this in C or in MATLAB, it's the same.

Once you have the matrix S in memory, adding your vector X of size Nx1 via

S = bsxfun(@plus, S, X);

will not increase the memory (significantly).

Titus