Large data mex file crashes generating 2e6 x 100 matrix
3 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
The main objective is take a vector(N,1), add random errors from a matrix (N,S), and output an (N,S) matrix such that I can run a simulated estimator for a larger S. Matlab is taking a lot of memory to hold errors for S = 100 simulations, so I decided to try doing this simple arithmetic in a C-mex file where the mex file loads a large error.txt file, adds it to the vector, and outputs the matrix. It works in theory (300x100 matrix) but when I ramp up to N = 2e6, let's say, Matlab crashes when I try the mex function.
I've also tried to compile for larger data arrays (tag: -largeArrayDims) and switched my int variables to mwSize, though it still crashed. I also just generated a hypothetical 2e6 x 100 matrix to see if the memory on my machine was okay, and it is. So I'm just not sure why this happens as it doesn't seem so large yet, and any thoughts would be greatly appreciated! My code is below, and I do this for two input vectors and output matrices. ~ Thank you!
/* Include files */
#include "math.h"
#include "mex.h"
#include "matrix.h"
#include <stdio.h>
/* Checks whether inputs are matrices */
#define IS_REAL_2D_FULL_DOUBLE(P) (!mxIsComplex(P) && \
mxGetNumberOfDimensions(P) == 2 && !mxIsSparse(P) && mxIsDouble(P))
#define IS_REAL_SCALAR(P) (IS_REAL_2D_FULL_DOUBLE(P) &&
mxGetNumberOfElements(P) == 1)
/*************************************************/
/* MAIN */
/*************************************************/
void mexFunction(int nlhs, mxArray *plhs[], int nrhs, const mxArray *prhs[])
{
/* Define output matrix */
#define fillAMat plhs[0]
#define fillBMat plhs[1]
/* Define input matrices */
#define Amat prhs[0]
#define Bmat prhs[1]
mwSize a, b, i;
/* Check the number of inputs is correct (Aoffer,Baccept,QbarA) */
if(nrhs < 2)
mexErrMsgTxt("Too few input arguments.");
else if(nrhs > 2)
mexErrMsgTxt("Too many input arguments.");
if(!IS_REAL_2D_FULL_DOUBLE(Amat))
mexErrMsgTxt("dataVec must be real 2D full double array.");
if(!IS_REAL_2D_FULL_DOUBLE(Bmat))
mexErrMsgTxt("dataMat must be real 2D full double array.");
/* Dimensions and pointers for inputs */
mwSize A = mxGetM(Amat);
mwSize S = 100;
mwSize N = S*A;
double *Amat_pt = mxGetPr(Amat);
double *Bmat_pt = mxGetPr(Bmat);
/* Output matrix */
fillAMat = mxCreateDoubleMatrix(A,S,mxREAL);
double *fillA_pt = mxGetPr(fillAMat);
fillBMat = mxCreateDoubleMatrix(A,S,mxREAL);
double *fillB_pt = mxGetPr(fillBMat);
/* Assignment test */
/* printMatEl(A_pt,Bset_pt,A,B); */
/***************************************************************/
/* Begin re-allocating */
/***************************************************************/
FILE *infileA, *infileB;
infileA = fopen("Error.txt", "r");
if (infileA == NULL) {
fprintf(stderr, "Can't open input file A in.list!\n");
exit(1);
}
float seconds;
char line[N];
i = 0; a = 0; b = 0;
while(fgets(line, N, infileA) != NULL)
{
if (a == A)
{
a = 0;
b++;
}
sscanf(line, "%f", &seconds);
/* error[i] = seconds; */
fillA_pt[a + A*b] = Amat_pt[a] + seconds;
i++; a++;
}
fclose(infileA);
infileB = fopen("Error.txt", "r");
if (infileB == NULL) {
fprintf(stderr, "Can't open input file B in.list!\n");
exit(1);
}
i = 0; a = 0; b = 0;
while(fgets(line, N, infileB) != NULL)
{
if (a == A)
{
a = 0;
b++;
}
sscanf(line, "%f", &seconds);
/* printf("%f\n",seconds); */
/* error[i] = seconds; */
fillB_pt[a + A*b] = Bmat_pt[a] + seconds;
i++; a++;
}
fclose(infileB);
free(line);
}
0 comentarios
Respuestas (2)
Titus Edelhofer
el 17 de Ag. de 2015
Hi,
I'm not sure where the crash comes from. But what I think is strange is the size of variable line. Why is it N, i.e., A * S? Shouldn't it be just the size to be able to hold one floating point number? In this case something like 40 should be more than enough ...?
Otherwise: on this page debugging of mex files is described, maybe when you see the line where it crashes that might help.
Titus
0 comentarios
Titus Edelhofer
el 17 de Ag. de 2015
Hi,
another thought: I'm not convinced that using a mex file will help you save memory. If you read the matrix S from your file into memory, it will have about 1.5 GB. This is regardless of doing this in C or in MATLAB, it's the same.
Once you have the matrix S in memory, adding your vector X of size Nx1 via
S = bsxfun(@plus, S, X);
will not increase the memory (significantly).
Titus
0 comentarios
Ver también
Categorías
Más información sobre Write C Functions Callable from MATLAB (MEX Files) en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!