fitclinear appears to use sgd solver even when sparsa is specified

Hi there,

I'm training some SVM's on moderate-dimensional data (a few thousand observations by [less than or equal to a few hundred] features) using the following core function call:

model = fitclinear(X_subset(train_idx,:),...
    Y_subset(train_idx),...
    'Regularization', 'lasso',...
    'Solver', {'sparsa'});

In order to assess the degree to which performance depends on the features jointly, rather than individually, I add them in one by one and re-run the classifier. The accuracy curve that I get out looks like this: (red and blue are two different experimental replicates)

As you can see, once the number of dimensions passes 100, the solver's accuracy changes dramatically, getting less accurate and also getting more stochastic. I presume this is because on the backend, MATLAB is changing the solver from sparsa to sgd, as implied by the docs but not explicitly stated. For a second set of data that I have, where the overall accuracy is higher (~80%), the effect is still present but not as dramatic.

Is there a way to prevent MATLAB from switching to sgd? I will try passing in the cofficients from the nfeatures=100 model as a warm start to the subsequent models, but even if that fixes my specific problem, this feels like a bug more generally worth reporting.

Thanks.

Answer 1

[the cyclist]

This is interesting. I'd be surprised that MATLAB makes that transition when you have explicitly specified the Solver .. but I agree with you that the mention of 100 features in the documentation is a tantalizing hint that it might be happening.

Can you upload the data? I'd be pretty interested to investigate. (I could also create a simulated dataset. This probably doesn't depend the exact data.)

Here would be my approach to trying to confirm your hypothesis. You can use the debugger to pause execution inside fitclinear, then step through the program to see where the Solver is actually set. You could then see whether MATLAB is actively ignoring the Name-Value input.

You might be able to make a copy of the MATLAB code (and put it in your path), then adapt that code to do what you want.

I will mention that I think it is also possible that you are just seeing some phenomenon where the lasso is failing to regularize (or finding some local minimum instead of a global one), but it seems to too extraordinarily coincidental.

[Jonah Pearl]

Thanks for the response -- I will try stepping through. The analysis isn't so straightforward (I'm balancing the classes in a particular way), and it's on some un-published data. I'd be happy to upload it to a google drive as a CSV + script and send you a link privately. If you try to reproduce it, you should use fairly sparse observations (~half obs are 0), and draw the features values from poisson distributions with fairly low means (1-5).

[Jonah Pearl]

AFAICT, the MEX code at the core of fitclinear (in `LinearImpl.m`) is being passed sparsa as the solver. So either it's a weird MEX optimization, or that's not the issue.

[the cyclist]

Sorry for the delayed reply.

It's a shame you get thwarted by MEX code. I'm not sure I have any further advice. Maybe if you contact support they could help you.

[Bruno Luong]

I don't know this function, but just my 2cents that sparse only supports double and not single. Do you have a single in your training data?

[Jonah Pearl]

You're referring to the data type of the values? I'll check -- but why would that effect only appear at > 100 dims?

[Bruno Luong]

As I said, I don't know ths function (I don't own the toolbox), just a wild guess so it make be wrong. just convert (cast) X_subset and Y_subset to double see if it makes any difference.

[the cyclist]

@Bruno Luong, I think that when @Jonah Pearl used the word "sparse" in a prior comment, he just meant that one class of observations has significantly fewer instances than the other. I don't think this has anything to do with the sparse matrices.

Also not to be confused with the sparsa (Sparse Reconstruction by Separable Approximation) solver.

[Jonah Pearl]

Sorry didn't mean to introduce jargon! But by sparse, I meant, a large fraction of all observations are 0's. However I'm not using any sparse matrices (ie the MATLAB class) in this analysis. I'll give it a try later.