fminsearch: infinite recursion within the program

The function below takes coin flip outcomes (e.g., "HTTHTHHT") as input and generates a randomness score for each outcome. As you can see, this function has two free parameters, "rep" and "prior".

 function randomness = benHMMN(flips,rep,prior)

I had group of people rate the degree of randomness for the same bunch of coin flip outcomes. I wish to find the values of "rep" and "prior" that best fit the human data (i.e., model predictions have the highest possible correlation with human judgments) using fminsearch.

flips = csvread('flips.csv');
human = csvread('judgment.csv');
nrand = human(:,2); % human ratings for "non-randomness"
[N K] = size(flips);

In order for fminsearch to work, I converted human randomness ratings to "non-random", so that the smaller the correlation, the better the model.

minNR = @(w) corr(benHMMN(flips,w(1), w(2)), nrand)
[w,fval] = fminsearch(minNR, [0.525 0.107])

Everything worked fine when I used MATLAB 2014a a few months ago. Today, I collected some new human data and wanted to fit the model again, but 2014a stopped working. So I installed 2017a (student license + optimization toolbox + statistics and machines learning toolbox) and performed fminsearch again. However, I constantly get an error message now: " Out of memory. The likely cause is an infinite recursion within the program."

Why would this happen? The code is exactly the same as before, just that the MATLAB version is different. What should I do to fix this error? Thanks!!

Below is the rest of the code:

incell = transmatrix(rep,prior);
trans = incell{1};
obs = incell{2};
privect = incell{3};
states = length(privect);

pxregular = zeros(N,1);
for i = 1:N
x = zeros(states,K);
x(:,1) = privect.*obs(:,flips(i,1));
for j = 2:K
  match = zeros(1,states);
  match = ones(states,1)*obs(:,flips(i,j))';
  x(:,j) = max(trans.*match .* (x(:,j-1)*ones(1,states)))';
end
pxregular(i) = max(x(:,K));
end
randomness = -K*log(2)-log(pxregular);
function outcell = transmatrix(rep,prior)
motifs = 22;
motif{1} = '1';
motif{2} = '2';
motif{3} = '12';
motif{4} = '21';
motif{5} = '112';
motif{6} = '121';
motif{7} = '211';
motif{8} = '221';
motif{9} = '212';
motif{10} = '122';
motif{11} = '1112';
motif{12} = '1121';
motif{13} = '1211';
motif{14} = '2111';
motif{15} = '1122';
motif{16} = '1221';
motif{17} = '2211';
motif{18} = '2112';
motif{19} = '2221';
motif{20} = '2212';
motif{21} = '2122';
motif{22} = '1222';

states = 0;
for i = 1:motifs
states = states + length(motif{i});
end
T = zeros(states,states);
O = zeros(states,2);
privect = zeros(1,states);
en = 0;
for i = 1:motifs
be = en+1;
en = en+length(motif{i});
privect(be) = prior^length(motif{i});
end
privect = (1-rep)*privect/sum(privect);

P = privect';
en = 0;
for i = 1:motifs
be = en+1;
en = en+length(motif{i});
for j = be:en
  T(j,:) = privect;
  T(j,be) = 0;
  O(j,motif{i}(j-be+1)-48) = 1;
end
if (be==en)
  T(be,be) = rep;
else
  for j = be:(en-1)
    T(j,j+1) = rep;
  end
  T(en,be) = rep;
end
end

outcell{1} = T; % transition matrix
outcell{2} = O; % emission matrix
outcell{3} = P; % initial state distribution

[Walter Roberson]

You do not show us the code for benHMMN . You show us a script and some functions, but it is not obvious what the script is for.

You also do not include your .csv, so we cannot run the code to debug it.