Hi Alfonso, Are you suggesting to me that the solution should not depend on rng settings? If that's the case, I need to increase the tolerance values in the test suite such that rng setting needs not be changed manually in one's solution. Any other suggestions from you would be greatly appreciated. Thanks.
I was just trying to figure out what the EvaluateSolution.p code might be doing. Yes, there is some strange behavior where "exact" solutions are not accepted, and sufficiently close random solutions are only accepted for very specific seed values (which would be impossible to guess for players). If possible, I would suggest just to post the actual code in EvaluateSolution or at least explicitly say what it is testing for so players do not have to guess
Thanks for your suggestions. My original purpose was to focus only on Monte Carlo simulations, and thus I intentionally put some checks in EvaluateSolution.p to reject, if possible, any solutions based on analytical expressions only. I understand it is impossible to achieve so for experienced users who might still try to bypass those checks and submit analytical solutions. As you suggested, I will explain explicitly what EvaluateSolution.p is testing and checking. The seed value issue might be due to the imperfection of pseudo random generators. Some specific seed values could generate random numbers which appear to be “more random” with respect to a specific problem. I took advantage of this by selecting a specific “good” seed value so that less trials are required to achieve the specified accuracy (which translates to less running time). If an arbitrary seed value is used (e.g., shuffle), then more trials are needed to achieve the same accuracy. As you commented, I will slightly loosen the tolerance requirement so that only a reasonable amount of trials could achieve the accuracy. Moreover, I will disallow the usage of rng so that similar attempts to taking advantage of the pseudo random number imperfection are avoided. Thanks again for your suggestions.
Thanks for the response!. And yes, disallowing analytical solutions is rather complicated because once players have an exact solution they can very easily compute a random instantiation of those estimates for any arbitrary sample size (e.g. using binomial distribution properties), which should be impossible to tell apart from actual Monte Carlo simulation results, so I really do not see a good way to effectively disallow those types of solutions. The "good seed" approach has the problem that it will only work for a very specific form of Monte Carlo simulation (other perfectly valid ways to generate your random samples would not benefit from the "good seed" selection, so you are effectively disallowing all forms of Monte Carlo simulations except the very specific one that you have in mind). In any way, just my two cents, and thanks for the very interesting problem!
Thanks for the insightful comments. I just updated the problem with a more detailed description of the Pfile. Also, I slightly improved the checks in the Pfile which should better tell apart (as I hope) analytical solutions from Monte Carlo simulations.
Test  Status  Code Input and Output 

1  Pass 
%%
% Thanks to Alfonso NietoCastanon
urlwrite('https://sites.google.com/a/alfnie.com/alfnie/software/SetSolutionScore.p?attredirects=0&d=1','SetSolutionScore.p');
rehash path;

2  Pass 
%%
fh = fopen('EvaluateSolution.p','wb');
fwrite(fh, hex2dec(reshape('7630312E30307630302E3030000C701C97F61FB1000002D5000001CF000004E73DD68930A391F7C60A534B45A03EAF72EB08941F39EE01BF25BAE04DF43CF342FC1A763DF6B8F26BBED0BD4F2ABBB5927B1EEBAE8795E487F6E4EF2737CBB6646BC4DF145E14664B3A4DACCD7CB01C4EC2328AD76F196231D2CA02CDC2B15466FBA5BDDF9E6C0E5DE12CF07B2AAA50BD2F04FFB92E9BECBE232E01031340A8EDCA5C10DAC01BBE43685ED0AB79D9C6F2A090FB4E0E75CAA236D3D73AD659E0705C42792BC77D85951B2FC49DE856FB97AFD74C1DB66C874EDF5517BCFA14C6706CA5E61DE60F2771B64F6D634B858A1A30AD7C49778534CCCE7551C637DC53846B02140046F729C5EB2DC9C65F16C2FC4F34EA9F03A2056B8218ACAB9A9A8BC5DC8F2F3312740F86626ABA38E00903CE76846DEE175BEE04DC0815E050E4CD95BA8E5BD27A0B57F2413B71A0E4837FB0F86328AA82732C584F1F55C6CCD79CBC69D052011BA93357AAE3568E0086F159C083D665645A38584955283925F900254A6562F0C755323C40805328D04F27FD863E8B774B52E27FC3AA30CE689AC57DEFEE274DBBC2A4D9D320CF19AD873AA0AE806721EE78496F7ADA99EA48060F26FDEE7ED5E9F85B27F79AE176A6E8EAC4DD5127299122FF88139BC4B56A2ED3C5338A72676C4C99AD6DEBCD8BB5EB09',2,[]).')); rehash path; fclose(fh);

3  Pass 
%%
fid = fopen('Xiangqi2.m');
delim = {' ', '\n', ',', '.', ';', '''', '@', '+', '', '*', '/', '\', '^', '>', '<', '=', '&', '', '~', '{', '}', '[', ']', '(', ')'};
file = textscan(fid, '%s', 'CommentStyle', '%', 'MultipleDelimsAsOne', 1, 'Delimiter', delim); fclose(fid);
assert(~any(ismember({'rng','RandStream','seed','state','twister','shufle','default'},file{1})));

4  Pass 
%%
a = 0; b = 0; Na = 2; Nb = 3; Nc = 2; tol = 1e6;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

5  Pass 
%%
a = 0; b = 1; Na = 1; Nb = 2; Nc = 1; tol = 1e6;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

6  Pass 
%%
a = 1; b = 0; Na = 3; Nb = 2; Nc = 1; tol = 1e6;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

7  Pass 
%%
a = 0.15; b = 0.85; Na = 4; Nb = 2; Nc = 1; tol = 1e4;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

8  Pass 
%%
a = 0.9; b = 0; Na = 3; Nb = 1; Nc = 2; tol = 1e3;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

9  Pass 
%%
a = 0.65; b = 0.3; Na = 3; Nb = 2; Nc = 2; tol = 1e3;
EvaluateSolution(a, b, Na, Nb, Nc, tol);

10  Pass 
%%
Na = 3; Nb = 2; Nc = 1; tol = 2e3;
p = sort(rand(2,30));
p = sort([p(1,:);diff(p);1p(2,:)]);
for k = size(p,2):1:1
a = p(3,k); b = p(2,k);
score(k) = EvaluateSolution(a, b, Na, Nb, Nc, tol);
end
SetSolutionScore(round(mean(score)));

1585 Solvers
1220 Solvers
786 Solvers
116 Solvers
230 Solvers