Is there a way to create a custom cvpartition?
14 visualizaciones (últimos 30 días)
Mostrar comentarios más antiguos
Hi,
I'd like you use crossvalidation in a model with a custom set of training/test sample vectors.
To use some of the built-in functionalities of Matlab I'd like to pass the input via the CVPartition Name/Value input parameter. Is there a way to define a custom set of train/test indices using cvpartition?
Thanks,
Christian
0 comentarios
Respuestas (3)
Shraddha Jain
el 25 de Jun. de 2021
Hi Christian,
As of now, the functionality to use custom indices in the cvpartition object is not available in the Statistics and Machine Learning toolbox. It might be considered in a future release.
3 comentarios
Giovanni Attolico
el 13 de Oct. de 2021
It seems to me that the indices are stored inside the cvpartition variable but they are read-only using the functions "training" and "test". While waiting for the "future release", it is not possible to define two functions "settraining" and "settest" that write indices in the variable? That would be really useful in many situation to allow the use of the resulting variable in all the situations where the current random partition can be used ...
Thanks
Giovanni
el 9 de En. de 2024
Custom partition has been recently introduced in MATLAB r2023B, but I suspect it does not work as expected since cross validation results from 2 neural netwroks differ even if trained and validated with the same folds (cv partition using the custom partition options with indices generated by crossvalind) and the networks are trained with the same parameters.
clear variables
close all
clc
tab = readtable("three_selected_feature.xlsx"); %features dataset
featuresTab = tab(:,2:end-1);
feature = table2array(tab(:,2:end-1));
y = table2array(tab(:,end));
for i = 1:6 %repeat for different training test splitting.
cTest = cvpartition(y,'HoldOut',0.2,'Stratify',true);
train = training(cTest);
y_train = y(train);
testing = test(cTest);
cvIndices = crossvalind('Kfold',size(y_train,1),5);
cv = cvpartition("CustomPartition",cvIndices);
% cv = cvpartition(y(train),"Resubstitution");
features_scaled = normalize(feature,'zscore');
modelNET1 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
modelNET2 = fitcnet(features_scaled(train,:),y_train,'PredictorNames',featuresTab.Properties.VariableNames,'CVPartition',cv);
%models trained with the same parameters and features and crossvalidated
%using custom partitions. cv is based on cvIndices generated by the
%crossvalind function. For the same "i" (for loop) results of the two
%models should be the same, but they differ.
% cvmodelLDA1 = crossval(modelLDA1,'CVPartition',cv);
lossNET1(i) = kfoldLoss(modelNET1);
accNET1(i) = 1 - lossNET1(i);
% cvmodelLDA2= crossval(modelLDA2,'CVPartition',cv);
lossNET2(i) = kfoldLoss(modelNET2);
accNET2(i) = 1 - lossNET2(i);
end
1 comentario
Drew
el 12 de Sept. de 2024
The differences observed above have nothing to do with using a custom partition, but are rather due to differences in initialization of layer weights. When building and validating models using any cvpartition (custom or not), remember that some model training algorithms have randomness in the model training process. For example, fitcnet includes random initialization of weights, as seen in the parameter LayerWeightsInitializer. Therefore, to get identical validation accuracy, it is necessary to set both the cvpartition and the random seed before training. For an example of getting matching predictions (and loss), see the answer that I posted to this question.
Drew
el 12 de Sept. de 2024
Custom partitions were introduced in R2023b. See https://www.mathworks.com/help/stats/cvpartition.html, and in particular the new cvpartition creation syntax 'c=cvpartition("CustomPartition",testSets)' . With this functionality, users can create cvparition objects that meet their partition requirements, such as for keeping certain observations grouped together because they come from the same underlying physical sample.
When building and validating models using any cvpartition (custom or not), remember that some model training algorithms have randomness in the model training process. For example, fitcnet includes random initialization of weights, as seen in the parameter LayerWeightsInitializer. Therefore, to get identical validation accuracy, it is necessary to set both the cvpartition and the random seed before training. This is ilustrated in the code below. When the random seed is set before training, the predictions are identical.
If this answer helps you, please remember to accept the answer.
% Load the data
t = readtable("fisheriris.csv");
% Two trials.
% In the first, do not reset random seed
% In the second, reset the random seed before each model training.
for j=1:2
% Create a 5-fold cross-validation partition
cv1 = cvpartition(t.Species, 'KFold', 5);
% Take the testsets from the above and use them to make a custom
% partition.
testsets = test(cv1,'all');
cv = cvpartition('CustomPartition',testsets);
% Set random number generator seed on trial 2 before model training
if (j==2) rng(0); end
% Train the first model with cross-validation. This builds a set of k
% cross-validation models.
net1 = fitcnet(t, 'Species', 'CVPartition', cv);
% Calculate validation error rate for the first model
errorRate1 = kfoldLoss(net1);
% Get predictions for the first model
predictions1 = kfoldPredict(net1);
% Set random number generator seed on trial 2 before model training
if (j==2) rng(0); end
% Train the second model with cross-validation. This builds a set of k
% cross-validation models.
net2 = fitcnet(t, 'Species', 'CVPartition', cv);
% Calculate validation error rate for the second model
errorRate2 = kfoldLoss(net2);
% Get predictions for the second model
predictions2 = kfoldPredict(net2);
% Compare predictions
samePredictions = strcmp(predictions1, predictions2);
differentIdx = find(~samePredictions);
% Report results
if (j==1)
fprintf('\nWithout resetting random seed: trial %d \n',j);
else
fprintf('\nWith resetting random seed: trial %d\n',j);
end
fprintf('Validation error rate for Model 1: %.2f%%\n', errorRate1 * 100);
fprintf('Validation error rate for Model 2: %.2f%%\n', errorRate2 * 100);
fprintf('Same predictions: %d\n', sum(samePredictions));
fprintf('Different predictions: %d\n', length(differentIdx));
if ~isempty(differentIdx)
fprintf('Differences at indices:\n');
for i = 1:length(differentIdx)
fprintf(' Index %d: Model1 = %s, Model2 = %s\n', ...
differentIdx(i), predictions1{differentIdx(i)}, predictions2{differentIdx(i)});
end
end
end
0 comentarios
Ver también
Categorías
Más información sobre Classification Ensembles en Help Center y File Exchange.
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!