Plot PCA data in 3D while maintaining grouping from k nearest neighbor classifation

Trying to make a 3D PCA scatter plot while grouping the data using k nearest neighbor classification. I need each axis to be labeled PCA 1, PCA 2, PCA 3 and each group to be labeled with a different colored dot!

I have the 2D plot example below:

% For KNN classification: class=knnclassify(TestingMatrix,SCORE,group,2,'euclidean','nearest');

%% Figures % Plots 2D PCA plots of the first 3 principal components SCOREX=TestingMatrix(:,1); SCOREY=TestingMatrix(:,2); SCOREZ=TestingMatrix(:,3);

figure; gscatter(SCOREX,SCOREY,class) xlabel('PCA 1'); ylabel('PCA 2')

figure; gscatter(SCOREX,SCOREZ,class) xlabel('PCA 1'); ylabel('PCA 3')

figure; gscatter(SCOREY,SCOREZ,class) xlabel('PCA 2'); ylabel('PCA 3')

[Sachin Meena]

what all have you tried?

Answer 1

[Purvaja]

Hi @Rachel Parise,

I understand that you want to plot principal components on 3D plot that are corresponding to the clusters that you achieved after performing “KNN classification” on your dimensionally reduced dataset. I have approached this in following steps:

1. Perform PCA on dataset and extract principal component vectors using “Coeff” returned by “PCA”
2. Predicting class labels after training using KNN classification
3. Plot the PCA transformed data on 3D graph, colour each based on their class.
4. The PCA lines (principal component vectors) will originate from the mean of the dataset in PCA space.
5. The PCA vectors (Coeff) have unit length, so they need to be scaled to make them visible.
6. I used sqrt (explained (1:3)) (square root of explained variance) to scale the lines based on data spread, where explained is returned by “PCA” function.

You can refer the following code for the above steps:

Note: Since “knnclassify” is deprecated, I have used “fitcknn” function instead for KNN classification. Also have assumed some dummy data of 100 samples having 5 dimensions using “randn(100,5)” function.

% Perform PCA 
[coeff, SCORE, ~, ~, explained] = pca(data); 

% Select first 3 principal components 
TestingMatrix = SCORE(:,1:3); 

% Train k-NN classifier (K=2) 
knnModel = fitcknn(TestingMatrix, group, 'NumNeighbors', 2, 'Distance', 'euclidean'); 

% Predict class labels using the trained k-NN model 
class = predict(knnModel, TestingMatrix); 

% 3D PCA Scatter Plot 
figure; 
scatter3(TestingMatrix(:,1), TestingMatrix(:,2), TestingMatrix(:,3), 50, class, 'filled'); 
xlabel('PCA 1'); ylabel('PCA 2'); zlabel('PCA 3'); 
title('3D PCA Scatter Plot with PCA Vectors & KNN Classification'); 
grid on;  
view(3); 
colorbar; 
hold on; 

% Plot PCA lines (principal component vectors) 
meanVals = mean(TestingMatrix, 1);  
scaleFactor = 3 * sqrt(explained(1:3))';  
labels = {'PCA 1', 'PCA 2', 'PCA 3'};  
colors = ['r', 'g', 'b'];  

for i = 1:3 
    % Plot PCA vector as an arrow 
    quiver3(meanVals(1), meanVals(2), meanVals(3), ... 
    coeff(1,i)*scaleFactor(i), coeff(2,i)*scaleFactor(i), coeff(3,i)*scaleFactor(i), ... 
    'LineWidth', 2, 'Color', colors(i), 'MaxHeadSize', 1); 
    
    % Label the PCA vector 
    text(meanVals(1) + coeff(1,i)*scaleFactor(i), ... 
    meanVals(2) + coeff(2,i)*scaleFactor(i), ... 
    meanVals(3) + coeff(3,i)*scaleFactor(i), ... 
    labels{i}, 'FontSize', 12, 'FontWeight', 'bold', 'Color', colors(i)); 
end

hold off; 

You can visualise the PCA components like this after implementation:

For more clarification on the functions used, you can refer to the following resources,

1. PCA: https://www.mathworks.com/help/stats/pca.html
2. Fitcknn : https://www.mathworks.com/help/stats/fitcknn.html
3. Quiver3: https://www.mathworks.com/help/matlab/ref/quiver3.html

Or you can access release specific documentation using these commands in your MATLAB command window respectively:

web(fullfile(docroot, 'stats/pca.html')) 
web(fullfile(docroot, 'stats/fitcknn.html')) 
web(fullfile(docroot, 'matlab/ref/quiver3.html')) 

Hope this helps you!