Surface Fitting using Neural Networks

Solve N-dimensional surface fitting problems with extremely high accuracy.

Neural Networks for Nonlinear Regression & Classification

Neural networks are universal function approximator, which means that given enough parameters, a neural net can approximate any multivariable continuous function to any desired level of accuracy.

Arbitrary dimension & high-precision function approximation. ( )

This package aims to solve problems related to scientific computing where the tolerance for approximation error is small. A second-order optimization solver is used, allowing the network to achieve significantly higher accuracy than commonly used first-order solvers.

NN = NeuralFit(x, y, [N, M]); % N: input dimension , M: output dimension.

Note

• Fit data using "NeuralFit", where N& Mare input & output dimension respectively.
• "x" must be a matrix and "y" is a matrix where D is number of data.
• "NeuralFit" use default setting, to adjust the architecture of neural networks or optimization solvers, please refer to "GeneralGuide.mlx".

Example

x = linspace(-2, 2, 20); y = x;
[X, Y] = meshgrid(x, y); U = X.^2 + Y.^2; Z = exp(-0.5*U).*cos(2*U);
data = [X(:), Y(:)].'; label = Z(:).';

NN = NeuralFit(data, label, [2, 1]); 
Prediction = NN.Evaluate(data);
PerformanceMetric = NN.Report;

figure();
surf(X, Y, Z); hold on; scatter3(data(1, :), data(2, :), label)

Application

Approximate highly nonlinear function Smoothing out data and estimate derivative

Handwritten digit recognition Find pattern form noisy data

Basic Network & Solver Set Up

data = linspace(0,2*pi,1000); label = (data).*sin(data) + cos(3*data);
LayerStruct = [1, 7, 7, 7, 1];
NN = Initialization(LayerStruct);
option.MaxIteration=600;
NN = OptimizationSolver(data, label, NN, option);
Prediction = NN.Evaluate(data);

Standard Template

• Two standard templates available for users to quickly call the main functions: "SimplifiedWorkflow.m" and "CustomizableWorkflow.m" The purpose of "SimplifiedWorkflow.m" is to assist beginners in quickly getting started, while the other template provides more flexibility.

Instruction and Example

• For detailed instructions on how to use the package, please refer to "GeneralGuide.mlx".
• "DigitRecognition.mlx" use a simple MLP architecture and achieves an accuracy of 97.6% on the testing set of the "MNIST" handwritten digit recognition dataset.
• "CurveFittingFromNoisyData.mlx" demonstrates how to use neural nets to fit noisy data and estimate derivatives.
• "CustomizableWorkflow.m" provide standard workflow for following multivariable function approximation.
• "MathModel.mlx" explains the mathematical model of neural nets and provides a step-by-step numerical example that may help users understand neural nets more easily.

Customizing the model and solver parameters.

• For detailed instructions, please refer to "GeneralGuide.mlx" in example page.

NN.Cost = 'MSE'; % specify cost function.
NN.InputAutoScaling = 'on'; % normalize the data automatically.
NN.ActivationFunction = 'gaussian'; % specify nonlinear activation.
LayerStruct = [InputDimension, 10, 10, 10, OutputDimension]; % define the size of network.
NN = Initialization(LayerStruct,NN); % initialize weights of neural net.
%% First-stage optimization
option.MaxIteration = 100;
option.Solver = 'ADAM';
option.BatchSize = 500;
NN = OptimizationSolver(data, label, NN, option);
%% Second-stage optimization
option.MaxIteration = 400;
option.Solver = 'BFGS';
NN = OptimizationSolver(data, label, NN, option);                                                                                                                    
Report = FittingReport(data, label, NN); % Quantify fitting performance.

Tips and considerations for training neural networks

• Please refer to the "Tips for Training Neural Networks.mlx" which provides detailed yet straightforward instructions to easily address the mentioned issues.
• Ensure that the data (i.e., input x) is distributed in similar magnitude. otherwise, it can make neural network training challenging. Therefore, it is recommended that users always preprocess their data (i.e., perform normalization) before starting the optimization process. If you are unfamiliar with preprocessing methods, the package also provides basic algorithms that should be sufficient for most situations.
• Make sure the standard deviation of the labels is not too small, as it can also make it difficult to train the neural network. The package includes built-in functions to handle this situation.

Optimization Solvers

1. Stochastic Gradient Descents (SGD)
2. Stochastic Gradient Descents with Momentum (SGDM)
3. Root Mean Square Propagation (RMSprop)
4. Adaptive Momentum Estimation (ADAM)
5. Adaptive Momentum Estimation with Weight decay (AdamW)
6. Broyden-Fletcher-Goldfarb-Shanno Method (BFGS)

Mathematical model of neural nets

• W,b are the weights matrices and bias vectors of neural nets.
• d is the depth of neural nets.
• is a point-wise non-linear function, such as tanh.
• For more detail, please refer to "MathModel.mlx".

Reference

1. Numerical Optimization, Nocedal & Wright.
2. Practical Quasi-Newton Methods for Training Deep Neural Networks, Goldfarb, et al.
3. Kronecker-factored Quasi-Newton Methods for Deep Learning, Yi Ren, et al.