Solving the Burgers Equations with Echo state networks
30 visualizaciones (últimos 30 días)
Mike el 1 de Sept. de 2022
found this example, which is very interesting for my work.
Can someone give me a hint, how to solve the PDE with an ESN, so change from ordinary neuronal network to reservoir?
Solve Partial Differential Equations Using Deep Learning - MATLAB & Simulink - MathWorks Deutschland
Any help is welcome.
Más respuestas (4)
David Willingham el 1 de Sept. de 2022
Are you able to ellaborate a little more about the problem you're looking to solve? We currently don't have an example for this, so I'm looking to see what might be the best material / support I can give you.
David Willingham el 9 de Sept. de 2022
I received an update from our development team on this (attached). Whilst not fully worked through, it should serve as a starting point.
The following is a brief description:
Solving PDEs with Echo State Networks
This repo demonstrates how to train an echo-state network to solve a PDE.
It follows this example.
Echo State Networks
An echo-state network is a discrete time recurrent model. Given a sequence x(t) the model computes a reservoir sequence z(t+1) = tanh(U*z(t)+V*x(t)). Then the model output is y(t) = W*z(t). Here the U,V,W are randomly initialized matrices. During training only the W matrix (the output matrix) is trained. This speeds up typical deep learning model training as there are far fewer parameters, and in fact the model z(t) -> y(t) is merely a linear model which can be fit with least squares or ridge regression in typical cases.
To solve PDEs with deep learning models we simply add a "soft constraint" into the model loss which is some norm of the PDE residual. This method is typically called Physics-Informed Neural Networks (PINNs).
Technical Details of using ESN to solve PDE
- Since the loss involves the PDE term it is not possible to optimize W with simple methods like least squares. This means we need to use gradient-based methods, which lose some of the efficiency gains of typical ESN workflows. In particular we need to re-compute the reservoir dynamics on every iteration of the gradient-based optimization as we need the autodiff system to trace through the reservoir so that it can compute the partial derivatives that define the PDE.
- Since ESN is a discrete time model we have to modify this example to be discrete time. In particular this means we have to estimate the partial derivative in time via finite differences.
Future or ToDo
- This example does not currently use any of the common and important techniques necessary to train an ESN well. For example see A Practical Guide to Appling Echo State Networks
- This example does not include loss terms to encourage the model to satisfy an initial condition or boundary condition as in the original example. This is simply for brevity, those terms can be added.
- The example splits the training data x(t) into subsequences x(1:N),x(N+1:N),.... In principle the N could be increased during training, and this may help fine-tune the model to be more accurate for longer time.
- This example does not use LBFGS to optimize the weights as in this example. It may be important to use LBFGS in PINN workflows according to literature.
- It may be possible to use the idea of CTESN to extend this to a continuous time model.