This interactive demo illustrates an adaptive model hierarchy for parametrized optimal control problems that is described in more detail in Application of an adaptive model hierarchy to parametrized optimal control problems. In this blog post we describe the background of the model hierarchy and the involved models. In particular, we show how machine learning can interact with traditional approaches and accelerate computations while ensuring certified accuracy.
In many applications, dynamical systems with a control input are of interest. Such systems arise for instance in engineering where mechanical systems might be steered into a desired state by applying an external control force. Another example could be the navigation of vehicles or even the control of space shuttles. In the field of optimal control, the goal is typically to determine a control policy that balances the deviation of the system from its target state and the costs of applying the control (for instance the required energy to change the system's state).
As an additional complexity of many optimal control problems, certain parameters of the underlying dynamical system might be unknown or can change depending on the considered scenario. Hence, the solution of the optimal control problem changes with the parameter. In many cases, for instance when investigating uncertainty of the system or trying to find an optimal value for the parameter, solutions of the optimal control problem for lots of different parameters have to be computed. This can result in severe computational issues when a single solution of the optimal control problem might already take minutes. Solving for hundreds or even thousands of parameters would be infeasible in that case.
The optimal control problems considered here consist of a linear control system and a quadratic cost measuring the deviation from the target and the control costs. The state of the dynamical system lives in a high-dimensional space and changes over time depending on the applied control. Due to the special structure of the optimal control problem, when fixing the parameter, the solution boils down to solving a linear system. Eventually, a conjugate gradient solver is used to iteratively solve the arising high-dimensional linear system for a new parameter. The model associated with the original high-dimensional optimal control problem is referred to as the full-order model (FOM). However, as described before, the solution of the FOM depends heavily on the parameter and solving high-dimensional linear systems for several hundreds of different parameters is prohibitively costly. This issue can be addressed with reduced order models and machine learning surrogates that are described in more detail below. In this particular blog post, we focus on combining surrogate models with the FOM in an adaptive hierarchy that is applicable when (approximate) solutions for many different parameter values are required.
In order to accelerate the solution of otherwise costly parametric optimal control problems, we discuss in the following an adaptive model hierarchy that replaces the high-fidelity full-order model by faster surrogates whenever possible. The hierarchy consists of three models (the surrogate models are described in more detail below) of decreasing cost and accuracy:
When queried for a parameter $\mu\in\mathcal{P}$ coming from a set of parameters $\mathcal{P}$, the hierarchy first tries the fastest model (ML-ROM). If the a posteriori error estimate exceeds the prescribed tolerance $\varepsilon>0$, it falls back to the RB-ROM, and finally to the FOM. Every FOM solve enriches the reduced basis; every RB-ROM solve generates new training data for the ML-ROM. The models are built adaptively — no offline phase is required. The adaptive model hierarchy has been introduced originally in A new certified hierarchical and adaptive RB-ML-ROM surrogate model for parametrized PDEs and we provide a more detailed description of the functionality of the hierarchy below. First of all, the following figure gives a good overview of the involved models and their relationships.
The following flowchart describes in more detail how the adaptive model hierarchy processes an incoming parameter $\mu$. The parameter is first passed to the ML-ROM (the cheapest and least accurate in the hierarchy) and the ML-ROM solution is computed. The error estimator of the ML-ROM is evaluated and the output compared to the prescribed tolerance $\varepsilon$. If the estimated error is below the tolerance, the ML-ROM solution is returned and none of the two other models has to be touched for the current parameter. If the estimated error is too large, the next model in the hierarchy — the RB-ROM — is called next. The RB-ROM is solved and its estimated error compared to the tolerance. If the RB-ROM solution is accurate enough, it is returned and at the same time used for improving the ML-ROM. If the estimated error of the RB-ROM exceeds the tolerance, the FOM is called. The solution of the FOM is returned (there is no accuracy check for the FOM required) and used to improve the RB-ROM and consequently also the ML-ROM. It is important to note that the error estimation ensures the accuracy of the solution delivered by the model hierarchy. In any case, the error of the model hierarchy's output is smaller than the given tolerance. This also holds for the machine learning predictions where reliable error certification is usually difficult. At the same time, computations are handled by the faster models whenever possible and only when really necessary, more costly models are solved. Even when models later in the hierarchy have to be called, this still generates training data for the other models in the hierarchy, this way improving their performance as well.
The following plot shows how a typical sequence of queries of the adaptive model hierarchy compares to querying the FOM or a general ROM in terms of computational costs. Typically, ROMs are built within an expensive offline phase and applied afterwards during the so-called online phase. The costs for evaluating the ROM during the online phase are usually much smaller than evaluating the FOM. The computational costs are therefore split into a costly offline phase and cheap evaluations online for unseen parameters. In contrast, the FOM does not have an offline phase but every solve of the FOM is costly on its own. The adaptive hierarchy also does not require an offline phase. However, some of the queries might lead to a higher computational effort due to the evaluation of more costly models in the hierarchy and potential updates or training of the cheaper models. The plot below shows an example for the computational time with respect to the number of queried parameters for the three scenarios discussed here.
Having seen the overall hierarchy, let us now describe the individual surrogate models in more detail.
Before moving to the machine learning surrogate, we briefly summarize the main ideas of reduced basis reduced order models. The particular reduced model applied in the interactive demo was introduced in Greedy controllability of finite dimensional linear systems. Here, we describe projection-based reduced models for parametric problems in a more general manner:
Consider a problem with a parameter-dependency in the underlying system, such that also the solutions $u(\mu)\in V$ vary with the parameter $\mu\in\mathcal{P}$. Assume further that the solutions live in a very high-dimensional space $V$. In many cases, although the solutions are parameter-dependent, they do not change arbitrary but depend smoothly in some way on the parameter. The smoothness of the dependency can be exploited by extracting a low-dimensional subspace $V_N$ that covers the main variety of the solutions. Therefore, instead of solving in the original high-dimensional space $V$, we restrict ourselves to a suitable low-dimensional subspace and solve within that subspace. This leads to an approximate solution $u_N(\mu)\in V_N$ which can be represented as a linear combination of a basis $v_1,\ldots,v_N\in V_N$ of the subspace $V_N$ — the so-called reduced basis. The reduced solution is therefore of the form $$u_N(\mu)=\sum_{i=1}^{N} \alpha_i(\mu)\cdot v_i,$$ where $\alpha_i(\mu)\in\mathbb{R}$ is the $i$-th coefficient depending on the parameter $\mu$. Solving in the reduced space is, when done properly, much faster than solving the original problem since only the reduced coefficients $\alpha_1,\ldots,\alpha_N$ have to be computed. The major computational effort is required when constructing the reduced basis itself. This is usually done based on some solutions of the high-dimensional problem, i.e. $u(\mu_1),\ldots,u(\mu_k)\in V$ for some parameters $\mu_1,\ldots,\mu_k\in\mathcal{P}$. Within the adaptive model hierarchy, high-dimensional solution data is available whenever the full-order model is called. We further note that the reduced model used here allows for an a posteriori error estimator that can be evaluated efficiently. Given a new parameter, we just need to solve in the subspace spanned by the reduced basis and, based on the approximate solution, compute an upper bound for the error with respect to the high-dimensional solution. This error estimate does in particular not require the high-dimensional solution itself and can be computed with a similar computational effort as solving in the subspace.
The ML-ROM approximates the map from parameters to reduced coefficients using kernel interpolation. It therefore avoids solving for the coefficients and replaces this step by a simple evaluation of the kernel surrogate. Interestingly, the a posteriori error estimator of the reduced basis ROM is still applicable and provides an upper bound for the error of the ML-ROM. The approach for constructing the ML-ROM is outlined in detail in Be greedy and learn: efficient and certified algorithms for parametrized optimal control problems. Before taking a closer look at the interactive demo, we briefly introduce kernel interpolation, which is presented by another interactive plot where a kernel surrogate for a given function can be constructed.
A kernel interpolant is a weighted sum of kernel functions centered at training points: $$s(\mu) = \sum_{j=1}^{M} w_j \cdot k(\mu, \mu_j),$$ where $k$ is a kernel function (for instance a Gaussian kernel $k(\mu, \mu') = \exp(-\gamma \|\mu - \mu'\|^2)$), $w_1,\ldots,w_M\in\mathbb{R}$ are the weights and $\mu_1,\ldots,\mu_M$ are the training points (also called centers). In its simplest form, the weights are chosen such that an interpolation property of the form $$s(\mu_j) = \alpha(\mu_j)\qquad\text{for }j=1,\ldots,M$$ holds. Here, $\alpha(\mu_j)=[\alpha_1(\mu_j),\ldots,\alpha_N(\mu_j)]\in\mathbb{R}^N$ is a vector containing the reduced coefficients associated with the parameter $\mu_j\in\mathcal{P}$. The kernel interpolant approximates vector-valued data in this case. Computing the weights amounts to solving a linear system of equations with dimension similar to the number of training data points. The kernel interpolant is thus constructed as a linear combination of kernel functions such that it interpolates the given data at the training points. Click on the plot below to add training points and see how the interpolation of a given function is built from individual Gaussian kernels centered at the training points.
Within the adaptive model hierarchy, the kernel surrogate is constructed adaptively using the available data provided by evaluations of the RB-ROM. The ML-ROM therefore interacts with and is built on the RB-ROM and makes further use of its a posteriori error estimator. This connection illustrates how machine learning methods can benefit from an integration with classical methods developed in different areas of applied mathematics.
For the ML-ROM, we are actually not restricted to kernel methods. Any machine learning method capable of approximating vector-valued input-output data in a supervised learning setting can be used. In particular the same a posteriori error estimator is still applicable and guarantees a sufficiently accurate approximation of the optimal control. The approach has for instance been tested with deep neural networks and Gaussian process regression as well.
We are now set to show the interactive demo for the hierarchy including the models presented above. To this end, we consider a parametric optimal control problem that is briefly described in the following.
The test problem is a parametrized 1D heat equation on the spatial domain $\Omega = [0,1]$ with two boundary controls,
where $\mu_1 \in [1,2]$ is the diffusivity and $\mu_2 \in [0.5, 1.5]$ determines the slope of the target state.
The demo below allows to query the hierarchy either by selecting specific parameters using the sliders,
by clicking inside of the parameter space box or by drawing random samples.
It is further possible to query the hierarchy for a whole batch of random parameters.
The results obtained by the adaptive model hierarchy are shown in the table below and the respective plots.
As before, the different colors refer to the three models involved here.
Besides the selected parameters, there are plots showing the estimated errors and the required time to process each query.
The last plot displays an animation of the solution computed for the selected parameter.
Feel free to try the hierarchy on your own with this interactive demo!
| Total queries: | 0 | |
| FOM calls: | 0 | avg - ms |
| RB-ROM calls: | 0 | avg - ms |
| ML-ROM calls: | 0 | avg - ms |
| Basis size: | 0 | |
| ML training samples: | 0 | |
| RB basis extension: | - | |
| ML training: | - | |
In this blog post we introduced an adaptive model hierarchy for solving parametrized optimal control problems. The hierarchy provides approximate optimal controls with certified accuracy based on the interaction of a full-order model, a reduced basis reduced order model and a machine learning surrogate. The surrogate models are created adaptively using information from more costly but also more accurate models in the hierarchy. Whenever possible, the faster surrogate models are used, leveraging the computational advantages of the machine learning surrogate and the reduced order model.
University of Graz, Leechgasse 34, 8010 Graz, Austria