Special Session 133: New developments on nonlinear expectations

When Reinforcement Learning Aligns with Estimate-Then-Plug-In? Insights from Continuous-Time Portfolio Selection

Min Dai The Hong Kong Polytechnic University Hong Kong Co-Author(s):    Min Dai, Yanwei Jia, Zhichao Lu

Abstract: Traditional dynamic decision-making under uncertainty typically follows an estimate-then-plug-in approach: specifying a model, estimating its parameters from historical data, and solving the resulting stochastic control problem analytically or numerically. In contrast, modern reinforcement learning (RL) directly learns optimal policies from data without estimating model parameters. Continuous-time RL methods have recently gained traction in financial decision-making, but it remains unclear when RL outperforms the traditional approach and why. We investigate these questions through continuous-time portfolio optimization using a relaxed control framework. Without transaction costs, we derive an analytical solution and develop a model-free q-learning algorithm to learn optimal trading strategies. Our theoretical results show that given continuous stock price data, the RL-learned policy matches the estimate-then-plug-in solution. Incorporating transaction costs, we find this equivalence still holds. Numerical experiments with synthetic data confirm these findings across various models. However, empirical studies demonstrate that our RL method significantly outperforms the traditional approach in realistic settings, likely because real-world stock price dynamics exhibit complex features not fully captured by parametric models.

A Two-fold Randomization Framework for Impulse Control Problems

Yuchao Dong Tongji University Peoples Rep of China Co-Author(s):

Abstract: We propose and analyze a randomization scheme for a general class of impulse control problems. The solution to this randomized problem is characterized as the fixed point of a compound operator which consists of a regularized nonlocal operator and a regularized stopping operator. This approach allows us to derive a semi-linear Hamilton-Jacobi-Bellman (HJB) equation. Through an equivalent randomization scheme with a compound Poisson measure, we establish a verification theorem that implies the uniqueness of the solution. Via an iterative approach, we prove the existence of the solution. The existence--and--uniqueness result ensures the randomized problem is well-defined. We then demonstrate that our randomized impulse control problem converges to its classical counterpart as the randomization parameter \(\blambda\) vanishes. This convergence, combined with the value function's \(\mathcal{C}^{2,\alpha}_{loc}\) regularity, confirms our framework provides a robust approximation and a foundation for developing learning algorithms. Under this framework, we propose an offline reinforcement learning (RL) algorithm. Its policy improvement step is naturally derived from the iterative approach from the existence proof, which enjoys a geometric convergence rate. We numerically demonstrate the effectiveness of our algorithm using a widely-studied example. The results show that our RL algorithm can learn the randomized solution, which accurately approximates its classical counterpart. A sensitivity analysis with respect to the volatility parameter \(\sigma\) in the state process effectively demonstrates the exploration--exploitation tradeoff.

Boundary stabilization of 1D hyperbolic balance laws

Long Hu Shandong University Peoples Rep of China Co-Author(s):

Abstract: One-dimensional hyperbolic balance laws arise in many physical models, such as traffic flow, shallow water waves, gas pipelines, and oil drilling. In this talk, we will focus on the stabilization of these systems by means of boundary feedback control. We will show how to design nonlocal feedback controllers to rapidly stabilize such systems (linear or quasilinear, autonomous or non-autonomous, with local or nonlocal source terms...). In particular, for the linear system, we will use two-step scheme to stabilize the system in optimal finite-time using either one-sided or two-sided control inputs.

Doubly Reflected Backward SDEs Driven by G-Brownian Motion with Quadratic Generator

Hanwu Li Shandong University Peoples Rep of China Co-Author(s):    Hanwu Li, Peng Luo and Mengbo Zhu

Abstract: In this talk, we study the doubly reflected backward stochastic differential equations driven by G-Brownian motion (G-BSDEs for short) when the generator has quadratic growth in the z-component. Based on the theory of G-BMO martingale and G-Girsanov theorem, we establish the existence and uniqueness result when the upper obstacle is almost a generalized G-It\^{o}`s process. Moreover, the solution can be approximated monotonically by the solutions to a family of penalized reflected G-BSDEs with a lower obstacle, which plays an important role to establish the relation between doubly reflected G-BSDEs and fully nonlinear partial differential equations with double obstacles.

Mean field portfolio games with major-minor agents and random horizon

Wenqiang Li Shandong University Peoples Rep of China Co-Author(s):

Abstract: We study a type of mean field portfolio games with a major and N minor agents with private information, where the major agent is not allowed to trade on the market after a given unpredictable and external default time to the market. The investment horizon of each minor agent is divided into two intervals with and without the major in the market, leading naturally to two sequential optimization problems. Using the martingale optimality principle, a Nash equilibrium of the limiting problem ($N\rightarrow\infty$) is characterized by a type of coupled multi-dimensional mean-field FBSDEs with quadratic growth and random horizon, which is solvable under a weak interaction assumption of minor agents. The convergency of Nash equilibrium between the limiting problem and the original (N+1) portfolio games is also verified. We study the well-posedness and the stability results of a extended BSDEs with quadratic growth and random horizon in the Appendix.

Anticipated backward stochastic evolution equations and maximum principle for path-dependent systems in infinite dimensions

Guomin Liu Nankai University Peoples Rep of China Co-Author(s):    Meng Wang, Jian Song

Abstract: In this paper, we study the Pontryagin`s maximum principle for the stochastic recursive optimal control problem of a class of path-dependent stochastic evolution equations. In the control system, the state process depends on its past trajectory, the control is delayed via an integral with respect to a general finite measure, and the final cost relies on the delayed state. To obtain the maximum principle, we introduce a functional adjoint operator for the non-anticipative path derivative and establish the well-posedness of an anticipated backward stochastic evolution equation in the path-dependent form, which serves as the adjoint equation.

Quadratic forward backward stochastic differential equations driven by G-Brownian motion

Peng Luo Shanghai Jiao Tong University Peoples Rep of China Co-Author(s):

Abstract: We consider quadratic forward backward stochastic differential equations driven by G-Brownian motion. Based on the theory of G-BMO martingale and G-Girsanov theorem, we establish the existence and uniqueness of solution on small time duration. We further provide some extensions.

A control method for solving high-dimensional fully coupled FBSDEs via deep learning

Ying Peng Shandong University Peoples Rep of China Co-Author(s):    Shaolin Ji, Shige Peng, Xichuan Zhang

Abstract: In this talk, we present a control method for solving a high-dimensional stochastic Hamiltonian system with boundary conditions, which is essentially a Forward Backward Stochastic Differential Equation (FBSDE in short). Different from existing methods, we first formulate a stochastic optimal control problem whose extended Hamiltonian system is exactly the system to be solved. Then two different algorithms to calculate the stochastic optimal control via deep neural networks are designed respectively. Comparing with the Deep FBSDE method, our proposed algorithms demonstrate more stable performance.

On Infinite-Time Mean Field Games and the Associated Elliptic Master Equations

Yongsheng Song Academy of Mathematics and Systems Science, Chinese Academy of Sciences Peoples Rep of China Co-Author(s):

Abstract: We study discounted infinite-time mean field games, for which a Nash equilibrium is constructed using the stochastic maximum principle and infinite-time McKean-Vlasov forward-backward stochastic differential equations. We then prove that, at the Nash equilibrium, the value function of the representative player provides a viscosity solution to the corresponding elliptic master equation. Furthermore, we establish the uniqueness of classical solutions to the elliptic master equation under displacement monotonicity and certain growth assumptions. This talk is based on a joint paper with Zeyu Yang.

Comparison theorems for multi-dimensional BSDEs with jumps and applications to stochastic LQ control

Zuoquan Xu The Hong Kong Polytechnic University Peoples Rep of China Co-Author(s):    Ying Hu; Xiaomin Shi; Zuo Quan Xu

Abstract: In this paper, we, for the first time, establish two comparison theorems for multi-dimensional backward stochastic differential equations with jumps. Our approach is novel and completely different from the existing ones for one-dimensional case. Using these and other delicate tools, we then construct solutions to coupled two-dimensional stochastic Riccati equation with jumps in both standard and singular cases. In the end, these results are applied to solve a cone-constrained stochastic linear-quadratic control problem and a mean-variance portfolio selection problem with jumps. Different from no jump problems, the optimal (relative) state processes may change their signs, which is of course due to the presence of jumps. This is a joint work with Ying Hu (Univ Rennes) and Xiaomin Shi (Shandong University of Finance and Economics).

Large Language Models training under Sublinear Expectation

Shuzhen Yang Shandong University Peoples Rep of China Co-Author(s):

Abstract: As the complexity of large language models (LLMs) continues to grow, the scale of training data expands exponentially, and the inherent uncertainty in data distribution has emerged as a nonnegligible critical challenge. We incorporate the sublinear expectation theory into LLMs to analyze uncertainties inherent in training data. Leveraging the $\phi$-max mean algorithm, we propose a novel cross-entropy loss function (Lossmax mean) and the corresponding training strategy (SLE-Strategy). Using a decoder-only Transformer as the base model, we conduct five sets of controlled experiments with different parameter sizes, comparing the model using SLE-Strategy (SLE-model) with model using traditional strategy. Experimental results demonstrate that the SLE-model achieves a substantial improvement in training efficiency, reducing the per-epoch training time by 57.46%-86.34% while maintaining a controllable performance gap. This innovative approach effectively balances cost and model performance, offering a promising direction for optimizing LLM training in resource-constrained scenarios.