Efficient and robust optimization is essential for training neural networks, enabling scientific machine learning models to converge rapidly to high accuracy while faithfully capturing complex physical behavior governed by partial differential equations (PDEs). In this work, we present advanced optimization strategies to accelerate the convergence of physics-informed neural networks (PINNs) for challenging PDEs.

Specifically, we develop efficient implementations of the Natural Gradient optimizer, as well as self-scaling BFGS and Broyden quasi-Newton methods, and demonstrate their performance on benchmark problems including the Helmholtz equation, Stokes flow, the inviscid Burgers equation, and the Euler equations for high-speed flows. Beyond optimizer development, we also propose new PINN-based formulations for solving the inviscid Burgers and Euler equations, and compare the resulting solutions against high-order numerical methods to provide a rigorous and fair assessment.

Finally, we address the challenge of scaling these quasi-Newton methods for batched training, enabling efficient and scalable solutions for large-scale scientific machine learning problems.