Neural networks have demonstrated remarkable success across various applications. However, the theoretical understanding of their properties has lagged behind the empirical evidence accumulated in recent years. In this talk, we investigate the potential of numerical methods for ordinary differential equations (ODEs) and partial differential equations (PDEs) to enhance our comprehension of these parametric models.