Operator overloading, dynamic graph based approaches such as PyTorch and AutoGrad. Their dynamic and interactive nature lets most programs be written and reasoned about more easily. However, they lead to interpreter overhead (particularly when composing many small operations), poorer scalability, and struggle to gain benefit from compiler optimization.
Both of these early approaches are only able to differentiate code written in a suitable manner for the framework, limiting their interoperability with other programs.
A more recent package for the Julia programming language – Zygote – resolves the issues that earlier attempts faced by treating the language's syntax as the graph. The intermediate representation of arbitrary code can then be differentiated directly, optimized, and compiled.
A programming language "currently under development and is not yet ready for use" called Myia allows defining a model using a subset of Python, which is compiled to Myia.
^ abcdInnes, Mike; Edelman, Alan; Fischer, Keno; Rackauckas, Chris; Saba, Elliot; Viral B Shah; Tebbutt, Will (2019), ?P: A Differentiable Programming System to Bridge Machine Learning and Scientific Computing, arXiv:1907.07587