Homepage

Tensor-based Optimal Control Approaches for Deep Learning - A brief intro

Machine learning builds models from data, which are then incorporated into decision-making algorithms. The awareness about the environmental impact of machine learning, specifically the learning phase, has recently increased. The carbon footprint of learning one large, but common, machine learning model is estimated to equal the CO2 emissions produced by five cars over their whole lifetimes. The road to sustainable machine learning is still long.

Deep learning is a popular machine learning framework with state-of-the-art applications in image and speech recognition, and computer vision. Learning a model from data is traditionally viewed as a regression problem. However, recently a new viewpoint emerged from applied mathematics that interprets the learning phase as an optimal control problem (OCP) constrained by differential equations. This formulation opens the door to more advanced methods for OCPs, not previously exploited in deep learning.

TOC4Deep will exploit powerful techniques from applied mathematics to solve the optimal control form of the learning problem. In particular, we will integrate three specific improvements: First, the problem will be formulated as a sparse, block linear system to be solved with a preconditioned Krylov method. Second, we will apply sparsity-promoting regularization as a deterministic form of dropout. Third, we will apply low-rank tensor structures as an alternative bias towards simpler models.

TOC4Deep has been funded by the UNA Europa network under the funding scheme UNA Europa Seed Funding - DIGITALIZED!. In particular, this project received funds from the Polish National Agency for Academic Exchange.