# Research Interests

We are interested in all aspects of mathematical imaging: the use of mathematical techniques to analyse and to improve real-world images, ranging from photographs made with consumer cameras to the images made with professional imaging devices in the sciences and medicine. These include techniques such as MRI (magnetic resonance imaging) and PET (positron emission tomography). Our current research concentrates in particular on higher order PDEs for image inpainting, and discontinuity-preserving higher-order variational approaches for the recovery of sparsely sampled data. Further themes include parameter learning, with the goal of building “black box” imaging tools suitable for use by non-professionals.

## Continuum Limits in Graphical Based Machine Learning Problems

In this project we look at variational problems on random geometric graphs which utilise Laplacian regularisation, for example probit and Baysian level set [1], kriging [2], and Ginzburg-Landau models [3,4]. The key idea is to move from a discrete problem (on the graph) to a continuum problem (defined on a subset of R^d) in the data rich limit. The variational methods we study come from machine learning and the discrete-to-continuum topology is metrised in the TLp space introduced in [5]. Current research includes better understanding the semi-supervised learning problem [2], and in particular how much training data one needs for asymptotic consistency.

Reference bibliography:

[1] Dunlop, M. M., Slepčev, D., Stuart, A. M., Thorpe, M. (2018). Large Data and Zero Noise Limits of Graph-Based Semi-Supervised Learning Algorithms. To appear in Applied and Computational Harmonic Analysis. ArXiv

[2] Slepčev, D., Thorpe, M. (2019). Analysis of p-Laplacian Regularization in Semi-Supervised Learning. To appear in the SIAM Journal on Mathematical Analysis. ArXiv

[3] Thorpe, M., Theil, F. (2017). Asymptotic Analysis of the Ginzburg-Landau Functional on Point Clouds. To appear in the Proceedings of the Poyal Society of Edinburgh Section A: Mathematics. ArXiv

[4] Cristoferi, R., Thorpe, M. (2018). Large Data Limit for a Phase Transition Model with the p-Laplacian on Point Clouds. To appear in the European Journal of Applied Mathematics. ArXiv

[5] García Trillos, N., Slepčev, D. (2016). Arch. Ration. Mech. Anal., 220(1):193-241. https://doi.org/10.1007/s00205-015-0929-z

## Template-Based Image Reconstruction from Sparse Tomographic Data

In this project, we developed a variational regularisation approach for template-based image reconstruction from indirect, noisy measurements as given, for instance, in X-ray computed tomography. An image is reconstructed from such measurements by deforming a given template image. The image registration is directly incorporated in the form of a partial differential equation that models the registration as either mass- or intensity-preserving transport from the template to the unknown reconstruction. In this way, outstanding reconstructions can be computed from only a few measurements.

Reference bibliography:

[1] Lang, L. F., Neumayer, S., Öktem, O., Schönlieb, C.-B. (submitted 2018). Template-Based Image Reconstruction from Sparse Tomographic Data. Preprint https://arxiv.org/abs/1810.08596

## Anisotropic variational models and PDEs for inverse imaging problems

In this project, we introduce a new higher-order total directional variation (TDV) regulariser for inverse imaging problems by taking into account the image gradient weighted by the structural content. Theoretical and numerical details are provided for different applications: the reconstruction of noisy images and videos, the image zooming and the interpolation of scattered surface data. The idea of using directional gradients for imaging applications is also used for the generalisation of the osmosis equation, introduced by Weickert and collaborators in 2013, to its anisotropic counter-part. Anisotropic osmosis is applied to the shadow removal problem thus improving upon the isotropic apporach by avoiding the burring artefact due to the isotropic diffusion.

Reference bibliography:

[1] Parisotto, S., Lellmann, J., Masnou, S., Schönlieb, C.-B. (2018). Higher-Order Total Directional Variation. Part I: Imaging Applications. ArXiv e-print https://arxiv.org/abs/1812.05023

[2] Parisotto, S., Masnous, S., Schönlieb, C.-B. (2018). Higher-Order Total Directional Variation. Part II: Analysis. ArXiv e-print https://arxiv.org/abs/1812.05061

[3] Parisotto, S., Schönlieb, C.-B. (2018). Total Directional Variation for Video Denoising. ArXiv e-print https://arxiv.org/abs/1812.05063

[4] Parisotto, S., Calatroni, L., Caliari, M., Schönlieb, C.-B., Weickert, J. (2019). Anisotropic osmosis filtering for shadow removal in images. Inverse Problems. https://doi.org/10.1088/1361-6420/ab08d2

## Mathematics for Cultural Heritage Applications

The last 50 years have seen an impressive developement of mathematical methods for the analysis and processing of digital images, mostly for photography and biomedical imaging: the arts have been mostly overlooked in this process, apart from a few exceptional works in the last 10 years. With the rapid emergence of digitisation in the arts, however, the arts domain is becoming increasingly receptive to digital image processing methods and the importance of paying attention to this therefore increases.

In this project, we explore multiple imaging applications to Cultural Heritage conversation challenges. We develop a new multi-modal non-invasive mid-infrared technique for detecting sub-superficial defects in fresco walls; we digitally fure multi-modal data via osmosis filter, unveiling hidden features of artworks; we propose a semi-supervised workflow for the detection and inpainting of defects in damaged illuminated manuscripts form the Fitzwilliam Museum's collection in Cambridge.

Reference bibliography:

[1] Daffara, C., Parisotto, S., Ambrosini, D. (2018). Multipurpose, dual-mode imaging in the 3-5 µm range (MWIR) for artwork diagnostics: A Systematic approach. Optics and Lasers in Engineering, Vol. 104, Pag. 266-273. https://doi.org/10.1016/j.optlaseng.2017.10.006

[2] Parisotto, S., Calatroni, L., Daffara, C. (2018). Digital Cultural Heritage Imaging via Osmosis Filtering. Image and Signal Processing. https://doi.org/10.1007/978-3-319-94211-7_44

[3] Calatroni, L., d'Autume, M., Hocking, R., Panayotova, S., Parisotto, S., Ricciardi, P., Schönlieb, C.-B. (2018). Unveiling the invisible: mathematical methods for restoring and interpreting illuminated manuscripts. Heritage Science, Vol. 6. https://doi.org/10.1186/s40494-018-0216-z

## Geometric Integration Methods for Optimisation

This project is concerned with the development and analysis of optimisation schemes based on geometric numerical integration methods. Discrete gradient methods are popular numerical schemes for solving systems of ODEs, and are known for preserving structures of the continuous system such as energy dissipation/conservation. Applying discrete gradients to dissipative ODEs/PDEs yields optimisation schemes that preserve the dissipative structure. For example, we consider a derivative-free discrete gradient method for optimising nonsmooth, nonconvex problems in a blackbox setting. This method has been shown to converge to optimal points of the objective function in a general, nonsmooth setting, while retaining favourable properties of gradient flow. This blackbox optimisation framework is useful, for instance, for bilevel optimisation of regularisation parameters in image processing.

## Flow of Microtubules in the Drosophila Oocyte

The focus of this project is to characterise directionality of plus ends of microtubules in confocal microscopy images of Drosophilia embryos. This goal is particularly challenging due to the high noise level in such data, making it almost impossible to distinguish EB1 fluorescently labelled comets from randomly distributed noise. To overcome this problem, we employ recently developed methods for joint motion estimation and image reconstruction. As a result we are able to estimate motion in image sequences where other state of the art methods fail.

Reference bibliography:

[1] Drechsler, M., Lang, L. F., Dirks, H., Burger, M., Schönlieb, C.-B., Palacios, I. M. (submitted 2019). Optical flow analysis reveals that Kinesin-mediated advection impacts on the orientation of microtubules. Preprint https://doi.org/10.1101/556043

## Cellular Mechanics of Drosophila

The main goal of this project is to understand cellular processes in the developing embryo of Drosophila. We aim to identify the mechanisms and driving forces of tissue formation, tissue regrowth after incurring significant damage, and the collective motion of cells. Mathematical models of, for instance, cell membranes typically include unknown physical parameters. A well-established way to infer these from measurements is to probe tissue mechanics with the help of laser ablations in order to measure e.g. recoil velocities. In this project, we aim to utilise image analysis and inverse probems to estimate these parameters automatically and reliably from confocal microscopy images of live Drosophila embryos.

## Mathematical Challenges in Electron Tomography

Electron microscopy is a powerful tool in the physical, biological, and industrial sciences advancing areas from nanotechnology to drug discovery. Tomography is a mathematical technique used to recover full 3D information from a sequence of 2D images. One of the classical challenges here is to get the best quality reconstruction from the smallest amount of data. Some of our work has been to introduce novel and customised regularisation techniques to address such problems. Recent hardware advances have also extended this to spectral images, where pixel-wise values of the 2D images are vectors rather than greyscale. This data is also very slow to acquire so we need new methods to reconstruct from little and very noisy data. [Image from O.Nicoletti et al., Nature, 502 80-84 (2013)]

## Faster PET Reconstruction by Stochastic Optimisation

This project is concerned with the efficient reconstruction of positron emission tomography by means of stochastic optimisation. In the last decade, many mathematical tools have been developed that have the ability to enhance clinical imaging in various ways. On the forefront of this wave are non-smooth priors that allow the reconstruction of a smooth image but do not prohibit jumps across meaningful areas like organs in medical imaging. Beside this these new tools also allow the incorporation of a-prior structual knowledge about the solution at hand. However, most of this progress has not been translated into clinical practice as most modern algorithms are too demanding for the huge data sizes encountered. In the past, algorithms have been made "applicable" to clinical practices by only considering a subset if the data at a time. While for some models this leads to satisfactory results, in general this ad-hoc strategy may yield to spurious artefacts. Motivated by the success of similar techniques in machine learning, in this project we extend modern algorithms for imaging that can handle non-smooth priors in a rigorous way to the subset setting by means of "randomisation". While the algorithm and thus its iterates are random, the variances of these are low and converge quickly to the desired deterministic solution. For more information on this project, see the following publications:

[1] Ehrhardt, M. J., Markiewicz, P. J., Schönlieb, C.-B. (2018). Faster PET Reconstruction with Non-Smooth Priors by Randomization and Preconditioning, https://arxiv.org/abs/1808.07150

[2] Ehrhardt, M. J., Markiewicz, P. J., Richtárik, P., Schott, J., Chambolle, A. & Schönlieb, C.-B. (2017). Faster PET Reconstruction with a Stochastic Primal-Dual Hybrid Gradient Method. In Proceedings of SPIE (Vol. 10394, pp. 1-12). San Diego. http://doi.org/10.1117/12.2272946.

[3] Chambolle, A., Ehrhardt, M. J., Richtárik, P., & Schönlieb, C.-B. (2017). Stochastic Primal-Dual Hybrid Gradient Algorithm with Arbitrary Sampling and Imaging Applications. to appear in SIAM Journal on Optimization. http://arxiv.org/abs/1706.04957.

## Segmentation and Relaxation Methods for Integer-Constrained Problems

In image segmentation problems, we look for a partition of the image domain into regions with certain characteristics. Such problems appear in areas such as image editing (separating foreground from background, merging multiple images), medical applications (separating gray and white matter, finding structures in medical images), and biological imaging (finding cells and nuclei, detecting cancerous cells). They can be viewed as variational problems with *integer* constraints, which are generally very hard to solve. We work on new ways of approximating such problems through convex relaxation techniques, encoding prior geometrical knowledge, and solving them numerically.

## Reconstruction of Digital Elevation Maps

In order to obtain high-quality height maps of terrain data, the usually sparse measurements need to be interpolated to a dense digital elevation map. We develop new higher-order regularizers for interpolating such maps that allow high-quality reconstruction from few measurements. The high order and non-convexity of these schemes makes them analytically as well as numerically very challenging.

## Model Selection by Bilevel Optimisation

A key issue in image denoising, and in inverse problems as a whole, is the correct choice of data priors and fidelity terms. Depending on this choice, different results are obtained. Several strategies, both physical - dictated by the physics behind the acquisition process - and statistically grounded (e.g. by estimating or learning noise and structure in the data), have been considered in the literature. We consider approaches that learn the model and the parameter choice by bilevel optimisation techniques.

## Non-smooth and Higher-order Variational & PDE Regularisation

One of the most successful image processing approaches is PDEs and variational models. Given a noisy image, its processed (denoised) version is computed as a solution of a PDE or as a minimiser of a functional (variational model). Both of these processes are regularising the given image. In favourable imaging approaches this is done by eliminating high-frequency features (noise) while preserving or even enhancing low-frequency features (object boundaries, edges). This gives rise to non-smooth, nonlinear terms in the model of possibly high differential order. The total variation regularizer is a typical example in this class. Beyond image denoising, such regularization procedures are successfully applied to image deblurring, inpainting and inverse problems in imaging in general. We are interested in theoretical and numerical aspects of non-smooth and, in particular, higher-order regularisation.

## High-resolution Magnetic Resonance Imaging & Emission Tomography

The quality of images reconstructed from measurements acquired with medical imaging tools such as magnetic resonance imaging (MRI) and emission tomography (PET and SPECT) usually suffers from acquisition noise and undersampling. For still being able to reconstruct high-resolution images the solution of the respective inverse problem is equipped with non-smooth regularizers - as outlined above. In this context we are interested in PET and dynamic MRI.

## Domain Decomposition Methods

We are interested in domain decomposition methods used in image processing. The following link gives more information on our research in this area: Domain decomposition methods for TV-minimization.

## Limited-Angle Tomography

In limited-angle tomography one can not look at an object from certain angles resulting in bad reconstructions of the image. To address this we apply a specialized inpainting method to the sinogram to fill in the missing data, which greatly improves the reconstruction.

## Unusual Boundary Conditions in Image Inpainting

Most inpainting algorithms implicitly assume that image data is to be extrapolated across all boundaries of the inpainting domain. This is not always appropriate - for example, the inpainting problems arising in 3D conversion and novel viewpoint synthesis give rise to two different types of boundaries - a data boundary across which we wish to extrapolate image information, as well as a free boundary where we do not. This issue together with the performance requirements of inpainting HD content in real or near to real time necessitate the development of specialized inpainting methods.

## Analysis of Neurological Structures in High Resolution MRI

The thalamus is a deep brain structure made of gray-matter, responsible to route nearly all afferent impulses to the cortex. It is subdivided in four major nuclei (each of them having specific functions and containing additional subdivisions). Thalamic changes have been related to several diseases, including Alzheimer's disease, Parkinson's disease, multiple sclerosis and other neurodegenerative dementias. The aim of the project is to divide the thalamus in its major nuclei using high resolution, multi-contrast MRI information.

## Dictionary-based Segmentation by Graph Clustering

A male pied flycatcher is a bird characterised by a distinctive bright blaze on his forehead whose shape and brightness vary among the other birds in the group. Such properties reflect behavioural and biological attitudes of the bird in the group, so a careful and accurate detection and measurement of this blaze is essential for statistical studies. We segment the blaze by using a dictionary-method that, by modelling the image as a graph and looking in the image for some characterising features of the blaze (such as texture and brightness) by a comparison with examples provided by the user, detects the blaze accurately.

## Quantitative Analysis of LiDAR, Hyperspectral, and RGB Images for Remote Sensing

The goal of this project is detection, segmentation, and quantitative analysis of each individual tree in forest regions from aerial images. Steps include registration and alignment of data from several sources, including LiDAR, hyperspectral, as well as RGB images, and the development of new detection methods for finding individual trees in large-scale datasets.

## Segmentation in Image Guided Radiotherapy for the Treatment of Prostate Cancer

We apply image segmentation methods in detecting the rectum in treatment day CT scan (MVCT) using prior manual segmentation information in planning day CT scan (kVCT). The segmentation result will be used to help clinicians to deliver proper dose to the cancer position of the patients and also help to analysis side effects quantitatively.

# Research Projects

## Current Projects

- Leverhulme Trust project on
*Unveiling the invisible*. Duration: January 2019 - December 2021. P.I.: C.-B. Schönlieb. Co.I.: S. Bucklow, A. Launaro, S. Panayotova. - Unilever and EPSRC IAA Partnership Development Award for
*Mathematical Image Analysis and Machine Learning for Better Food Microstructures*. Duration: March 2018 - February 2021. P.I.: C.-B. Schönlieb and P. Schütz. - NPL postdoctoral fellowship grant for
*The mathematics of measurement*. Duration: March 2018 - February 2021. P.I.: A. Forbes and C.-B. Schönlieb. - Global Alliance funding for
*Mathematical and statistical theory of imaging*. Duration: January 2017 - December 2018. P.I.: C.-B. Schönlieb. - EPSRC Centre for
*Mathematical and Statistical Analysis of Multimodal Clinical Imaging*. Duration: March 2016 - February 2020. P.I.: C.-B. Schönlieb. Co-Is: J. Aston, S. Bohndiek, E. Bullmore, N. Burnet, T. Fokas, F. Gilbert, A. Hansen, S. Reichelt, J. Rudd, R. Samworth, G. Treece, G. Williams. - MSCA-RISE project
*Challenges in Preservation of Structure (CHiPS)*, EU Horizon 2020 program. Duration: January 2016 - December 2019. P.I.: Elena Celledoni (NTNU). Cambridge lead: C.-B. Schönlieb.

## Past Projects

- Leverhulme Trust project on
*Breaking the non-convexity barrier*. Duration: November 2015 - October 2018. P.I.: C.-B. Schönlieb. Co.I.: M. Benning, L. Gladden, M. Möller. - Alan Turing Institute seed funding for
*Personalised breast cancer screening*. Duration: October 2017 - April 2018 P.I.: M. van der Schaar and C.-B. Schönlieb. - LMS Undergraduate Research Bursary for
*Bilevel optimisation for learning the sampling pattern in Magnetic Resonance Tomography*, P.I.: C.-B. Schönlieb. Duration: July 2016 - August 2016. Co.I.s: M. Benning, M. Ehrhardt. - EPSRC project on Efficient computational tools for inverse imaging problems. Duration: 01 January 2015 - 31 December 2017. P.I: C.-B. Schönlieb. Co-I.: T. Valkonen.
- CCI Collaborative Fund on Assessing the conservation quality of tropical forest unmanned aerial vehicles. Duration: 01 September 2014 -- 01 September 2016. PIs: D. Coomes, J. Lindsell, C.-B. Schönlieb, T. Swinfield.
- Wellcome Trust/ University of Cambridge Senior ISSF internship for the project Development of Image Analysis Algorithms for Monitoring Forest Health from Aircraft. Duration: 01 May 2014 -- 31 March 2015. PIs: X. Cai, D. Coomes, C.-B. Schönlieb.
- LMS-Scheme 3 funding for four meetings to be held in the UK on Current frontiers in inverse problems: from theory to applications, 2014.
- Leverhulme Early Career Fellowship awarded by the Leverhulme Trust and the Newton Trust. Duration: 01 November 2013 -- 31 October 2016. PI: L. Lellmann
- EPSRC first grant Nr. EP/J009539/1 Sparse & Higher-order Image Restoration. Duration: 03 May 2012 – 02 May 2014. PI: C.-B. Schönlieb
- Royal Society International Exchange Award Nr. IE110314 High-order Compressed Sensing for Medical Imaging. Duration: 01 Jan 2012 – 31 Dec 2013. PI’s: M. Burger & C.-B. Schönlieb. See online news of the University of Münster.
- EPSRC / Isaac Newton Trust Small Grant Non-smooth geometric reconstruction for high resolution MRI imaging of fluid transport in bed reactors. Duration: 01 July 2012 - 30 June 2013. PI: C.-B. Schönlieb
- Mathworks Academic Support for Development of Matlab Tools for the Numerical Analysis Tripos. Duration: July-September 2012. P.I.’s: S. Cowley, A. Iserles, C.-B. Schönlieb and A. Shadrin.