# Horizon 2020 RISE grant started March 2018

A 4 year grant NoMADS,  Nonlocal Methods for Arbitrary Data Sources, as part of the RISE program, started on March 2018.

Includes Collaboration between universities (Munster, Cambridge, UCLA, Bordeaux, Carnegie Mellon, Technion and more..) and industry.

# Paper of Raz & Guy accepted

The paper, based on the master thesis of Raz Nossek is now accepted (Sept 2017):

R. Nossek & G. Gilboa, “Flows generating nonlinear eigenfunctions”, Journal of Scientific Computing.

(a pdf of the accepted version will be published soon, see arXiv version)

# Flows Generating Nonlinear Eigenfunctions

### Image above: Eigenfunction of TGV found by the proposed flow.

Abstract:

Nonlinear variational methods have become very powerful tools for many image processing tasks. Recently a new line of research has emerged, dealing with nonlinear eigenfunctions induced by convex functionals. This has provided new insights and better theoretical understanding of convex regularization and introduced new processing methods. However, the theory of nonlinear eigenvalue problems is still at its infancy. We present a new flow that can generate nonlinear eigenfunctions of the form $T(u)=\lambda u$, where $T(u)$ is a nonlinear operator and $\lambda \in \mathbb{R}$ is the eigenvalue. We develop the theory where $T(u)$ is a subgradient element of a regularizing one-homogeneous functional, such as total-variation (TV) or total-generalized-variation (TGV). We introduce two flows: a forward flow and an inverse flow; for which the steady state solution is a nonlinear eigenfunction. The forward flow monotonically smooths the solution (with respect to the regularizer) and simultaneously increases the $L^2$ norm. The inverse flow has the opposite characteristics. For both flows, the steady state depends on the initial condition, thus different initial conditions yield different eigenfunctions. This enables a deeper investigation into the space of nonlinear eigenfunctions, allowing to produce numerically diverse examples, which may be unknown yet. In addition we suggest an indicator to measure the affinity of a function to an eigenfunction and relate it to pseudo-eigenfunctions in the linear case

# Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches

### Ester Hait, Guy Gilboa, “Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches”, accepted to IEEE Trans. Image Processing, 2017.

Abstract:

We propose to combine semantic data and registration algorithms to solve various image processing problems such as denoising, super-resolution and color-correction.
It is shown how such new techniques can achieve significant quality enhancement, both visually and quantitatively, in the case of facial image enhancement. Our model assumes prior high quality data of the person to be processed, but no knowledge of the degradation model.
We try to overcome the classical processing limits by using semantically-aware patches, with adaptive size and location regions of coherent structure and context, as building blocks. The method is demonstrated on the problem of cellular photography enhancement of dark facial images for different identities, expressions and poses

# TV Transform & Nonlinear Spectral Theory

A new framework is proposed for variational analysis and processing. It defines a functional-based nonlinear transform and inverse-transform. The framework is developed in the context of total-variation (TV), but it can be generalized to other functionals.

An eigenfunction, with respect to the subdifferential of the functional, such as a disk in the TV case, yields an impulse in the transform domain. This can be viewed as a generalization of known spectral approaches, based on linear algebra, which are extensively used in image-processing, e.g. for segmentation.

Following the Fourier intuition, a spectrum can be computed to analyze dominant scales in the image. Moreover, new nonlinear low-pass, high-pass and band-pass filters can be designed with full contrast and edge preservation.

Movie explained: Top (from left): image layers, phi bands (phi(t) = u_{tt}*t). Bottom: Ideal TV High-pass-filter, ideal TV Low-pass-filter at different scales. Ideal is in the sense of eigenfunctoin preservation.

Related papers

1. R. Nossek, G. Gilboa, “Flows generating nonlinear eigenfunctions”, accepted to J. of Scientific Computing, 2017.
2. G. Gilboa, “Semi-inner-products for convex functionals and their use in image decomposition”, Journal of Mathematical Imaging and Vision (JMIV), Vol. 57, No. 1, pp. 26-42, 2017.
3. M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), 2017.
4. M. Benning, G. Gilboa, J.S. Grah, C. Bibiane Schönlieb, “Learning Filter Functions in Regularisers by Minimising Quotients.” In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), pp. 511-523. Springer, 2017.
5. M. Burger, G. Gilboa, M. Moeller, L. Eckardt, D. Cremers, “Spectral decompositions using one-homogeneous functionals”, SIAM J. Imaging Sciences, Vol. 9, No. 3, pp. 1374-1408, 2016.
6. M. Benning, G. Gilboa, C.B. Schönlieb, “ Learning parametrised regularisation functions via quotient minimisation”. PAMM-16, 16(1), 933-936, 2016.
7. G. Gilboa, M. Moeller, M. Burger, “Nonlinear spectral analysis via one-homogeneous functionals – overview and future prospects”, accepted to the Journal of Mathematical Imaging and Vision (JMIV), Vol. 56, No. 2, pp 300–319, 2016.
8. D. Horesh, G. Gilboa, “Separating surfaces for structure-texture decomposition using the TV transform”, IEEE Trans. Image Processing, Vol. 25, No. 9, pp. 4260 – 4270, 2016.
9. G. Gilboa, “A total variation spectral framework for scale and texture analysis”. SIAM Journal on Imaging Sciences, Vol. 7, No. 4, pp. 1937–1961,  2014.
10. G. Gilboa, “Nonlinear Band-Pass Filtering Using the TV Transform”, Proc. European Signal Processing Conference (EUSIPCO-2014), Lisbon, p. 1696 – 1700, 2014.
11. G. Gilboa, “A spectral approach to total variation”, Scale-Space and Variational Methods”, SSVM 2013, LNCS 7893, p. 36-47, 2013.
12. G. Gilboa, CCIT Report 833, Technion, 2013.

# TIP Paper of Ety

Ety’s paper in IEEE Trans. on Image Processing is published, “Blind Facial Image Quality Enhancement Using Non-Rigid Semantic Patches “.

# Semi-Inner-Products for Convex Functionals and Their Use in Image Decomposition

### Guy Gilboa, Journal of Mathematical Imaging and Vision (JMIV), Vol. 57, No. 1, pp. 26-42, 2017.

Abstract:

Semi-inner-products in the sense of Lumer are extended to convex functionals. This yields a Hilbert-space like structure to convex functionals in Banach spaces. In particular, a general expression for semi-inner-products with respect to one homogeneous functionals is given. Thus one can use the new operator for the analysis of total variation and higher order functionals like total-generalized-variation (TGV). Having a semi-inner-product, an angle between functions can be defined in a straightforward manner. It is shown that in the one homogeneous case the Bregman distance can be expressed in terms of this newly defined angle. In addition, properties of the semi-inner-product of nonlinear eigenfunctions induced by the functional are derived. We use this construction to state a sufficient condition for a perfect decomposition of two signals and suggest numerical measures which indicate when those conditions are approximately met.

# Sharpening and Enhancement

Main topics

• Image sharpening using forward-and-backward diffusion (simultaneous sharpening and denoising) – enhancing edges while keeping low noise.
• Complex diffusion processes – using the imaginary part which imarges to be a stable nonlinear edge detector to guide the diffusion process.
• Inverse scale space –  a generalization of Bregman iterations from a variational to a new non-standard PDE formulation (with Burger, Osher and Xu).

Complex diffusion real and imaginary kernels

Relaxed inverse scale space

Related papers

1. E. Hait, G. Gilboa, “Blind facial image quality enhancement using non-rigid semantic patches.” IEEE Transactions on Image Processing Vol. 26, No. 6, pp. 2705-2720, 2017.
2. M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), 2017.
3. O. Spier, T. Treibitz, G. Gilboa, “In situ target-less calibration of turbid media”, Int. Conf. on Computational Photography (ICCP), Stanford Univ., 2017.
4. D. Horesh, G. Gilboa, “Separating surfaces for structure-texture decomposition using the TV transform”, IEEE Trans. Image Processing, Vol. 25, No. 9, pp. 4260 – 4270, 2016.
5. O. Katzir, G. Gilboa, “A Maximal Interest-Point Strategy Applied to Image Enhancement with External Priors”, Proc. IEEE Int. Conf. Image Processing (ICIP), 2015.
6. G. Gilboa, “Nonlinear Scale Space with Spatially Varying Stopping Time”, PAMI, Vol. 30, No. 12, pp. 2175-2187, 2008.
7. M. Welk, G. Gilboa, J. Weickert, “Theoretical foundations for discrete forward-and-backward diffusion filtering”. SSVM 2009, pp. 527-538, 2009.
8. M. Burger, G. Gilboa, S. Osher, J. Xu, , “Nonlinear inverse scale space methods”, Communications in Mathematical Sciences (CMS) Vol 4, No.1, pp. 179-212, 2006.
9. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image sharpening by flows based on triple well potentials”,  J. of Math. Imaging and Vision, 20:121-131, 2004.
10. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image enhancement and denoising by complex diffusion processes”, IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 26, No. 8, pp. 1020-1036, 2004.
11. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Forward-and-Backward diffusion processes for adaptive image enhancement and denoising”, IEEE Trans. on Image Processing, Vol. 11, No. 7, pp. 689-703, 2002.
12. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Regularized shock filters and complex diffusion”,  – ECCV-’02, LNCS 2350, pp. 399-313, Springer-Verlag 2002.
13. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image enhancement segmentation and denoising by time dependent nonlinear diffusion processes”, Proc. IEEE ICIP-’01, Thessaloniki, Greece, vol. 3, pp. 134-137, 2001.
14. M. Burger, S. Osher, J. Xu, G. Gilboa, “Nonlinear Inverse Scale Space Methods for Image Restoration”, Variational and Level-Set Methods (VLSM) 2005,  LNCS 3752, pp. 25-36, Springer-Verlag, 2005.
15. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Complex diffusion processes for image filtering”, Scale-Space ’01, LNCS 2106, pp. 299-307, Springer-Verlag 2001.
16. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Resolution enhancement by forward-and-backward nonlinear diffusion processes”, Nonlinear Signal and Image Processing, Baltimore, Maryland, June 2001.
17. N. Sochen, G. Gilboa, Y.Y. Zeevi, “Color image enhancement by a forward-and-backward adaptive Beltrami flow”, AFPAC-2000, LNCS 1888, pp. 319-328, 2000, Springer-Verlag.
18. G. Gilboa, Y.Y. Zeevi, N. Sochen, “Signal and image enhancement by a generalized forward-and-backward adaptive diffusion process”, EUSIPCO-2000, Tampere, Finland, Sept. 2000.
19. G. Gilboa, Y.Y. Zeevi, N. Sochen, “Anisotropic selective inverse diffusion for signal enhancement in the presence of noise”, Proc. IEEE ICASSP-2000, Istanbul, Turkey, vol. I, pp. 211-224, June 2000.

# Nonlinear Spectral Image Fusion

### M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, Proc. Scale-Space and Variational Methods (SSVM), 2017.

Abstract:

In this paper we demonstrate that the framework of nonlinear spectral decompositions based on total variation (TV) regularization is very well suited for image fusion as well as more general image manipulation tasks. The well-localized and edge-preserving spectral TV decomposition allows to select frequencies of a certain image to transfer particular features, such as wrinkles in a face, from one image to another. We illustrate the effectiveness of the proposed approach in several numerical experiments, including a comparison to the competing techniques of Poisson image editing, linear osmosis, wavelet fusion and Laplacian pyramid fusion. We conclude that the proposed spectral TV image decomposition framework is a valuable tool for semi- and fully-automatic image editing and fusion.