Theoretical Foundations for Pseudo-Inversion of Nonlinear Operators, SSVM-2023

Eyal Gofer, Guy Gilboa, Accepted to SSVM-2023 (oral)

9th International Conference, SSVM 2023, Santa Margherita di Pula, Italy, May 21–25, 2023, Proceedings, Springer LNCS 14009, pp. 29-41, 2023.

arXiv preprint

Springer conference proceedings

Abstract

The Moore-Penrose inverse is widely used in physics, statistics and various fields of engineering. Among other characteristics, it captures well the notion of inversion of linear operators in the case of overcomplete data. In data science, nonlinear operators are extensively used. In this paper we define and characterize the fundamental properties of a pseudo-inverse for nonlinear operators.
The concept is defined broadly. First for general sets, and then a refinement for normed spaces. Our pseudo-inverse for normed spaces yields the Moore-Penrose inverse when the operator is a matrix. We present conditions for existence and uniqueness of a pseudo-inverse and establish theoretical results investigating its properties, such as continuity, its value for operator compositions and projection operators, and others. Analytic expressions are given for the pseudo-inverse of some well-known, non-invertible, nonlinear operators, such as hard- or soft-thresholding and ReLU. Finally, we analyze a neural layer and discuss relations to wavelet thresholding and to regularized loss minimization.

Graph Laplacian for Semi-Supervised Learning, accepted to SSVM-2023

Or Streicher, Guy Gilboa, accepted to SSVM 2023 (oral)

9th International Conference, SSVM 2023, Santa Margherita di Pula, Italy, May 21–25, 2023, Proceedings, Springer LNCS 14009, pp. 250-262, 2023.

Springer conference proceedings

Arxiv preprint

Abatract

Semi-supervised learning is highly useful in common scenarios
where labeled data is scarce but unlabeled data is abundant. The
graph (or nonlocal) Laplacian is a fundamental smoothing operator for
solving various learning tasks. For unsupervised clustering, a spectral embedding
is often used, based on graph-Laplacian eigenvectors. For semisupervised
problems, the common approach is to solve a constrained
optimization problem, regularized by a Dirichlet energy, based on the
graph-Laplacian. However, as supervision decreases, Dirichlet optimization
becomes suboptimal. We therefore would like to obtain a smooth
transition between unsupervised clustering and low-supervised graphbased
classification.
In this paper, we propose a new type of graph-Laplacian which is adapted
for Semi-Supervised Learning (SSL) problems. It is based on both density
and contrastive measures and allows the encoding of the labeled data directly
in the operator. Thus, we can perform successfully semi-supervised
learning using spectral clustering. The benefits of our approach are illustrated
for several SSL problems.

The Underlying Correlated Dynamics in Neural Training, preprint

Rotem Turjeman, Tom Berkov, Ido Cohen, Guy Gilboa

Arxiv preprint

Training of neural networks is a computationally intensive task. The significance of understanding and modeling the training dynamics is growing as increasingly larger networks are being trained. We propose in this work a model based on the correlation of the parameters’ dynamics, which dramatically reduces the dimensionality. We refer to our algorithm as \emph{correlation mode decomposition} (CMD). It splits the parameter space into groups of parameters (modes) which behave in a highly correlated manner through the epochs.
We achieve a remarkable dimensionality reduction with this approach, where networks like ResNet-18, transformers and GANs, containing millions of parameters, can be modeled well using just a few modes. We observe each typical time profile of a mode is spread throughout the network in all layers. Moreover, our model induces regularization which yields better generalization capacity on the test set. This representation enhances the understanding of the underlying training dynamics and can pave the way for designing better acceleration techniques.

Analysis of Branch Specialization and its Application in Image Decomposition

Jonathan Brokman & Guy Gilboa, arXiv 2206.05810

Abstract

Branched neural networks have been used extensively for a variety of tasks. Branches are sub-parts of the model that perform independent processing followed by aggregation. It is known that this setting induces a phenomenon called Branch Specialization, where different branches become experts in different sub-tasks. Such observations were qualitative by nature. In this work, we present a methodological analysis of Branch Specialization. We explain the role of gradient descent in this phenomenon. We show that branched generative networks naturally decompose animal images to meaningful channels of fur, whiskers and spots and face images to channels such as different illumination components and face parts.

How to Guide Adaptive Depth Sampling?

Ilya Tcenov, Guy Gilboa, arXiv preprint

Abstract

Recent advances in depth sensing technologies allow fast electronic maneuvering of the laser beam, as opposed to fixed mechanical rotations. This will enable future sensors, in principle, to vary in real-time the sampling pattern. We examine here the abstract problem of whether adapting the sampling pattern for a given frame can reduce the reconstruction error or allow a sparser pattern. We propose a constructive generic method to guide adaptive depth sampling algorithms.
Given a sampling budget B, a depth predictor P and a desired quality measure M, we propose an Importance Map that highlights important sampling locations. This map is defined for a given frame as the per-pixel expected value of M produced by the predictor P, given a pattern of B random samples. This map can be well estimated in a training phase. We show that a neural network can learn to produce a highly faithful Importance Map, given an RGB image. We then suggest an algorithm to produce a sampling pattern for the scene, which is denser in regions that are harder to reconstruct. The sampling strategy of our modular framework can be adjusted according to hardware limitations, type of depth predictor, and any custom reconstruction error measure that should be minimized. We validate through simulations that our approach outperforms grid and random sampling patterns as well as recent state-of-the-art adaptive algorithms.

Adaptive Anisotropic Total Variation – Analysis and Experimental Findings of Nonlinear Spectral Properties

J. of Mathematical Imaging and Vision (JMIV), Vol. 64, pp. 916–938, 2022.

Shai Biton and Guy Gilboa

pdf

Springer link

Abstract

Our aim is to explain and characterize the behavior of adaptive total-variation (TV) regularization. TV has been widely used as an edge-preserving regularizer. However, objects are often over-regularized by TV, becoming blob-like convex structures of low curvature. This phenomenon was explained mathematically in the analysis of Andreau et al. They have shown that a TV regularizer can spatially preserve perfectly sets which are nonlinear eigenfunctions of the form $\lambda u \in \partial J_{TV}(u)$, where $\partial J_{TV}(u)$ is the TV subdifferential. For TV, these shapes are indeed convex sets of low-curvature.
A compelling approach to better preserve structures is to use adaptive anisotropic functionals, which adapt the regularization in an image-driven manner, with strong regularization along edges and low across them.
This follows the seminal work of Weickert on anisotropic diffusion. Adaptive anisotropic TV (A$^2$TV) was successfully used in several studies in the past decade. However, there is little analysis of the type of structures which can be well preserved. In this study we address this question by a joint methodology of mathematical derivations and experiments.

We rely on a recently developed theory of Burger et al on nonlinear spectral analysis of one-homogeneous functionals. We have that eigenfunction sets, admitting $\lambda u \in \partial J_{A^2TV}(u)$, are perfectly preserved under A$^2$TV-flow or minimization with $L^2$ square fidelity. We thus investigate these eigenfunctions theoretically and numerically. We prove non-convex sets can be eigenfunctions in certain conditions and provide numerical results which characterize well the relations between the degree of local anisotropy of the functional and the admitted maximal curvature. A nonlinear spectral representation is formulated, where shapes are well preserved and can be manipulated effectively. Finally, examples of possible applications related to shape manipulation and guided regularization of medical and depth data are shown.

Total-Variation – Fast Gradient Flow and Relations to Koopman Theory

Ido Cohen, Tom Berkov, arXiv preprint

Code

Abstract

The space-discrete Total Variation (TV) flow is analyzed using several mode decomposition techniques. In the one-dimensional case, we provide analytic formulations to Dynamic Mode Decomposition (DMD) and to Koopman Mode Decomposition (KMD) of the TV-flow and compare the obtained modes to TV spectral decomposition. We propose a computationally efficient algorithm to evolve the one-dimensional TV-flow. A significant speedup by three orders of magnitude is obtained, compared to iterative minimizations. A common theme, for both mode analysis and fast algorithm, is the significance of phase transitions during the flow, in which the subgradient changes. We explain why applying DMD directly on TV-flow measurements cannot model the flow or extract modes well. We formulate a more general method for mode decomposition that coincides with the modes of KMD. This method is based on the linear decay profile, typical to TV-flow. These concepts are demonstrated through experiments, where additional extensions to the two-dimensional case are given.

PhIT-Net: Photo-consistent Image Transform for Robust Illumination Invariant Matching

The 32nd British Machine Vision Conference (BMVC), Nov. 2021.

Damian Kaliroff and Guy Gilboa

BMVC link to paper, video and code

Abstract

We propose a new and completely data-driven approach for generating a photo- consistent image transform. We show that simple classical algorithms which operate in the transform domain become extremely resilient to illumination changes. This considerably improves matching accuracy, outperforming the use of state-of-the-art invariant representations as well as new matching methods based on deep features. The transform is obtained by training a neural network with a specialized triplet loss, designed to emphasize actual scene changes while attenuating illumination changes. The transform yields an illumination invariant representation, structured as an image map, which is highly flexible and can be easily used for various tasks.

Adaptive LiDAR Sampling and Depth Completion using Ensemble Variance

Accepted to IEEE Trans. Image Processing, 2021.

Eyal Gofer, Shachar Praisler, Guy Gilboa, arXiv

Project details

Abstract

This work considers the problem of depth completion, with or without image data, where an algorithm may measure the depth of a prescribed limited number of pixels. The algorithmic challenge is to choose pixel positions strategically and dynamically to maximally reduce overall depth estimation error. This setting is realized in daytime or nighttime depth completion for autonomous vehicles with a programmable LiDAR. Our method uses an ensemble of predictors to define a sampling probability over pixels. This probability is proportional to the variance of the predictions of ensemble members, thus highlighting pixels that are difficult to predict. By additionally proceeding in several prediction phases, we effectively reduce redundant sampling of similar pixels. Our ensemble-based method may be implemented using any depth-completion learning algorithm, such as a state-of-the-art neural network, treated as a black box. In particular, we also present a simple and effective Random Forest-based algorithm, and similarly use its internal ensemble in our design. We conduct experiments on the KITTI dataset, using the neural network algorithm of Ma et al. and our Random Forest based learner for implementing our method. The accuracy of both implementations exceeds the state of the art. Compared with a random or grid sampling pattern, our method allows a reduction by a factor of 4-10 in the number of measurements required to attain the same accuracy.

Examining the Limitations of Dynamic Mode Decomposition through Koopman Theory Analysis

Ido Cohen, Guy Gilboa, arXiv preprint 2107.07456, 2021

Abstract

This work binds the existence of Koopman Eigenfunctions (KEF’s), the geometric of the dynamics, and the validity of Dynamic Mode Decomposition (DMD) to one coherent theory. Viewing the dynamic as a curve in the state-space allows us to formulate an existence condition of KEF’s and their multiplicities. These conditions lay the foundations for system reconstruction, global controllability, and observability for nonlinear dynamics.

DMD can be interpreted as a finite dimension approximation of Koopman Mode Decomposition (KMD). However, this method is limited to the case when KEF’s are linear combinations of the observations. We examine the limitations of DMD through the analysis of Koopman theory. We propose a new mode decomposition technique based on the typical time profile of the dynamics. An overcomplete dictionary of decay profiles is used to sparsely represent the dynamic. This analysis is also valid in the full continuous setting of Koopman theory, which is based on variational calculus.

We demonstrate applications of this analysis, such as finding KEF’s and their multiplicities, calculating KMD, dynamics reconstruction, global linearization, and controllability.