JMIV 2024: Generalized Inversion of Nonlinear Operators
Eyal Gofer, Guy Gilboa, J. Mathematical Imaging and Vision (JMIV), 2024 Springer Open Access link Inversion of operators is a fundamental concept in data processing. Inversion of linear operators is well studied, supported by established theory. When an inverse either does not exist or is not unique, generalized inverses are used. Most notable is the […]
ICLR 2024: Enhancing Neural Training via a Correlated Dynamics Model
Jonathan Brokman, Roy Betser, Rotem Turjeman, Tom Berkov, Ido Cohen, Guy Gilboa, ICLR 2024 Related preprint As neural networks grow in scale, their training becomes both computationally demanding and rich in dynamics. Amidst the flourishing interest in these training dynamics, we present a novel observation: Parameters during training exhibit intrinsic correlations over time. Capitalizing on […]
TOG 2024: Spectral Total-Variation Processing of Shapes – Theory and Applications
Jonathan Brokman, Martin Burger, Guy Gilboa, ACM Transactions on Graphics, 2024, https://doi.org/10.1145/3641845 Related preprint We present an analysis of total-variation (TV) on non-Euclidean parameterized surfaces, a natural representation of the shapes used in 3D graphics. Our work explains recent experimental findings in shape spectral TV [Fumero et al., 2020] and adaptive anisotropic spectral TV [Biton and […]
DXAI: Explaining Classification by Image Decomposition
Elnatan Kadar, Guy Gilboa arxiv preprint We propose a new way to explain and to visualize neural network classification through a decomposition-based explainable AI (DXAI). Instead of providing an explanation heatmap, our method yields a decomposition of the image into class-agnostic and class-distinct parts, with respect to the data and chosen classifier. Following a fundamental […]
Critical Points ++: An Agile Point Cloud Importance Measure for Robust Classification, Adversarial Defense and Explainable AI
Yossef Meir Levi, Guy Gilboa The ability to cope accurately and fast with Out-Of-Distribution (OOD) samples is crucial in real-world safety demanding applications. In this work we first study the interplay between critical points of 3D point clouds and OOD samples. Our findings are that common corruptions and outliers are often interpreted as critical points. […]
Generalized Inversion of Nonlinear Operators
Eyal Gofer, Guy Gilboa Inversion of operators is a fundamental concept in data processing. Inversion of linear operators is well studied, supported by established theory. When an inverse either does not exist or is not unique, generalized inverses are used. Most notable is the Moore-Penrose inverse, widely used in physics, statistics, and various fields of […]
EPiC: Ensemble of Partial Point Clouds for Robust Classification, ICCV 2023
Meir Yossef Levi, Guy Gilboa Accepted to ICCV 2023. Robust point cloud classification is crucial for real-world applications, as consumer-type 3D sensors often yield partial and noisy data, degraded by various artifacts. In this work we propose a general ensemble framework, based on partial point cloud sampling. Each ensemble member is exposed to only partial […]
Additive Class Distinction Maps using Branched-GANs
Elnatan Kadar, Jonathan Brokman, Guy Gilboa Arxiv preprint We present a new model, training procedure and architecture to create precise maps of distinction between two classes of images. The objective is to comprehend, in pixel-wise resolution, the unique characteristics of a class. These maps can facilitate self-supervised segmentation and objectdetection in addition to new capabilities in explainable […]
BASiS: Batch Aligned Spectral Embedding Space, CVPR 2023
Or Streicher, Ido Cohen, Guy Gilboa; Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 10396-10405 CVPR Repository Arxiv preprint Graph is a highly generic and diverse representation, suitable for almost any data processing problem. Spectral graph theory has been shown to provide powerful algorithms, backed by solid linear algebra […]
Theoretical Foundations for Pseudo-Inversion of Nonlinear Operators, SSVM-2023
Eyal Gofer, Guy Gilboa, Accepted to SSVM-2023 (oral) 9th International Conference, SSVM 2023, Santa Margherita di Pula, Italy, May 21–25, 2023, Proceedings, Springer LNCS 14009, pp. 29-41, 2023. arXiv preprint Springer conference proceedings Abstract The Moore-Penrose inverse is widely used in physics, statistics and various fields of engineering. Among other characteristics, it captures well the […]
Graph Laplacian for Semi-Supervised Learning, accepted to SSVM-2023
Or Streicher, Guy Gilboa, accepted to SSVM 2023 (oral) 9th International Conference, SSVM 2023, Santa Margherita di Pula, Italy, May 21–25, 2023, Proceedings, Springer LNCS 14009, pp. 250-262, 2023. Springer conference proceedings Arxiv preprint Abatract Semi-supervised learning is highly useful in common scenarios where labeled data is scarce but unlabeled data is abundant. The graph […]
The Underlying Correlated Dynamics in Neural Training, preprint
Rotem Turjeman, Tom Berkov, Ido Cohen, Guy Gilboa Arxiv preprint Training of neural networks is a computationally intensive task. The significance of understanding and modeling the training dynamics is growing as increasingly larger networks are being trained. We propose in this work a model based on the correlation of the parameters’ dynamics, which dramatically reduces […]
Analysis of Branch Specialization and its Application in Image Decomposition
Jonathan Brokman & Guy Gilboa, arXiv 2206.05810 Abstract Branched neural networks have been used extensively for a variety of tasks. Branches are sub-parts of the model that perform independent processing followed by aggregation. It is known that this setting induces a phenomenon called Branch Specialization, where different branches become experts in different sub-tasks. Such observations […]
How to Guide Adaptive Depth Sampling?
Ilya Tcenov, Guy Gilboa, arXiv preprint Abstract Recent advances in depth sensing technologies allow fast electronic maneuvering of the laser beam, as opposed to fixed mechanical rotations. This will enable future sensors, in principle, to vary in real-time the sampling pattern. We examine here the abstract problem of whether adapting the sampling pattern for a […]
Adaptive Anisotropic Total Variation – Analysis and Experimental Findings of Nonlinear Spectral Properties
J. of Mathematical Imaging and Vision (JMIV), Vol. 64, pp. 916–938, 2022. Shai Biton and Guy Gilboa pdf Springer link Abstract Our aim is to explain and characterize the behavior of adaptive total-variation (TV) regularization. TV has been widely used as an edge-preserving regularizer. However, objects are often over-regularized by TV, becoming blob-like convex structures […]
Total-Variation – Fast Gradient Flow and Relations to Koopman Theory
Ido Cohen, Tom Berkov, arXiv preprint Code Abstract The space-discrete Total Variation (TV) flow is analyzed using several mode decomposition techniques. In the one-dimensional case, we provide analytic formulations to Dynamic Mode Decomposition (DMD) and to Koopman Mode Decomposition (KMD) of the TV-flow and compare the obtained modes to TV spectral decomposition. We propose a […]
PhIT-Net: Photo-consistent Image Transform for Robust Illumination Invariant Matching
The 32nd British Machine Vision Conference (BMVC), Nov. 2021. Damian Kaliroff and Guy Gilboa BMVC link to paper, video and code Abstract We propose a new and completely data-driven approach for generating a photo- consistent image transform. We show that simple classical algorithms which operate in the transform domain become extremely resilient to illumination changes. […]
Adaptive LiDAR Sampling and Depth Completion using Ensemble Variance
Accepted to IEEE Trans. Image Processing, 2021. Eyal Gofer, Shachar Praisler, Guy Gilboa, arXiv Project details Abstract This work considers the problem of depth completion, with or without image data, where an algorithm may measure the depth of a prescribed limited number of pixels. The algorithmic challenge is to choose pixel positions strategically and […]
Examining the Limitations of Dynamic Mode Decomposition through Koopman Theory Analysis
Ido Cohen, Guy Gilboa, arXiv preprint 2107.07456, 2021 Abstract This work binds the existence of Koopman Eigenfunctions (KEF’s), the geometric of the dynamics, and the validity of Dynamic Mode Decomposition (DMD) to one coherent theory. Viewing the dynamic as a curve in the state-space allows us to formulate an existence condition of KEF’s and […]
Total-Variation Mode Decomposition
Ido Cohen, Tom Berkov, Guy Gilboa, “Total-Variation Mode Decomposition”, Proc. SSVM 2021, pp. 52-64 Abstract In this work we analyze the Total Variation (TV) flow applied to one dimensional signals. We formulate a relation between Dynamic Mode Decomposition (DMD), a dimensionality reduction method based on the Koopman operator, and the spectral TV decomposition. DMD is adapted by time rescaling […]
Nonlinear Spectral Processing of Shapes via Zero-homogeneous Flows
Jonathan Brokman, Guy Gilboa, Proc. SSVM, “Nonlinear Spectral Processing of Shapes via Zero-homogeneous Flows”, pp. 40-51, 2021 Abstract In this work we extend the spectral total-variation framework, and use it to analyze and process 2D manifolds embedded in 3D. Analysis is performed in the embedding space – thus “spectral arithmetics” manipulate the shape directly. This […]
Nonlinear Power Method for Computing Eigenvectors of Proximal Operators and Neural Networks
Accepted to SIAM J. on Imaging Scienes, 2021 Leon Bungert, Ester Hait-Fraenkel , Nicolas Papadakis and Guy Gilboa, arXiv Neural networks have revolutionized the field of data science, yielding remarkable solutions in a data-driven manner. For instance, in the field of mathematical imaging, they have surpassed traditional methods based on convex regularization. However, a fundamental […]
Modes of Homogeneous Gradient Flows
Accepted to SIAM J. on Imaging Sciences, 2021 Ido Cohen, Omri Azencot, Pavel Lifshitz, Guy Gilboa, arXiv, July 2020 Finding latent structures in data is drawing increasing attention in broad and diverse fields such as fluid dynamics, signal processing, and machine learning. In this work, we formulate Dynamic Mode Decomposition (DMD) for two types […]
Revealing stable and unstable modes of denoisers through nonlinear eigenvalue analysis
Ester Hait-Fraenkel, Guy Gilboa, Revealing stable and unstable modes of denoisers through nonlinear eigenvalue analysis. J. Vis. Commun. Image Represent. 75: 103041 (2021) Ester Hait-Fraenkel, Guy Gilboa, arXiv In this paper, we propose to analyze stable and unstable modes of generic image denoisers through nonlinear eigenvalue analysis. We attempt to find input images for which […]
Experts with Lower-Bounded Loss Feedback: A Unifying Framework
Eyal Gofer, Guy Gilboa, arxiv preprint The most prominent feedback models for the best expert problem are the full information and bandit models. In this work we consider a simple feedback model that generalizes both, where on every round, in addition to a bandit feedback, the adversary provides a lower bound on the loss of […]
Iterative Methods for Computing Eigenvectors of Nonlinear Operators
Guy Gilboa, arXiv preprint A chapter to appear in Handbook of Mathematical Models and Algorithms in Computer Vision and Imaging. In this chapter we are examining several iterative methods for solving nonlinear eigenvalue problems. These arise in variational image-processing, graph partition and classification, nonlinear physics and more. The canonical eigenproblem we solve is $T(u)=\lambda […]
NeurIPS 2020: Deeply Learned Spectral Total Variation Decomposition
Tamara G. Grossmann, Yury Korolev, Guy Gilboa, Carola-Bibiane Schönlieb, arXiv 2020 Accepted for NeurIPS 2020. Non-linear spectral decompositions of images based on one-homogeneous functionals such as total variation have gained considerable attention in the last few years. Due to their ability to extract spectral components corresponding to objects of different size and contrast, such decompositions […]
Super-Pixel Sampler – a Data-driven Approach for Depth Sampling and Reconstruction
Adam Wolff, Shachar Praisler, Ilya Tcenov and Guy Gilboa, “Super-Pixel Sampler – a Data-driven Approach for Depth Sampling and Reconstruction”, accepted to ICRA (Int. Conf. on Robotics and Automation) 2020. Paper See the video of our mechanical prototype Abstract Depth acquisition, based on active illumination, is essential for autonomous and robotic navigation. LiDARs (Light Detection […]
Self-Supervised Unconstrained Illumination Invariant Representation
Damian Kaliroff, Guy Gilboa, arXiv We propose a new and completely data-driven approach for generating an unconstrained illumination invariant representation of images. Our method trains a neural network with a specialized triplet loss designed to emphasize actual scene changes while downplaying changes in illumination. For this purpose we use the BigTime image dataset, which contains […]
Energy Dissipating Flows for Solving Nonlinear Eigenpair Problems
Ido Cohen, Guy Gilboa, “Energy dissipating flows for solving nonlinear eigenpair problems”, Journal of Computational Physics 375 (2018), 1138-1158. This work is concerned with computing nonlinear eigenpairs, which model solitary waves and various other physical phenomena. We aim at solving nonlinear eigenvalue problems of the general form $T(u)=\lambda Q(u)$. In our setting $T$ is a […]
Spectral Total-Variation Local Scale Signatures for Image Manipulation and Fusion
Ester Hait, Guy Gilboa, “Spectral Total-Variation Local Scale Signatures for Image Manipulation and Fusion”, preprint, HAL 01722459, 2018. We propose a unified framework for isolating, comparing and differentiating objects within an image. We rely on the recently proposed total-variation transform, yielding a continuous, multi-scale, fully edge-preserving, local descriptor, referred to as spectral total-variation local […]
Theoretical Analysis of Flows Estimating Eigenfunctions of One-homogeneous Functionals
Jean-Francois Aujol, Guy Gilboa, Nicolas Papadakis, “Theoretical Analysis of Flows Estimating Eigenfunctions of One-homogeneous Functionals”, accepted to SIAM J. on Imaging Sciences. Nonlinear eigenfunctions, induced by subgradients of one-homogeneous functionals (such as the 1-Laplacian), have shown to be instrumental in segmentation, clustering and image decomposition. We present a class of ows for nding such eigenfunctions, […]
A Discrete Theory and Efficient Algorithms for Forward-and-Backward Diffusion Filtering
Martin Welk, Joachim Weickert, Guy Gilboa, “A Discrete Theory and Efficient Algorithms for Forward-and-Backward Diffusion Filtering”, preprint. Image enhancement with forward-and-backward (FAB) diffusion lacks a sound theory and is numerically very challenging due to its diffusivities that are negative within a certain gradient range. In our paper we address both problems. First we establish a comprehensive theory for space-discrete […]
Flows Generating Nonlinear Eigenfunctions
Raz Nossek, Guy Gilboa, “Flows Generating Nonlinear Eigenfunctions”, J. of Scientific Computing , Volume 75, Issue 2, pp 859–888, 2018. Image above: Eigenfunction of TGV found by the proposed flow. Abstract: Nonlinear variational methods have become very powerful tools for many image processing tasks. Recently a new line of research has emerged, dealing with nonlinear […]
Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches
Ester Hait, Guy Gilboa, “Blind Facial Image Quality Enhancement using Non-Rigid Semantic Patches”, accepted to IEEE Trans. Image Processing, 2017. Abstract: We propose to combine semantic data and registration algorithms to solve various image processing problems such as denoising, super-resolution and color-correction. It is shown how such new techniques can achieve significant quality enhancement, both […]
Semi-Inner-Products for Convex Functionals and Their Use in Image Decomposition
Guy Gilboa, Journal of Mathematical Imaging and Vision (JMIV), Vol. 57, No. 1, pp. 26-42, 2017. Abstract: Semi-inner-products in the sense of Lumer are extended to convex functionals. This yields a Hilbert-space like structure to convex functionals in Banach spaces. In particular, a general expression for semi-inner-products with respect to one homogeneous functionals is given. Thus one […]
Nonlinear Spectral Image Fusion
M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, Proc. Scale-Space and Variational Methods (SSVM), 2017. Abstract: In this paper we demonstrate that the framework of nonlinear spectral decompositions based on total variation (TV) regularization is very well suited for image fusion as well as more […]
In Situ Target-Less Calibration of Turbid Media
1. O. Spier, T. Treibitz, G. Gilboa, “In situ target-less calibration of turbid media”, Int. Conf. on Computational Photography (ICCP), Stanford Univ., 2017. Abstract: The color of an object imaged in a turbid medium varies with distance and medium properties, deeming color an unstable source of information. Assuming 3D scene structure has become relatively easy to estimate, the […]
Robust Recovery of Heavily Degraded Depth Measurements
G. Drozdov, Y. Shapiro, G. Gilboa, “Robust recovery of heavily degraded depth measurements”, Int. Conf. On 3D Vision (3DV), Stanford University, 2016. Abstract: The revolution of RGB-D sensors is advancing towards mobile platforms for robotics, autonomous vehicles and consumer hand-held devices. Strong pressures on power consumption and system price require new powerful algorithms that can robustly handle very low […]
A Depth Restoration Occlusionless Temporal Dataset
D. Rotman, G. Gilboa, “A depth restoration occlusionless temporal dataset”, Int. Conf. On 3D Vision (3DV), Stanford University, 2016. Abstract: Depth restoration, the task of correcting depth noise and artifacts, has recently risen in popularity due to the increase in commodity depth cameras. When assessing the quality of existing methods, most researchers resort to the popular Middlebury dataset; however, […]
Spectral Decompositions using One-Homogeneous Functionals
Martin Burger, Guy Gilboa, Michael Moeller, Lina Eckardt, Daniel Cremers “Spectral Decompositions using One-Homogeneous Functionals”, SIAM Journal on Imaging Sciences, Vol. 9, No. 3, pp. 1374-1408, 2016. Abstract: This paper discusses the use of absolutely one-homogeneous regularization functionals in a variational, scale space, and inverse scale space setting to define a nonlinear spectral decomposition of […]
Separation Surfaces in the Spectral TV Domain for Texture Decomposition
Dikla Horesh and Guy Gilboa, “Separation Surfaces in the Spectral TV Domain for Texture Decomposition”, IEEE Trans. Image Processing, Vol. 25, No. 9, pp. 4260 – 4270, 2016. Abstract: In this paper we introduce a novel notion of separation surfaces for image decomposition. A surface is embedded in the spectral total-variation (TV) three dimensional domain and […]
Nonlinear Spectral Analysis via One-homogeneous Functionals – Overview and Future Prospects
Guy Gilboa, Michael Moeller, Martin Burger, “Nonlinear Spectral Analysis via One-homogeneous Functionals – Overview and Future Prospects”, accepted to Journal of Mathematical Imaging and Vision (JMIV), 2016. Abstract: We present in this paper the motivation and theory of nonlinear spectral representations, based on convex regularizing functionals. Some comparisons and analogies are drawn to the fields of […]
Learning Nonlinear Spectral Filters for Color Image Reconstruction
Michael Moeller, Julia Diebold, Guy Gilboa, Daniel Cremers, “Learning Nonlinear Spectral Filters for Color Image Reconstruction”, ICCV 2015. Abstract: This paper presents the idea of learning optimal filters for color image reconstruction based on a novel concept of nonlinear spectral image decompositions recently proposed by Gilboa. The general idea is to use total variation regularization […]
A Maximal Interest-Point Strategy Applied to Image Enhancement with External Priors
O. Katzir and G. Gilboa, “A Maximal Interest-Point Strategy Applied to Image Enhancement with External Priors”, IEEE International Conference on Image Processing (ICIP-2015) , pp. 1324-1328, 2015. Abstract: Curvature is a fundamental component in differential geometry. It is used extensively in signal, image and shape processing, as a feature and in segmentation flows and regularization processes. […]
On the Role of Non-local Menger Curvature in Image Processing
G. Gilboa, E. Appleboim, E. Saucan and Y.Y. Zeevi, “On the Role of Non-local Menger Curvature in Image Processing”, IEEE International Conference on Image Processing (ICIP-2015) , pp. 4337-4341, 2015. Abstract: Curvature is a fundamental component in differential geometry. It is used extensively in signal, image and shape processing, as a feature and in segmentation flows […]
A spectral framework for one-homogeneous functionals
M. Burger, L. Eckardt, G. Gilboa, M. Moeller, “A spectral framework for one-homogeneous functionals”, Proc. Scale Space and Variational Methods in Computer Vision (SSVM), 2015. Abstract: This paper discusses a generalization of spectral representations related to convex one-homogeneous regularization functionals, e.g. total variation or $\ell^1$-norms. Those functionals serve as a substitute for a Hilbert space […]
Multiscale Texture Orientation Analysis using Spectral Total-Variation Decomposition
D. Horesh, G. Gilboa, “Multiscale orientation detection using the total variation transform”, Proc. Scale Space and Variational Methods in Computer Vision (SSVM), 2015. Abstract: Multi-level texture separation can considerably improve texture analysis, a significant component in many computer vision tasks. This paper aims at obtaining precise local texture orientations of images in a multiscale manner, […]
Fundamentals of Non-local Total Variation Spectral Theory
J.-F. Aujol, G. Gilboa, N. Papadakis, “Non local total variation spectral framework”, Proc. Scale Space and Variational Methods in Computer Vision (SSVM), 2015. Abstract: Eigenvalue analysis based on linear operators has been extensively used in signal and image processing to solve a variety of problems such as segmentation, dimensionality reduction and more. Recently, nonlinear spectral […]
A Total Variation Spectral Framework for Scale and Texture Analysis
G. Gilboa, “A Total Variation Spectral Framework for Scale and Texture Analysis”, SIAM Journal on Imaging Sciences, Vol. 7, No. 4, pp. 1937–1961, 2014. Abstract: A new total variation (TV) spectral framework is presented. A TV transform is proposed which can be interpreted as a spectral domain, where elementary TV features, like disks, approach impulses. A […]
Nonlinear Band-Pass Filtering Using the TV Transform
G. Gilboa, “Nonlinear Band-Pass Filtering Using the TV Transform”, Proc. European Signal Processing Conference (EUSIPCO-2014), Lisbon, pp. 1696 – 1700, 2014. Abstract: A distinct family of nonlinear filters is presented. It is based on a new formalism, defining a nonlinear transform based on the TV-functional. Scales in this sense are related to the size of the object and […]
Report on the Total Variation Spectral Framework
G. Gilboa, “A Total Variation Spectral Framework for Scale and Texture Analysis”, CCIT Report 833, 2013. Abstract: A new total variation (TV) spectral framework is presented. A TV transform is proposed which can be interpreted as a spectral domain, where elementary TV features, like disks, approach impulses. A reconstruction formula from the pectral to the spatial […]
A Spectral Approach to Total Variation
G. Gilboa, “A spectral approach to total variation”, Scale-Space and Variational Methods, SSVM 2013, LNCS 7893, p. 36-47, 2013. Abstract: The total variation (TV) functional is explored from a spectral perspective. We formulate a TV transform based on the second time derivative of the total variation flow, scaled by time. In the transformation domain disks yield impulse […]
Expert regularizers for task specific processing
G. Gilboa, “Expert regularizers for task specific processing”, Scale-Space and Variational Methods, SSVM 2013, LNCS 7893, p. 24-35, 2013 Abstract: This study is concerned with constructing expert regularizers for specific tasks. We discuss the general problem of what is desired from a regularizer, when one knows the type of images to be processed. The aim is to improve […]
Nonlocal linear image regularization and supervised segmentation
G. Gilboa, S. Osher, “Nonlocal linear image regularization and supervised segmentation”, SIAM Multiscale Mod. Simul. (MMS), Vol. 6, No. 2, pp. 595-630, 2007. Abstract A nonlocal quadratic functional of weighted differences is examined. The weights are based on image features and represent the affinity between different pixels in the image. By prescribing different formulas for the weights, one […]
Nonlinear Scale Space with Spatially Varying Stopping Time
G. Gilboa, “Nonlinear Scale Space with Spatially Varying Stopping Time”, PAMI, Vol. 30, No. 12, pp. 2175-2187, 2008. Abstract: A general scale space algorithm is presented for denoising signals and images with spatially varying dominant scales. The process is formulated as a partial differential equation with spatially varying time. The proposed adaptivity is semi-local and is in conjunction with the […]
Estimation of optimal PDE-based denoising in the SNR sense
G. Gilboa, N. Sochen, Y.Y. Zeevi, “Estimation of optimal PDE-based denoising in the SNR sense”, IEEE Trans. on Image Processing Vol. 15, No. 8, pp. 2269-2280, 2006. Abstract: This paper is concerned with finding the best PDE-based denoising process, out of a set of possible ones. We focus either on finding the proper weight of the fidelity term […]
Nonlocal Convex Functionals for Image Regularization
G. Gilboa, J. Darbon, S. Osher and T. Chan, Nonlocal Convex Functionals for Image Regularization. UCLA CAM Report 06-57 Abstract: We examine weighted nonlocal convex functionals. The weights determine the affinities between different regions in the image and are computed according to image features. The L1 energy of this type can be viewed as a nonlocal extension of […]