Category: Uncategorized

Total-Variation – Fast Gradient Flow and Relations to Koopman Theory

Ido Cohen, Tom Berkov,  arXiv preprint 

Code

Abstract

The space-discrete Total Variation (TV) flow is analyzed using several mode decomposition techniques. In the one-dimensional case, we provide analytic formulations to Dynamic Mode Decomposition (DMD) and to Koopman Mode Decomposition (KMD) of the TV-flow and compare the obtained modes to TV spectral decomposition. We propose a computationally efficient algorithm to evolve the one-dimensional TV-flow. A significant speedup by three orders of magnitude is obtained, compared to iterative minimizations. A common theme, for both mode analysis and fast algorithm, is the significance of phase transitions during the flow, in which the subgradient changes. We explain why applying DMD directly on TV-flow measurements cannot model the flow or extract modes well. We formulate a more general method for mode decomposition that coincides with the modes of KMD. This method is based on the linear decay profile, typical to TV-flow. These concepts are demonstrated through experiments, where additional extensions to the two-dimensional case are given.

photograph

TV Transform & Nonlinear Spectral Theory

A new framework is proposed for variational analysis and processing. It defines a functional-based nonlinear transform and inverse-transform. The framework is developed in the context of total-variation (TV), but it can be generalized to other functionals.

An eigenfunction, with respect to the subdifferential of the functional, such as a disk in the TV case, yields an impulse in the transform domain. This can be viewed as a generalization of known spectral approaches, based on linear algebra, which are extensively used in image-processing, e.g. for segmentation.

Following the Fourier intuition, a spectrum can be computed to analyze dominant scales in the image. Moreover, new nonlinear low-pass, high-pass and band-pass filters can be designed with full contrast and edge preservation.

spectv_logo2b

Movie explained: Top (from left): image layers, phi bands (phi(t) = u_{tt}*t). Bottom: Ideal TV High-pass-filter, ideal TV Low-pass-filter at different scales. Ideal is in the sense of eigenfunctoin preservation.

Related papers

  1. R. Nossek, G. Gilboa, “Flows generating nonlinear eigenfunctions”, accepted to J. of Scientific Computing, 2017.
  2. G. Gilboa, “Semi-inner-products for convex functionals and their use in image decomposition”, Journal of Mathematical Imaging and Vision (JMIV), Vol. 57, No. 1, pp. 26-42, 2017.
  3. M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), 2017.
  4. M. Benning, G. Gilboa, J.S. Grah, C. Bibiane Schönlieb, “Learning Filter Functions in Regularisers by Minimising Quotients.” In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), pp. 511-523. Springer, 2017.
  5. M. Burger, G. Gilboa, M. Moeller, L. Eckardt, D. Cremers, “Spectral decompositions using one-homogeneous functionals”, SIAM J. Imaging Sciences, Vol. 9, No. 3, pp. 1374-1408, 2016.
  6. M. Benning, G. Gilboa, C.B. Schönlieb, “ Learning parametrised regularisation functions via quotient minimisation”. PAMM-16, 16(1), 933-936, 2016.
  7. G. Gilboa, M. Moeller, M. Burger, “Nonlinear spectral analysis via one-homogeneous functionals – overview and future prospects”, accepted to the Journal of Mathematical Imaging and Vision (JMIV), Vol. 56, No. 2, pp 300–319, 2016.
  8. D. Horesh, G. Gilboa, “Separating surfaces for structure-texture decomposition using the TV transform”, IEEE Trans. Image Processing, Vol. 25, No. 9, pp. 4260 – 4270, 2016.
  9. G. Gilboa, “A total variation spectral framework for scale and texture analysis”. SIAM Journal on Imaging Sciences, Vol. 7, No. 4, pp. 1937–1961,  2014.
  10. G. Gilboa, “Nonlinear Band-Pass Filtering Using the TV Transform”, Proc. European Signal Processing Conference (EUSIPCO-2014), Lisbon, p. 1696 – 1700, 2014.
  11. G. Gilboa, “A spectral approach to total variation”, Scale-Space and Variational Methods”, SSVM 2013, LNCS 7893, p. 36-47, 2013.
  12. G. Gilboa, CCIT Report 833, Technion, 2013.

photograph

Sharpening and Enhancement

Main topics

  • Image sharpening using forward-and-backward diffusion (simultaneous sharpening and denoising) – enhancing edges while keeping low noise.
  • Complex diffusion processes – using the imaginary part which imarges to be a stable nonlinear edge detector to guide the diffusion process.
  • Inverse scale space –  a generalization of Bregman iterations from a variational to a new non-standard PDE formulation (with Burger, Osher and Xu).

Complex diffusion real and imaginary kernels

Complex_diff_kernel

Relaxed inverse scale space

relaxed_iss_initial_conditions

Related papers

  1. E. Hait, G. Gilboa, “Blind facial image quality enhancement using non-rigid semantic patches.” IEEE Transactions on Image Processing Vol. 26, No. 6, pp. 2705-2720, 2017.
  2. M Benning, M. Moeller, R. Nossek, M. Burger, D. Cremers, G. Gilboa, C. Schoenlieb, “Nonlinear Spectral Image Fusion”, In International Conference on Scale Space and Variational Methods in Computer Vision (SSVM), 2017.
  3. O. Spier, T. Treibitz, G. Gilboa, “In situ target-less calibration of turbid media”, Int. Conf. on Computational Photography (ICCP), Stanford Univ., 2017.
  4. D. Horesh, G. Gilboa, “Separating surfaces for structure-texture decomposition using the TV transform”, IEEE Trans. Image Processing, Vol. 25, No. 9, pp. 4260 – 4270, 2016.
  5. O. Katzir, G. Gilboa, “A Maximal Interest-Point Strategy Applied to Image Enhancement with External Priors”, Proc. IEEE Int. Conf. Image Processing (ICIP), 2015.
  6. G. Gilboa, “Nonlinear Scale Space with Spatially Varying Stopping Time”, PAMI, Vol. 30, No. 12, pp. 2175-2187, 2008.
  7. M. Welk, G. Gilboa, J. Weickert, “Theoretical foundations for discrete forward-and-backward diffusion filtering”. SSVM 2009, pp. 527-538, 2009.
  8. M. Burger, G. Gilboa, S. Osher, J. Xu, , “Nonlinear inverse scale space methods”, Communications in Mathematical Sciences (CMS) Vol 4, No.1, pp. 179-212, 2006.
  9. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image sharpening by flows based on triple well potentials”,  J. of Math. Imaging and Vision, 20:121-131, 2004.
  10. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image enhancement and denoising by complex diffusion processes”, IEEE Trans. on Pattern Analysis and Machine Intelligence (PAMI), Vol. 26, No. 8, pp. 1020-1036, 2004.
  11. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Forward-and-Backward diffusion processes for adaptive image enhancement and denoising”, IEEE Trans. on Image Processing, Vol. 11, No. 7, pp. 689-703, 2002.
  12. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Regularized shock filters and complex diffusion”,  – ECCV-’02, LNCS 2350, pp. 399-313, Springer-Verlag 2002.
  13. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Image enhancement segmentation and denoising by time dependent nonlinear diffusion processes”, Proc. IEEE ICIP-’01, Thessaloniki, Greece, vol. 3, pp. 134-137, 2001.
  14. M. Burger, S. Osher, J. Xu, G. Gilboa, “Nonlinear Inverse Scale Space Methods for Image Restoration”, Variational and Level-Set Methods (VLSM) 2005,  LNCS 3752, pp. 25-36, Springer-Verlag, 2005.
  15. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Complex diffusion processes for image filtering”, Scale-Space ’01, LNCS 2106, pp. 299-307, Springer-Verlag 2001.
  16. G. Gilboa, N. Sochen, Y.Y. Zeevi, “Resolution enhancement by forward-and-backward nonlinear diffusion processes”, Nonlinear Signal and Image Processing, Baltimore, Maryland, June 2001.
  17. N. Sochen, G. Gilboa, Y.Y. Zeevi, “Color image enhancement by a forward-and-backward adaptive Beltrami flow”, AFPAC-2000, LNCS 1888, pp. 319-328, 2000, Springer-Verlag.
  18. G. Gilboa, Y.Y. Zeevi, N. Sochen, “Signal and image enhancement by a generalized forward-and-backward adaptive diffusion process”, EUSIPCO-2000, Tampere, Finland, Sept. 2000.
  19. G. Gilboa, Y.Y. Zeevi, N. Sochen, “Anisotropic selective inverse diffusion for signal enhancement in the presence of noise”, Proc. IEEE ICASSP-2000, Istanbul, Turkey, vol. I, pp. 211-224, June 2000.
imeg

Nonlocal Operators in Image Processing

Processing image in a nonlocal manner using a variational approach. Nonlocal operators such as gradient and divergence are defined and used to form nonlocal functionals, such as nonlocal total variation (NLTV). Also evolution processes can be used (e.g. for denoising), for instance nonlocal diffusion. Segmentation, NL-TV-L1 and NL-TV-G minimizations are examined, as well as extensions of Chambolle’s projection algorithm to solve numerically the minimization problems.

Related papers (see PDF’s in publicaton part)

  1. J.-F. Aujol, G. Gilboa, N. Papadakis, “Non local total variation spectral framework”, Scale Space and Variational Methods in Computer  Vision, SSVM 2015, LNCS 9087, p. 66-77, 2015.
  2. G. Gilboa, E. Appleboim, E. Saucan, Y.Y. Zeevi, “On the Role of Non-local Menger Curvature in Image Processing”, Proc. IEEE Int. Conf. Image Processing (ICIP), pp. 4337-4341, 2015. Recognized as part of the Top 10% papers in ICIP 2015.
  3. G. Gilboa, S. Osher, “Nonlocal Operators with Applications to Image Processing”, SIAM Multiscale Mod. Simul. (MMS), Vol. 7, No. 3, pp. 1005-1028, 2008.
  4. G. Gilboa, S. Osher, “Nonlocal linear image regularization and supervised segmentation”, SIAM Multiscale Mod. Simul. (MMS), Vol. 6, No. 2, pp. 595-630, 2007.
  5. G. Gilboa, S. Osher,  “Nonlocal evolutions for image regularization”, Proc. SPIE Electronic Imaging, SPIE Vol. 6498, 64980U, 2007.
  6. G. Gilboa, J. Darbon, S. Osher and T. Chan, “Nonlocal Convex Functionals for Image Regularization“, UCLA CAM Report 06-57.
imeg

Depth Cameras

Previous related publications (mostly patents):

  1. G. Drozdov, Y. Shapiro, G. Gilboa, “Robust recovery of heavily degraded depth measurements”, Int. Conf. On 3D Vision (3DV), Stanford Univ., 2016.
  2. D. Rotman, G. Gilboa, “A depth restoration occlusionless temporal dataset”, Int. Conf. On 3D Vision (3DV), Stanford Univ., 2016.
  3. D.N. Rotman, O. Cohen, G. Gilboa, “Frame Rate Reduction of Depth Cameras by RGB-Based Depth Prediction”, CCIT Report 867, 2014.
  4. G. Gilboa, D. Cohen, G. Yahav, “Ambient light alert for an image sensor”, US 2013/0208091, 2013.
  5. G. Gilboa, “Learning from high quality depth measurements”, US 2012/0249738, 2012.
  6.  G. Gilboa, A. Adler, S. Katz, “Depth camera compatibility” – part I, US 2011/0187820, 2011.
  7.  S. Katz, A. Adler, G. Gilboa, “Depth camera compatibility” – part II, US 2011/0187819, 2011.
  8.  A. Adler, S. Katz, G. Gilboa, J. Tardif, “De-aliasing depth images”, US 2011/0234756, 2011.
  9.  G. Gilboa, D. Cohen, G. Yahav, E. Larry, S. Felzenshtein, “Adaptive high dynamic range camera”, US 2012/0287242, 2012.