Many seemingly static scenes contain subtle changes that are invisible to the naked human eye. However, it is possible to pull out these small changes from videos through the use of algorithms we have developed. We give a way to visualize these small changes by amplifying them and we present algorithms to pull out interesting signals from these videos, such as the human pulse, sound from vibrating objects and the motion of hot air.

Software and Code

Eulerian Video Magnification code Matlab code and executables implementing Eulerian video processing for amplifying color and motion changes. Phase Based Video Motion Processing code Matlab code implementing the new and improved phase-based motion magnification pipeline. Learning-based Video Motion Magnification code Tensorflow implementation of the learning-based motion magnification pipeline. Videoscope Web interface for motion and color magnification. Upload your videos and have them magnified!

Publications (Magnifying Motion and Color Changes)

Tae-Hyun Oh*, Ronnachai Jaroensri*, Changil Kim, Mohamed Elgharib, Frédo Durand, William T. Freeman, Wojciech Matusik Learning-based Video Motion Magnification European Conference on Computer Vision (ECCV), 2018 [Paper] [Webpage] A learning-based approach to motion magnification with reduced artifact and better noise handling.

Mohamed A. Elgharib, Mohamed Hefeeda, Frédo Durand, William T. Freeman Video Magnification in Presence of Large Motions IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015 [Paper] [Webpage] A new video technique to motion magnify interesting small motions that are combined with large motions.

Frédo Durand, William T. Freeman, Michael Rubinstein A World of Movement Scientific American, Volume 312, Number 1, January 2015 [Article in SciAm] [Videos] An expository article describing several of the motion magnification techniques and applications we have worked on.

Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman Riesz Pyramids for Fast Phase-Based Video Magnification Computational Photography (ICCP), 2014 IEEE International Conference on [Paper] [Webpage] Provides the quality of the previous phase-based technique with the real-time speed of the original linear technique.

Neal Wadhwa, Michael Rubinstein, Frédo Durand, William T. Freeman Phase-based Video Motion Processing ACM Transactions on Graphics, Volume 32, Number 4 (Proc. SIGGRAPH), 2013. [Paper] [Webpage] [BibTeX] A new technique to amplify small motions that solves the noise amplification and intensity clipping artifacts of the previous linear method by manipulating the phase in sub-bands of videos.

Michael Rubinstein, Neal Wadhwa, Frédo Durand, William T. Freeman Revealing Invisible Changes In The World Science Vol. 339 No. 6119, Feb 1 2013 [Article in Science] [Video] [NSF SciVis 2012] [BibTeX] An expository video showcasing our results and explaining our technique.

Hao-Yu Wu, Michael Rubinstein, Eugene Shih, John Guttag, Frédo Durand, William T. Freeman Eulerian Video Magnification for Revealing Subtle Changes in the World ACM Transactions on Graphics, Volume 31, Number 4 (Proc. SIGGRAPH), 2012 [Paper] [Webpage] [BibTeX] The first Eulerian method to amplify small motions and color variations in videos.

Ce Liu, Antonio Torralba, William T. Freeman, Frédo Durand, Edward H. Adelson Motion Magnification ACM Transactions on Graphics, Volume 24, Number 3 (Proc. SIGGRAPH), 2005 [Paper] [Webpage] The original Lagrangian method to amplify small motions by explicitly estimating them and then warping the frame by amplified motion amounts.

Publications (Analysis of Small Motions)

Abe Davis, Katherine L. Bouman, Justin G. Chen, Michael Rubinstein, Frédo Durand, William T. Freeman Visual Vibrometry: Estimating Material Properties from Small Motions in Video IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2015 [Paper] [Webpage] A method to estimate the material properties of from small motions in videos.

Tianfan Xue, Michael Rubinstein, Neal Wadhwa, Anat Levin, Frédo Durand, William T. Freeman Refraction Wiggles for Measuring Fluid Depth and Velocity from Video Proc. of European Conference on Computer Vision (ECCV), 2014. [Paper] [Webpage] A method to recover the velocity and depth of hot air from video.

Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham Mysore, Frédo Durand, William T. Freeman The Visual Microphone: Passive Recovery of Sound from Video ACM Transactions on Graphics, Volume 33, Number 4 (Proc. SIGGRAPH), 2014. [Paper] [Webpage] A technique to recover sound from videos of objects subtly vibrating in response to sound.

Justin G. Chen, Neal Wadhwa, Young-Jin Cha, Frédo Durand, William T. Freeman, Oral Buyukozturk Structural Modal Identification through High Speed Camera Video: Motion Magnification Proceedings of the 32nd International Modal Analysis Conference (2014) [Paper] A validation that the motion magnified motions are indeed real and a way to compute the mode shapes of a cantilevered beam from video.

Guha Balakrishnan, Frédo Durand, John Guttag Detecting Pulse from Head Motions in Video Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on [Paper] [Video] A method to measure pulse from the Newtonian motion of the head as blood flows into it.

People

Faculty:

Students, Postdocs and Affiliates:

Related Work and Other Applications

Visible Imaging of Global MHD on MAST Ryan, D. A. Plasma Science, IEEE Transactions on , vol.42, no.10, pp.2556,2557, Oct. 2014 [Paper] Video magnification applied to plasma physics.

Auto Localization and Segmentation of Occluded Vessels in Robot-Assisted Partial Nephrectomy. Amir-Khalili, A., Peyrat, J. M., Abinahed, J., Al-Alao, O., Al-Ansari, A., Hamarneh, G., and Abugharbieh, R. In Medical Image Computing and Computer-Assisted Intervention鈥揗ICCAI 2014 (pp. 407-414). Springer International Publishing. [Paper] The temporal frequency of subtle vibrations is used to segment blood vessels in medical imagery.

Talks

A Big World of Small Motions, Michael Rubinstein Invited talk at 2013 International Conference in Computational Photography (April 20, 2013) Invited talk at TEDx @ Beacon Street 2014 (November 15, 2014)

Revealing What's Hidden in Videos, William T. Freeman Plenary Talk at 2014 International Conference on Image Processing (October 29, 2014)