[payload] Velocity Vector Fields, Computer Vision and Digital Video

[Adam Sobieski <[e-mail]>]

Internet Engineering Task Force,
Payload Working Group,
 
Greetings.  A digital video technology topic is that challenges in motion estimation have included that motion was three-dimensional while video images were projections of 3D motion onto two-dimensional bitmaps.  Velocity and digital video are interrelated topics and exciting research, applicable to both computer vision and digital video topics, is underway with regard to real-time 3D velocity vector fields with Kinect sensors.
 
An ICCV 2011 paper, Kinecting the Dots: Particle Based Scene Flow from Depth Sensors, by Simon Hadfield (http://personal.ee.surrey.ac.uk/Personal/R.Bowden/publications/2011/ICCV/Hadfield_ICCV11.pdf, http://www.youtube.com/watch?v=10CWOZcGMv0) illustrates velocity vector fields from Kinect data and describes a CPU implementation while a parallel GPU implementation with real-time performance is anticipated.
 
With regard to 3D video, current technologies include MPEG multiview coding (http://www.ietf.org/id/draft-ietf-payload-rtp-mvc-02.txt), free viewpoint video, and 2D-plus-depth video, resembling the data from RGBD sensors, and, in addition to previous techniques for super-resolution, both the depth stream data and computed data, such as surface geometry and velocity vector fields, can be of use to heuristics for super-resolving video streams. 2D-Plus-Depth Based Resolution and Frame-rate Up-conversion Technique for Depth Video by Jinwook Choi, Dongbo Min, and Kwanghoon Sohn discusses some related super-resolution topics.
 
Interesting scenarios include: video calls, video conferences, video blogs and video forums. With the super-resolution of streams from RGBD sensors, streaming and storage efficiencies can be enhanced.  In addition to 2D-plus-depth, stereoscopic 3D video, and MPEG MVC, also possible are: 3D video streaming, video calling and conferencing, interoperable with head tracking, 2D+Velocity or RGBV video, and 3D video formats based upon, instead of sequences of bitmaps, sequences of point clouds or 3D surfaces.
 
With regard to the numerical precision of depth measurements and that of velocity and acceleration vector fields, functions to obtain 3D probability distributions around each 3D point are of interest. Accuracy Analysis of Kinect Depth Data by K. Khoshelham discusses the numerical precision of Kinect sensor depth measurements. 3D with Kinect by Jan Smisek, Michal Jancosek and Tomas Pajdla also discusses the calibration of and accuracy analysis of Kinect sensors as well as structure from motion.  Accuracy analysis of depth measurements and of the timestamps of the depth and RGB data streams can be of use to numerical algorithms for precisely obtaining and utilizing depth measurements and velocity and acceleration vector fields.
 
Additionally, towards combined uses of depth and RGB data, where depth data can super-resolve RGB data and RGB data can super-resolve depth data, some approaches include edge detection, silhouettes, visual hull construction, and topics discussed in Silhouette-based 3D Model Reconstruction from Multiple Images by Adem Yasar Mulayim, Ulas Yılmaz, and Volkan Atalay, and Shape-From-Silhouette of Articulated Objects and its Use for Human Body Kinematics Estimation and Motion Capture by German K. M. Cheung, Simon Baker, and Takeo Kanade.  Shape from shading is discussed in Shape from Shading: A Survey by Ruo Zhang, Ping-Sing Tsai, James Edwin Cryer, and Mubarak Shah and Recovering Surface Reflectance and Multiple Light Locations and Intensities from Image Data by S. Xu and A. M. Wallace.



Kind regards,
 
Adam Sobieski