nemo
/
hn-classics
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
hn-classics/_stories/2009/11323177.md

223 lines
13 KiB
Markdown
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

[Source](http://gl.ict.usc.edu/Research/3DTeleconferencing/ "Permalink to Achieving Eye Contact in a One-to-Many 3D Video Teleconferencing System ")
# Achieving Eye Contact in a One-to-Many 3D Video Teleconferencing System
Achieving Eye Contact in a One-to-Many 3D Video Teleconferencing System
![][1]
![][2]
* ![][1]
* [RESEARCH][3]
* [PUBLICATIONS][4]
* [SOFTWARE & DATA][5]
* [PRODUCTION][6]
* [PEOPLE][7]
* [JOBS][8]
###### Achieving Eye Contact in a One-to-Many 3D Video Teleconferencing System
_SIGGRAPH 2009 Emerging Technologies_
_SIGGRAPH Papers 2009 Conference Proceedings_
##### Andrew Jones    Magnus Lang    Graham Fyffe    Xueming Yu    Jay Busch   
Ian McDowall*    Mark Bolas**    Paul Debevec   
####### USC Institute for Creative Technologies    *Fakespace Labs    **USC ICT and USC School of Cinematic Arts   
######## Introduction
During face-to-face conversation, eye contact, and gaze direction provide important visual cues to express emotion, attention, and interest. Unfortunately, 2D video telecoferencing does not recreate accurate eye contact breaking the emotion connecting between participants. When a remote participant looks directly into the camera, everyone watching the video stream sees the same image of the participant looking towards them; if the participant looks away from the camera, no one receives the eye gaze. Our one-to-many teleconferencing system uses a novel arrangement of 3D acquisition, transmission, and display technologies to achieve accurate reproduction of gaze direction and eye contact.
######## 3D Display
![][9] ![][10]
The 3D display works by projecting high-speed video onto a rapidly spinning mirror made from brushed aluminum. As the mirror turns, it reflects a different and accurate image to each potental viewer. The size, geometry, and material of the spinning surface have been optimized for the display of a life-sized human face. Its two-sided shape provides two passes of a display surface to each viewer per full rotation, achieving a 30Hz visual update at 900 rpm. A pair of high speed DLP projectors project 8,640 1-bit (black or white) frames per second using a specially coded DVI video signal. Instead of rendering a color image,each projector takes a 24-bit color frame of video and displays each bit sequentially as seperate frames. Effectively, the mirror reflects 144 unique views of the scene across a 180-degree field of view with an angular view separation of 1.25 degrees.
######## 3D Scanning
![][11]
A 90 degree field of view 2D video feed allows the remote participant to view the audience interacting with their three-dimensional image on the 3D display. A polarized beam-splitter is used to virtually place the camera close to the position of the eyes of the three-dimensional head. The video from the aligned 3D display camera is transmitted to the remote participant where it is shown on a geometrically calibrated LCD screen.
![][12]
To correct the vertical perspective on the 3D display, we use marker-less face detection from OpenCV to track viewers based on the 2D video feed. In this way, the display's horizontal parallax provides binocular stereo with no lag as the viewers move their heads horizontally, while vertical parallax is achieved through tracking.
![][13]
The ICT Graphics Laboratory graciously acknowledges the hardware support of nVIDIA Quadro FX 5800 cards provided by for this project, achieving our rendering speeds of 8,640 frames per second.
######## Downloads
**SIGGRAPH 2009 Paper and Video**
[3DTeleconferencing-USCICT-SIGGRAPH2009.pdf][14] (9.8 MB)
[3DTeleSIGGRAPH09_864x486_H264.mov][15] (265 MB)
**Live 3D Teleconferencing Demo Video - Dec 1, 2008**
[3DTeleconferencing-USC-ICT-20081201.mov][16] (120MB)
**Technology Explanation Video**
[3DTeleconferencing-USC-ICT-20081201b.mov][17] (261MB)
**IEEE Spectrum's Sally Adee tries out the system**
[3DTeleconferencing-USC-ICT-20081201c.mov][18] (175MB)
**SIGGRAPH 2009**
![][19] ![][20] ![][21] ![][22] ![][23]
![][24] ![][25] ![][26] ![][27] ![][28]
![][29] ![][30] ![][31] ![][32] ![][33]
![][34] ![][35] ![][36] ![][37]
**3D Teleconferencing Images**
![][38] ![][39] ![][40] ![][41] ![][42]
![][43] ![][44] ![][45]
![][46] ![][47] ![][48] ![][49] ![][50]
![][51] ![][52] ![][53] ![][54] ![][55]
![][56]
######## Frequently Asked Questions
**How does the display work?**
_ We project high-speed (4,320 frames per second) video of different views of the face onto a spinning mirror. As the mirror spins, the different views are reflected to all different viewpoints around the display. In our teleconferencing system, seventy-two views of the face rendered by a PC with a modern graphics card are projected across 180 degrees field of view. By the time the mirror rotates to reflect images from one eye position to another, a different view of the scene is projected, so the result appears to be three-dimensional. _
**Is this a hologram?**
_ It's similar to the "Holograms" in the Star Wars movies in that it's a three-dimensional animated image of a person floating in space, and the displayed objects can have the general appearance of solidity rather than appearing as illuminated volumes of mist. However, no holographic film or traditional holography is involved to achieve the three-dimensional effect. _
**Could the display be in color?**
_ Yes, we'd just need to modify a three-chip DLP video projector to achieve high speed video projection instead of the single-chip projector we've used in our prototype. We could also use multiple single-chip projectors to achieve the effect. Or, as in [the previous iteration][57], we achieved color with a multicolored display surface. _
**Is this like CNN's "Hologram" they showed during their 2008 presidential election coverage?**
_ No, our system produces a real 3D image you can actually see whereas CNN's "hologram" was just a visual effect. To the home viewer, CNN's anchor Wolf Blitzer appeared to be looking at three-dimensional images of guests Jessica Yellin and Will.i.am, but this was a video overlay effect done for the home audience with no actual 3D display technology involved. In CNN's studio, Blitzer [ was actually staring across empty space][58] toward a standard 2D flat panel television. _
**What is the spinning surface you're projecting onto?**
_ It's a steep tent shape formed by two panels of 8-by-10 inch brushed aluminum - the same material found on many kitchen appliances. We use this material since it is inexpensive and spreads light vertically but not horizontally, which allows people of any height to see the image formed on the display. It rotates fifteen times per second, reflecting thirty frames per second of 3D video from the two sides of the tent. _
**Could you display a whole body?**
_ Yes, though to display a person life size we would need a larger spinning mirror to project onto. At the end of our first 3D Display project video, we showed a five-inch-tall 3D image of Bruce (one of our group's researchers) realistically running in place. _
**Why are you doing this? What is the benefit of a 3D teleconference?**
_ We are hoping to better recreate the natural, rich communication that people experience in person-to-person conversations, especially the effects of gaze, attention, and eye contact between a speaker and one or more other people. These effects are not reproduced in conventional 2D videoconferences: if the speaker looks at the camera, they appear to be looking at everyone; if they look away, they are looking at no one. With our system, the speaker can look at any of the people he or she is speaking to, and all participants see a natural view of the speaker. _
######## Media Coverage
**Forbes**
[**E-Gang '09: The Web's New Video Stars**][59] edited by Kerry A. Dolan and Elizabeth Corcoran, August 05, 2009
[**E-Gang: Beaming Up Holograms**][60]** Shorter Version**
[**E-Gang: Beaming Up Holograms**][61]** With Supplemented Video**
**Forbes Video Network Kym McNicholas Video**
**Laser Focus World**
**Audience interaction with 'hologram' enabled by spinning-mirror system** February 01, 2009
**Researchers Demo Spinning Mirror-Based Interactive Volumetric Display System** by Barbara G. Goode, December 22, 2008
**Investor's Business Daily**
**New Vision Gaining As Holograms Debut In Business And TV** by Julie Vallone, February 13, 2009
**Telepresence Options**
[**3D Telepresence is Near**][62] by John Serrao, December 17, 2008
**Computer Government News**
**Institute Unveils 3-D Videoconferencing Prototype** by David F. Carr, December 8, 2008
**The Wall Street Journal**
[**The Way We'll Watch**][63] by Sarah McBride, December 8, 2008
**Tech Companies Pursue 3D Technology** December 8, 2008
**The Chronicle of Higher Education**
**The Wired Campus: U. of Southern California Professor Unveils Holographic Videoconferencing System** by Jeffrey R. Young, December 3, 2008
**USC News**
**New Dimension Added to Meetings** by Carl Marziali, December 3, 2008
**IEEE Spectrum Online**
**Star Wars-style hologram conferencing at the Army Science Conference** by Sally Adee, December 3, 2008
**Jump Into Tomorrow**
**Thank You Obi-Wan Kenobi** December 1, 2008
| ----- |
| © 2008-2017 [ USC ][64] [ Institute for Creative Technologies ][65] | [ ![][66] ][64] |
[1]: http://vgl.ict.usc.edu/WebPageSource/images/vgl.png
[2]: http://vgl.ict.usc.edu/WebPageSource/images/burger2.png
[3]: http://vgl.ict.usc.edu/research_HD.php
[4]: http://vgl.ict.usc.edu/publications.php
[5]: http://vgl.ict.usc.edu/softwaredata.php
[6]: http://vgl.ict.usc.edu/production.php
[7]: http://vgl.ict.usc.edu/people.php
[8]: http://vgl.ict.usc.edu/jobs.php
[9]: http://gl.ict.usc.edu/images/sm_mirror.jpg
[10]: http://gl.ict.usc.edu/images/sm_Cynthia.jpg
[11]: http://gl.ict.usc.edu/images/sm_patterns.jpg
[12]: http://gl.ict.usc.edu/images/sm_tracking.jpg
[13]: http://gl.ict.usc.edu/NVIDIAlogo.png
[14]: http://gl.ict.usc.edu/3DTeleconferencing-USCICT-SIGGRAPH2009.pdf
[15]: http://gl.ict.usc.edu/3DTeleSIGGRAPH09_864x486_H264.mov
[16]: http://gl.ict.usc.edu/3DTeleconferencing-USC-ICT-20081201.mov
[17]: http://gl.ict.usc.edu/3DTeleconferencing-USC-ICT-20081201b.mov
[18]: http://gl.ict.usc.edu/3DTeleconferencing-USC-ICT-20081201c.mov
[19]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9567_sm.jpg
[20]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9571_sm.jpg
[21]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9582_sm.jpg
[22]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9586_sm.jpg
[23]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9599_sm.jpg
[24]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9600_sm.jpg
[25]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9614_sm.jpg
[26]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9616_sm.jpg
[27]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9617_sm.jpg
[28]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9620_sm.jpg
[29]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9657_sm.jpg
[30]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9672_sm.jpg
[31]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9673_sm.jpg
[32]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9674_sm.jpg
[33]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9682_sm.jpg
[34]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9687_sm.jpg
[35]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9690_sm.jpg
[36]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9697_sm.jpg
[37]: http://gl.ict.usc.edu/images/SIGGRAPH2009/IMG_9706_v02_sm.jpg
[38]: http://gl.ict.usc.edu/images/13_sm.jpg
[39]: http://gl.ict.usc.edu/images/14_sm.jpg
[40]: http://gl.ict.usc.edu/images/23_sm.jpg
[41]: http://gl.ict.usc.edu/images/18_sm.jpg
[42]: http://gl.ict.usc.edu/images/17_sm.jpg
[43]: http://gl.ict.usc.edu/images/20_sm.jpg
[44]: http://gl.ict.usc.edu/images/26_sm.jpg
[45]: http://gl.ict.usc.edu/images/27_sm.jpg
[46]: http://gl.ict.usc.edu/images/high/sm_IMG_0709.jpg
[47]: http://gl.ict.usc.edu/images/high/sm_IMG_0655.jpg
[48]: http://gl.ict.usc.edu/images/high/sm_IMG_0679.jpg
[49]: http://gl.ict.usc.edu/images/high/sm_IMG_2406.jpg
[50]: http://gl.ict.usc.edu/images/high/sm_IMG_2363.jpg
[51]: http://gl.ict.usc.edu/images/high/sm_IMG_2390.jpg
[52]: http://gl.ict.usc.edu/images/high/sm_IMG_2414.jpg
[53]: http://gl.ict.usc.edu/images/high/sm_IMG_2421.jpg
[54]: http://gl.ict.usc.edu/images/high/sm_IMG_2425.jpg
[55]: http://gl.ict.usc.edu/images/high/sm_IMG_2435.jpg
[56]: http://gl.ict.usc.edu/images/high/sm_IMG_2439.jpg
[57]: http://gl.ict.usc.edu/Research/3DDisplay/
[58]: http://www.vizrt.com/casestudies/35438/Vizrts_Hologram_greets_U.S._Election_2008
[59]: https://www.forbes.com/2009/08/05/online-video-innovators-technology-breakthroughs-e-gang-09_land.html
[60]: https://www.forbes.com/forbes/2009/0824/egang-09-teleconferencing-3d-beaming-up-holograms.html
[61]: https://www.forbes.com/2009/08/06/online-video-innovation-technology-e-gang-09-usc.html
[62]: http://www.telepresenceoptions.com/2008/12/3d_telepresence_is_near/
[63]: https://www.wsj.com/articles/SB122833913230576869
[64]: http://www.usc.edu/
[65]: http://ict.usc.edu/
[66]: http://vgl.ict.usc.edu/WebPageSource/images/usc.png
Reference in New Issue View Git Blame Copy Permalink