Skip to main content
SpringerLink
Log in

Lighting virtual objects in a single image via coarse scene understanding

  • Research Paper
  • Published:
Science China Information Sciences Aims and scope Submit manuscript

Abstract

Achieving convincing visual consistency between virtual objects and a real scene mainly relies on the lighting effects of virtual-real composition scenes. The problem becomes more challenging in lighting virtual objects in a single real image. Recently, scene understanding from a single image has made great progress. The estimated geometry, semantic labels and intrinsic components provide mostly coarse information, and are not accurate enough to re-render the whole scene. However, carefully integrating the estimated coarse information can lead to an estimate of the illumination parameters of the real scene. We present a novel method that uses the coarse information estimated by current scene understanding technology to estimate the parameters of a ray-based illumination model to light virtual objects in a real scene. Our key idea is to estimate the illumination via a sparse set of small 3D surfaces using normal and semantic constraints. The coarse shading image obtained by intrinsic image decomposition is considered as the irradiance of the selected small surfaces. The virtual objects are illuminated by the estimated illumination parameters. Experimental results show that our method can convincingly light virtual objects in a single real image, without any pre-recorded 3D geometry, reflectance, illumination acquisition equipment or imaging information of the image.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Madsen C B, Nielsen M. Towards probe-less augmented reality: a position paper. In: Proceedings of the International Conference on Computer Graphics Theory and Applications, Madeira-Funchal, 2008. 255–261

    Google Scholar 

  2. Lalonde J F, Efros A, Narasimhan S. Estimating natural illumination from a single outdoor image. In: Proceedings of the IEEE International Conference on Computer Vision, Kyoto, 2009. 183–190

    Google Scholar 

  3. Liu Y L, Qin X Y, Xu S H, et al. Light source estimation of outdoor scenes for mixed reality. Vis Comput, 2009, 25: 637–646

    Article  Google Scholar 

  4. Liu Y L, Qin X Y, Xing G Y, et al. A new approach to outdoor illumination estimation based on statistical analysis for augmented reality. J Vis Comput Animat, 2010, 21: 321–330

    Google Scholar 

  5. Chen X W, Wang K, Jin X. Single image based illumination estimation for lighting virtual object in real scene. In: Proceedings of the International Conference on Computer-Aided Design and Computer Graphics, Jinan, 2011. 450–455

    Google Scholar 

  6. Xing G Y, Liu Y L, Qin X Y, et al. On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In: Proceedings of the International Conference on Computer-Aided Design and Computer Graphics, Jinan, 2011. 439–442

    Google Scholar 

  7. Liu Y L, Granier X. Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans Vis Comput Graph, 2012, 18: 573–580

    Article  Google Scholar 

  8. Marschner S R, Greenberg D P. Inverse lighting for photography. In: Proceedings of the Color Imaging Conference: Color Science, Systems and Applications, Scottsdale, 1997. 262–265

    Google Scholar 

  9. Debevec P. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, New York, 1998. 189–198

    Google Scholar 

  10. Mei X, Ling H B, Jacobs D W. Illumination recovery from image with cast shadows via sparse representation. IEEE Trans Image Process, 2011, 20: 2366–2377

    Article  MathSciNet  Google Scholar 

  11. Saxena A, Sun M, Ng A Y. Make3d: learning 3d scene structure from a single still image. IEEE Trans Patt Anal Mach Intell, 2009, 31: 824–840

    Article  Google Scholar 

  12. Liu B, Gould S, Koller D. Single image depth estimation from predicted semantic labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1253–1260

    Google Scholar 

  13. Tappen M F, Adelson E H, Freeman W T. Estimating intrinsic component images using non-linear regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New York, 2006. 1992–1999

    Google Scholar 

  14. Panagopoulos A, Wang C H, Samaras D, et al. Illumination estimation and cast shadow detection through a higherorder graphical model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, 2011. 673–680

    Google Scholar 

  15. Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24: 381–395

    Article  MathSciNet  Google Scholar 

  16. Frahm J M, Koeser K, Grest D, et al. Markerless augmented reality with light source estimation for direct illumination. In: Proceedings of the European Conference on Visual Media Production, London, 2005. 211–220

    Google Scholar 

  17. Basri R, Jacobs D, Kemelmacher I. Photometric stereo with general, unknown lighting. Int J Comput Vis, 2007, 72: 239–257

    Article  Google Scholar 

  18. Hoiem D, Efros A A, Hebert M. Recovering surface layout from an image. Int J Comput Vis, 2007, 75: 151–172

    Article  Google Scholar 

  19. Gupta A, Efros A A, Hebert M. Blocks world revisited: image understanding using qualitative geometry and mechanics. In: Proceedings of the European Conference on Computer Vision, Heraklion, 2010. 482–496

    Google Scholar 

  20. Shen L, Tan P, Lin S. Intrinsic image decomposition with non-local texture cues. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 2008. 1–7

    Google Scholar 

  21. Bousseau A, Paris S, Durand F. User assisted intrinsic images. ACM Trans Graph, 2009, 28: 130:1–130:10

    Article  Google Scholar 

  22. Jiang X Y, Schofield A J, Wyatt J L. Correlation-based intrinsic image extraction from a single image. In: Proceedings of the European Conference on Computer Vision, Berlin, 2010. 58–71

    Google Scholar 

  23. Barrow H, Tenenbaum J. Recovering intrinsic scene characteristics from images. In: Proceedings of the International Conference on Computer Vision Systems, Nice, 1978. 3–26

    Google Scholar 

  24. Huang P J, Gu Y T, Wu X L, et al. Time-varying clustering for local lighting and material design. Sci China Ser F-Inf Sci, 2009, 52: 445–456

    MATH  Google Scholar 

  25. Garg K, Nayar S K. Photorealistic rendering of rain streaks. ACM Trans Graph, 2006, 25: 996–1002

    Article  Google Scholar 

  26. Wenger A, Gardner A, Tchou C, et al. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans Graph, 2005, 24: 756–764

    Article  Google Scholar 

  27. Chabert C F, Einarsson P, Jones A, et al. Relighting human locomotion with flowed reflectance fields. In: Proceedings of the Eurographics Symposium on Rendering, Cyprus, 2006. 183–194

    Google Scholar 

  28. Peers P, Tamura N, Matusik W, et al. Post-production facial performance relighting using reflectance transfer. ACM Trans Graph, 26: 52:1–52:10

  29. Jin X, Zhao M T, Chen X W, et al. Learning artistic lighting template from portrait photographs. In: Proceedings of the European Conference on Computer Vision, Heraklion, 2010. 101–114

    Google Scholar 

  30. Chen X W, Chen M M, Jin X, et al. Face illumination transfer through edge-preserving filters. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, 2011. 281–287

    Google Scholar 

  31. Chen X W, Jin X, Zhao Q P, et al. Artistic illumination transfer for portraits. Comput Graph Forum, 2012, 31: 1425–1434

    Article  Google Scholar 

  32. Supan P, Stuppacher I, Haller M. Image based shadowing in real-time augmented reality. Int J Virtual Real, 2006, 5: 1–7

    Google Scholar 

  33. Pilet J, Geiger A, Lagger P, et al. An all-in-one solution to geometric and photometric calibration. In: Proceedings of the IEEE and ACM Symposium on Mixed and Augmented Reality, Santa Barbara, 2006. 69–78

    Google Scholar 

  34. Jensen T, Andersen M S, Madsen C B. Real-time image based lighting for outdoor augmented reality under dynamically changing illumination conditions. In: Proceedings of the International Conference on Computer Graphics Theory and Applications, Setúbal, 2006. 364–371

    Google Scholar 

  35. Haber T, Fuchs C, Bekaert P, et al. Relighting objects from image collections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 627–634

    Google Scholar 

  36. Karsch K, Hedau V, Forsyth D, et al. Rendering synthetic objects into legacy photographs. ACM Trans Graph, 2011, 30: 157:1–157:12

    Article  Google Scholar 

  37. Phong B T. Illumination for computer generated pictures. Commun ACM, 1975, 18: 311–317

    Article  Google Scholar 

  38. Moré J J. The Levenberg-Marquardt algorithm: implementation and theory. Numer Anal, 1977, 630: 105–116

    Google Scholar 

  39. Lalonde J F, Efros A A, Narasimhan S G. Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans Graph, 2009, 28: 131:1–131:10

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. State Key Laboratory of Virtual Reality Technology and Systems, School of Computer Science and Engineering, Beihang University, Beijing, 100191, China

    XiaoWu Chen, Xin Jin & Ke Wang

Authors
  1. XiaoWu Chen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xin Jin
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Ke Wang
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Xin Jin.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Chen, X., Jin, X. & Wang, K. Lighting virtual objects in a single image via coarse scene understanding. Sci. China Inf. Sci. 57, 1–14 (2014). https://doi.org/10.1007/s11432-013-4936-0

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11432-013-4936-0

Keywords

Search

Navigation