Research 2


Mixed Reality

Occlusion handling


  • Omnidirectional Video Completion

    We proposed a novel omnidirectional video completion framework based on depth estimation. First, we recover the depth of the scene from a pixel motion model constrained by known camera pose. The depth map is further improved by a structureaware refinement. The refined depth map is then employed for color propagation into the holes.

  • Visibility-based blending

    There are many situations in which virtual objects are presented half-transparently on a background in real time applications. In such cases, we often need to show the object with constant visibility. To overcome this problem, we present a framework for blending images based on a subjective metric of visibility. In our method, a blending parameter is locally and adaptively optimized so that the visibility of each location achieves the targeted level.

  • Aerial perspective rendering

    In outdoor Mixed Reality (MR), objects distant from the observer suffer from an effect called aerial perspective that fades the color of the objects and blends it to the environmental light color. We present a turbiditybased method for rendering a virtual object with aerial perspective effect in a MR application.

Localization / tracking

  • Synthesis-based localization

    A robust image-based alignment method to be used in outdoor environments is proposed. In the proposed method, the albedo of real objects is estimated using 3D shapes of these objects in advance, and the appearance is reproduced from the albedo and current light environment. The appearance of real objects and reproduced image becomes close, so a robust image-based alignment is achieved.

Virtual Reconstruction of Cultural Heritage Assets

  • Virtual Asuka-kyo Project

    We developed Mixed Reality (MR) contents that reconstructed the ancient capital of Asuka-Kyo and applied a fast shading and shadowing method that used shadowing planes. We conducted a subjective evaluation experiment with Head Mounted Display, which showed that displaying these contents increased the audience's knowledge of both Asuka-Kyo and MR technologies. We also conducted impression evaluation tests with and without shading and shadowing.