The benefit of Handheld Augmented Reality (HAR) is that it enables users to Augmented Reality (AR) content in a mobile context. The 3D positioning is fundamental in HAR content creation. Due to the hardware limitations of handheld devices and restrictions in the use environment, the accurate 3D positioning can be impossible. We present an intuitive 3D positioning method for HAR called SlidAR. It utilizes 3D ray-casting and epipolar geometry for virtual object positioning. The position can be adjusted with a slide gesture. The SlidAR method does not require a perfect 3D reconstruction of the environment nor any virtual depth cues.

We propose a novel system for visual overlay of rectangular virtual object onto real environment observed by tablet PC with camera. This system allows us to visually simulate the layout of virtual rectangular objects such as advertising posters in the field. For estimating the pose and position of the tablet PC in the 3D structure of the target environment, we proposed the procedure using the motion sensor in tablet PC. In this paper, we introduce two demonstrations about the deformation for a rectangular virtual object and the reflection of light such as sunlight.

Accurate eye-gaze tracking is possible through active methods which use infra-red light sources and cameras. Although it is desirable to recover the eye-gaze from natural images, passive methods that work on these images are less accurate. We present a novel method to passively estimate the eye-gaze in known environments. We use corneal imaging to determine the position of the eye and use it to recover and accurate orientation of the eye. In our demo the participants can compare the eye-gaze estimated by the traditional approach, we use the implementation from, and our solution.

Augmented Reality (AR) is a technology which superimposes computer graphics (CG) images onto a user's view of the real world. A commonly used AR display device is an Optical See-Through Head-Mounted Display (OST-HMD), which is a transparent HMD, enabling users to observe the real-world directly, with CG added to it. A common problem in such systems is the mismatch between the properties of the user's eyes and the virtual camera used to generate CG. The goal of our system, is to accurately reflect the state of the user's eyes in our renderings. Using an auto Refractometer, we measure the user's pupil size and accommodative state and feed these values into a realtime raytracer. The resulting renderings accurately reflect the Depth-of- Field (DoF) blur effect the user perceives in their view of the real world. In our demo, users can verify the accuracy of our system by simultaneously observing 3D-printed objects next to their CG counterparts. We believe that our technique is the most promising way for achieving augmentations that are indistinguishable from real objects. Integrating it into consumer OST-HMDs will be a crucial step for bringing photo realistic AR to the masses.

In Mixed Reality (MR) space, we can change appearances of a real object (R) and virtual object (V). In this research, we visually change movements of the real and virtual objects and observe the influences of the virtual object's dynamics on the sense of the weight of the real object. In this demo, participants can experience this illusion under some conditions that moving virtual liquid is superimposed on the real object.

This demonstration proposes a diminished reality method for hiding a pedestrian from a hand-held camera sequence. In the proposed method, the pedestrian is extracted from each frame of input video sequence using HOG based people detector. The detected area is excluded from each frame by masking, and then two frames are stitched for generating a stitched background image. By using the stitched background image, the pedestrian's mask for hiding is made. It is used for overlaying the background pixel to the pedestrian in the original frame. This demonstration presents to erase a pedestrian using a hand-held video camera.

Optical See-Through Head-Mounted Displays are an increasingly common medium for presenting AR content. A problem in these systems, though, is the existence of focal rivalry resulting from the distance disparity between real objects and the virtual screen of the display. Due to the focal limitations of the human eye, this focal discrepancy results in onscreen images appearing blurred during fixation on real objects. The goal of our SharpView system is to ameliorate this problem through application of a variable sharpening filter. Our demonstration allows users to experience the improved legibility of CG content provided by the SharpView technique.

In this paper, we proposed an augmented reality application for mobile devices, DroneAR, which is use for farmer perspective in agriculture by using unmanned aerial vehicle (UAV). Although there has been a lot of types about the use of UAVs within the agricultural sector including spraying, seeding, remote sensing, precision agriculture, frost mitigation and variable rate dispersal, AR technology will reduce farmer inputs and increase their yields. Our goal is to increase farm efficiency. A demonstration has been implemented to present the concept based on the global positioning system (GPS) to automatically display the points of interest (POI) labels on real-time.

We propose an MR "domino toppling" attraction as a case study of MR systems, focusing on the Real_Virtual (R_V) and Virtual_Real (V_R) continuum. In this attraction, the user experiences that real and virtual domino blocks seamlessly pushes each other through a video-seethrough head-mounted display. Therefore, the user may believe that attractive MR effects happen in the real space. To achieve the seamless R_V/V_R transitions, we implemented simple switching devices. In addition, we use diminished reality techniques to visually remove the devices to make it harder for the user to distinguish R_V/V_R transitions.

We present a system that realizes a spatial interaction between a handheld device and external displays via imperceptible on-screen markers. Our system can compute a full pose of a camera installed on handheld devices by extracting a marker from external displays, while a user doesn't notice where the marker is. This technology enables users to see Augmented Reality objects on top of the external display through their handheld devices. Our method consists of two parts: marker embedding on external displays and marker detection. Compared to previous research, our system allows that external displays' content can change and don't need to be known in advance.

We present a system in which users can experience several methods of diminished reality (DR) that removes or diminishes undesirable objects from their perceived environment. Compared to augmented/mixed reality, DR has no appropriate examples of experience-based systems. Therefore, as an example of such DR systems, we developed a system named "Magical Mystery Room, 2nd Stage" that removes objects like a magic show. In this system, the users can see various types of DR methods such as image inpainting-based DR and pre-observation based DR, and experience their advantages and disadvantages. We build a full-scale set in which an user can move around to present various situations depending on lightings and the user's viewpoint. We expect that experience-based DR system offers opportunities to discuss the future of DR.

In this demonstration, we present a mobile binocular augmented reality (AR) system, which can provide an immersive experience in museums. We tested and evaluated several different platforms to select an appropriate AR system for museums. As a result, we designed and implemented a smartphone-based binocular AR system. Smartphones are lightweight but powerful, thus the implemented system can employ various tracking techniques and provide rich augmented 3D contents in actual museums. The proposed smartphone-based mobile binocular AR viewer can also be used in other areas such as tourism, entertainment, and military applications.

As the estimating human motion using Kinect camera usually occurs that partial overlay, caused either by objects or body posture, are to be taken into consideration. Thus, we propose the multiple Kinect cameras for 3D human skeleton posture using axis replacement method. The objective of this work is to understand the ability of using multiple Kinects as tracking devices using proposed method that uses two Kinect cameras for acquire distributed values and described a fusing axis replacement method for each joint positions. To show the effectiveness of our work, we performed the experiment that compare series of fused skeleton obtained from two Kinects that computed using proposed method and others. The results shown that proposed method have more distributed values than average method and original values from single Kinect method significantly. Furthermore skeleton overlay problem of using single Kinect method could be reduced.

We can't grasp a sighted but distant object because of the constraint of our arm length. By using a virtual hand, we extend our body and grasp the distant object in those situations. In conventional virtual hand systems, users usually must wear a data glove or set a camera in the environment. These necessities narrow usable situations of a virtual hand. In this research, we propose a new virtual hand which is manipulated by a smartphone. It realizes the normal use of a virtual hand in more practical scene while we normally carry our smartphone in a daily life. For implementing the proposed system, the hand pose estimation is a key process because measurable real hand data is not enough. For displaying five fingers, we focus on the hand pose estimation from three touch points on the screen and a tilt information of the smartphone's body.

Our demonstration shows the method 3D object pose estimation using smartphone especially temporary unseen object. By using inertial sensors in smartphones, we can obtain image-independent pose estimation. In this study, we propose a simple a sensor fusion method which can help image-based 3D camera pose estimation method. We demonstrate that our system shows better tracking results than an image tracking method in an obstructed scene.

Augmented Reality(AR) is technology to extend the real environ- ment that people are perceived by the computer. Among the plu- rality of AR technology, this paper is focused on marker-based AR technology. It has a problem that the technology is available on the flat surface only. Therefore, this paper proposes the original AR marker design and processing method that is also recognizable on the foldable and cylindrical surface.

We present a novel virtual reality system that enables users to fullynavigate a photo-realistic scene generated from omnidirectional images,on a head mounted display (HMD). In contrast to traditionalimage-based rendering (IBR) systems, requiring dense image series,our system requires only a sparse set. We have validated oursystem with over 300 users. In addition, we use a scene structurebased technique for spatially interpolating views between images.To enhance the immersion, we propose a novel displaying techniqueof interpolated views according to user's head motion. Preliminaryresults of this displaying technique are demonstrated.

Coloring books capture the imagination of children and provide them with one of their earliest opportunities for creative expression. However, given the proliferation and popularity of digital devices, real-world activities like coloring can seem unexciting, and children become less engaged in them. Augmented reality holds unique potential to impact this situation by providing a bridge between real-world activities and digital enhancements. In this paper, we present an augmented reality coloring book App in which children color characters in a printed coloring book and inspect their work using a mobile device. The drawing is detected and tracked, and the video stream is augmented with an animated 3-D version of the character that is textured according to the child’s coloring. This is possible thanks to several novel technical contributions. We present a texturing process that applies the captured texture from a 2-D colored drawing to both the visible and occluded regions of a 3-D character in real time. We develop a deformable surface tracking method designed for colored drawings that uses a new outlier rejection algorithm for real-time tracking and surface deformation recovery. We present a content creation pipeline to efficiently create the 2-D and 3-D content. And, finally, we validate our work with two user studies that examine the quality of our texturing algorithm and the overall App experience.

In this work, we present a new automatic system for scene reconstruction, which delivers high- level structural models. We start with identifying planar regions in depth images obtained with a SLAM system. Our main contribution is an approach which identifies constraints such as incidence and orthogonality of planar surfaces and uses them in an incremental optimization framework to extract high-level structural models. The result is a manifold mesh with a low number of polygons, immediately useful in many Augmented Reality applications.

Volumetric methods provide efficient, flexible and simple ways of integrating multiple depth images into a full 3D model. They provide dense and photorealistic 3D reconstructions, and parallelised implementations on GPUs achieve real-time performance on modern graphics hardware. To run such methods on mobile devices, providing users with freedom of movement and instantaneous reconstruction feedback, remains challenging however. In this paper we present a range of modifications to existing volumetric integration methods based on voxel block hashing, considerably improving their performance and making them applicable to tablet computer applications. We present (i) optimisations for the basic data structure, and its allocation and integration; (ii) a highly optimised raycasting pipeline; and (iii) extensions to the camera tracker to incorporate IMU data. In total, our system thus achieves frame rates up 43 Hz on a Nvidia Shield Tablet and 820 Hz on a Nvidia GTX Titan X GPU, or even beyond 1 kHz without visualisation.

VWe present the first pipeline for real-time volumetric surface reconstruction and dense 6DoF camera tracking running purely on standard, off-the-shelf mobile phones. Using only the embedded RGB camera, our system allows users to scan objects of varying shape, size, and appearance in seconds, with real-time feedback during the capture process. Unlike existing state of the art methods, which produce only point-based 3D models on the phone, or require cloud-based processing, our hybrid GPU/CPU pipeline is unique in that it creates a connected 3D surface model directly on the device at 25Hz. In each frame, we perform dense 6DoF tracking, which continuously registers the RGB input to the incrementally built 3D model, minimizing a noise aware photoconsistency error metric. This is followed by efficient key-frame selection, and dense per-frame stereo matching. These depth maps are fused volumetrically using a method akin to KinectFusion, producing compelling surface models. For each frame, the implicit surface is extracted for live user feedback and pose estimation. We demonstrate scans of a variety of objects, and compare to a Kinect-based baseline, showing on average ~1.5cm error. We qualitatively compare to a state of the art point-based mobile phone method, demonstrating an order of magnitude faster scanning times, and fully connected surface models.

In the last few years, the advancement of head mounted display technology and optics has opened up many new possibilities for the field of Augmented Reality. However, many commercial and prototype systems often have a single display modality, fixed field of view, or inflexible form factor. In this paper, we introduce Modular Augmented Reality (ModulAR), a hardware and software framework designed to improve flexibility and hands-free control of video see-through augmented reality displays and augmentative functionality. To accomplish this goal, we introduce the use of integrated eye tracking for on-demand control of vision augmentations such as optical zoom or field of view expansion. Physical modification of the device’s configuration can be accomplished on the fly using interchangeable camera-lens modules that provide different types of vision enhancements. We implement and test functionality for several primary configurations using telescopic and fisheye camera-lens systems, though many other customizations are possible. We also implement a number of eye-based interactions in order to engage and control the vision augmentations in real time, and explore different methods for merging streams of augmented vision into the user’s normal field of view. In a series of experiments, we conduct an in depth analysis of visual acuity and head and eye movement during search and recognition tasks. Results show that methods with larger field of view that utilize binary on/off and gradual zoom mechanisms outperform snapshot and sub-windowed methods and that type of eye engagement has little effect on performance.

We present a method which can quickly and robustly match 2D and 3D point patterns based on their sole spatial distribution, but it can also handle other cues if available. This method can be easily adapted to many transformations such as similarity transformations in 2D/3D, and affine and perspective transformations in 2D. It is based on local geometric consensus among several local matchings and a refinement scheme. We provide two implementations of this general scheme, one for the 2D homography case (which can be used for marker or image tracking) and one for the 3D similarity case. We demonstrate the robustness and speed performance of our proposal on both synthetic and real images and show that our method can be used to augment any (textured/textureless) planar objects but also 3D objects.

We present a natural gesture interface for ambient-objects using a wearable RGB-D sensor. The aim of this work is to propose a methodology that determines accurately where a user is pointing at when gesturing with their finger. First, the wearable RGB-D sensor is affixed around the user forehead. A calibration between the user’s eyes and the RGB-D camera is performed by having the user move their fingers along their line of sight. We detect the fingertip in the depth camera and then find the direction of the line of sight. Finally we estimate where the user is pointing at in the RGB image in different scenarios with a depth map, a detected object and a controlled virtual element. To validate our methods, we perform a point-to-screen experiment. Results demonstrate that when a user is interacting with a display up to 1.5 meters away, our natural gesture interface has an average error of 2.1cm. In conclusion, the presented technique is a viable option for a reliable user interaction.

In this extended abstract, we present a model that aims to provide developers with an extensive and extensible set of context-aware interaction techniques, greatly facilitating the creation of meaningful AR-based user experiences. To provide a complete view of the model, we detail the different aspects that form its theoretical foundations, while also discussing several considerations for its correct implementation.

An important yet unsolved problem in computer vision and Augmented Reality (AR) is to compute the 3D shape of nonrigid objects from live RGB videos. When the object's shape is provided in a rest pose, this is the Shape-from-Template (SfT) problem. We present a general framework for realtime SfT. This handles generic objects, complex deformations and most of the difficulties present in real imaging conditions. Achieving this has required new solutions to two core sub-problems in SfT: robust registration and fast 3D shape inference. For registration we propose Deformable Render-based Block Matching (DRBM), which is a tracking-based solution that combines the advantages of feature-based and direct approaches without their main disadvantages. Shape inference is achieved by solving a single sparse linear least squares system for each frame, which is done quickly with a Geometric Multi-Grid method. On a standard desktop PC we archive up to 21fps depending on the object. Code will be released to the community.

In this work, we propose a multi-user system for tracking and mapping, which accommodates mobile clients with different capabilities, mediated by a server capable of providing real-time structure from motion. Clients share their observations of the scene according to their individual capabilities. This can involve only keyframe tracking, but also mapping and map densification, if more computational resources are available. Our contribution is a system architecture that lets heterogeneous clients contribute to a collaborative mapping effort, without prescribing fixed capabilities for the client devices. We investigate the implications that the clients' capabilities have on the collaborative reconstruction effort and its use for AR applications.

In this paper we propose an adaptive Augmented Reality interface for general hand gestures based on a probabilistic model. The proposed interface provides multiple interfaces and the corresponding gesture inputs by recognizing a context of the hand shape which requires the accurate recognition of static and dynamic hand states. For the accuracy, we present a hand representation that is robust to the hand shape variation, and the extraction of hand features based on the fingertip posteriors from a GMM model. Experimental results show that both context-sensitivity and accurate hand gesture recognition are achieved throughout the quantitative evaluation and its implementation as a three-in-one virtual interface.

Securing one’s personal space is quite important in leading a comfortable social life. However, it is difficult to maintain an appropriate interpersonal distance all the time. Therefore, we propose an interpersonal distance control system with a video see-through system, consisting of a head-mounted display (HMD), depth sensor, and RGB camera. The proposed system controls the interpersonal distance by changing the size of the person in the HMD view. In this paper, we describe the proposed system and conduct an experiment to confirm the capability of the proposed system. Finally, we show and discuss the results of the experiment.

The ideal AR x-ray vision should enable users to clearly observe and grasp not only occludees, but also occluders. We propose a novel selective visualization method of both occludee and occluder layers with dynamic opacity depending on the user's gaze depth. Using the gaze depth as a trigger to select the layers has a essential advantage over using other gestures or spoken commands in the sense of avoiding collision between user's intentional commands and unintentional actions. Our experiment by a visual paired-comparison task shows that our method has achieved a 20% higher success rate, and significantly reduced 30% of the average task completion time than a non-selective method using a constant and half transparency.

This demo explores how to augment live coding as an improvisatory musical performance practice with evolutionary sound synthesis, audio feature extraction and linked semantic data formats to aid the performer through the complexities of digital performance environments. The ¥textit{evolver} programming environment uses gene expression synthesis to generate complex synthesis graphs during live coding performances. The main principle guiding the evolution is based on extracting audio features for comparison to a target sound. The audio features are also employed as an informative framework for the performer in an otherwise featureless collection of synthesizer. The structure of the evolutionary synthesis environment is published in a Semantic Web ontology. The synthesizers are stored in a database using the JSON-LD data format to enable linking the synthesizer data to the ontology and external data sources.

The project NarcissUs is a critical and artistic approach towards the development of intelligent machines. The main goal is to unfold the potential and unexpected outcomes that the learning process from artificial beings in mixed reality environments may encounter. It consists on a feedback loop setup between a computer generated being (NarcissUs), a camera, a moving mirror, a face recognition algorithm and an evolving learning machine seeking for self awareness. NarcissUs seek for its reflection, learn from it and evolve through their expressive exchanges. Would IT be able to tell that the reflection is nothing else but itself?

This paper demonstrates a mixed-reality installation named Wanderl_st, which utilizes the digital dartboard system as an agent of map navigation. The project enable users to explore new geo-locations according to the actual score and relative positions of thrown darts on board. Meanwhile, the routes are calculated and the map directional guidance is translated spatially into acoustic outputs.

Mental manipulations of space, normally called as spatial visualization or visual thinking,are thought to be important in various field. However many studies in the past concluded that it was difficult to define, measure, and help develop such skills. This paper introduces an application of augmented reality where you can cut, rotate, and fold various kind of objects and argues how such virtual environments could help the situation.

For a digital clay modeling, AR technique is applied with highly cost-performed devices; web camera, print-out markers, and a pair of chopsticks, or "hashi" in Japanese. A user can build up a particle-based model by HASHI from scratch.

Historical inquiry involves investigating compelling questions by analyzing historical sources to construct evidence-based accounts of the past. However, teaching students to do history is challenging. This paper discusses the design of CI-Spy, a mobile augmented reality system that explicitly teaches inquiry strategies and engages students to practice the doing of history in an augmented real-world context. As a case study for the design of the application, we de- signed and embedded multiple augmented reality activities within an instructional unit using a local historic site (the Christiansburg Institute, or CI). We conducted a pilot study with elementary students to learn how and to what extent AR technologies can sup- port learning inquiry strategies and processes. After using our system, students demonstrated a greater understanding of inquiry and gained significant insight into the hidden history of CI.