Patent application number | Description | Published |
20110285810 | Visual Tracking Using Panoramas on Mobile Devices - A mobile platform maps and tracks the orientation of the mobile platform in real-time as the mobile platform rotates. The mobile platform captures a plurality of camera images as the camera rotates and projects the plurality of camera images consecutively to a panoramic cylindrical map. The map is extended by projecting areas of any camera image that correspond to unmapped portions of the panoramic cylindrical map. Keypoints are extracted from mapped portions of the map and using the keypoints the orientation of the camera is determined. Accumulated error is removed when closing the panoramic cylindrical map by extending the map horizontally beyond 360 degrees to produce overlapping regions and extracting and matching keypoints from the overlapping regions. Additionally, an incorrect estimate of the initial orientation of the map may be corrected by determining the correct orientation and re-projecting the map onto a map with the correct orientation. | 11-24-2011 |
20110285811 | ONLINE CREATION OF PANORAMIC AUGMENTED REALITY ANNOTATIONS ON MOBILE PLATFORMS - A mobile platform generates a panoramic map by rotating a camera and stores the panoramic map. The mobile platform performs a computer vision operation on the panoramic map while continuing to generate the panoramic map. The mobile platform may determine the location of the camera when the panoramic map is generated, produce an annotation on the panoramic map and transmit the determined location, annotation, and portion to a storage device. In another example, the mobile platform may receive an annotation, a portion of a source panoramic map associated with the annotation, and a location where the source panoramic map was generated. The mobile platform may compare the portion of the source panoramic map to the panoramic map to identify a position on the panoramic map that corresponds to the portion of the source panoramic map, and display the annotation based on the determined position on the panoramic map. | 11-24-2011 |
20110286631 | REAL TIME TRACKING/DETECTION OF MULTIPLE TARGETS - A mobile platform detects and tracks at least one target in real-time, by tracking at least one target, and creating an occlusion mask indicating an area in a current image to detect a new target. The mobile platform searches the area of the current image indicated by the occlusion mask to detect the new target. The use of a mask to instruct the detection system where to look for new targets increases the speed of the detection task. Additionally, to achieve real-time operation, the detection and tracking is performed in the limited time budget of the (inter) frame duration. Tracking targets is given higher priority than detecting new targets. After tracking is completed, detection is performed in the remaining time budget for the frame duration. Detection for one frame, thus, may be performed over multiple frames. | 11-24-2011 |
20120075433 | EFFICIENT INFORMATION PRESENTATION FOR AUGMENTED REALITY - Information to be displayed is filtered to reduce the amount of information which has to be arranged on screen to increase comprehensibility. The filter preserves the information encoded in the visualization by removing redundant elements by first clustering similar elements and then selecting a single representative from each cluster. Additionally, the layout of the information is optimized based on an evaluation of element comprehensibility in order to achieve a compact presentation suitable for small screen devices. The compact presentation of data may be updated on a mobile platform with real-time frame rates by pre-computing multiple view points and displaying a frame coherent transition between layouts, so that temporal coherency is retained during camera movements. | 03-29-2012 |
20120300020 | REAL-TIME SELF-LOCALIZATION FROM PANORAMIC IMAGES - Real-time localization is performed using at least a portion of a panoramic image captured by a camera on a mobile device. A panoramic cylindrical map is generated using images captured by the camera, e.g., as the camera rotates. Extracted features from the panoramic cylindrical map are compared to features from a pre-generated three-dimensional model of the environment. The resulting set of corresponding features may be used to determine the pose of the camera. For example, the set of corresponding features may be converted into rays between the panoramic cylindrical map and the three-dimensional model, where the intersection of the rays is used to determine the pose. The relative orientation of the camera may also be tracked by comparing features from each new image to the panoramic cylindrical map, and the tracked orientation may be fused with the pose. | 11-29-2012 |
20130271625 | PHOTOMETRIC REGISTRATION FROM ARBITRARY GEOMETRY FOR AUGMENTED REALITY - Photometric registration from an arbitrary geometry for augmented reality is performed using video frames of an environment captured by a camera. A surface reconstruction of the environment is generated. A pose is determined for the camera with respect to the environment, e.g., using model based tracking using the surface reconstruction. Illumination data for the environment is determined from a video frame. Estimated lighting conditions for the environment are generated based on the surface reconstruction and the illumination data. For example, the surface reconstruction may be used to compute the possible radiance transfer, which may be compressed, e.g., using spherical harmonic basis functions, and used in the lighting conditions estimation. A virtual object may then be rendered based on the lighting conditions. Differential rendering may be used with lighting solutions from the surface reconstruction of the environment and a second surface reconstruction of the environment combined with the virtual object. | 10-17-2013 |
20130314441 | IMAGE-DRIVEN VIEW MANAGEMENT FOR ANNOTATIONS - A mobile device uses an image-driven view management approach for annotating images in real-time. An image-based layout process used by the mobile device computes a saliency map and generates an edge map from a frame of a video stream. The saliency map may be further processed by applying thresholds to reduce the number of saliency levels. The saliency map and edge map are used together to determine a layout position of labels to be rendered over the video stream. The labels are displayed in the layout position until a change of orientation of the camera that exceeds a threshold is detected. Additionally, the representation of the label may be adjusted, e.g., based on a plurality of pixels bounded by an area that is coincident with a layout position for a label in the video frame. | 11-28-2013 |
20140016821 | SENSOR-AIDED WIDE-AREA LOCALIZATION ON MOBILE DEVICES - A mobile device uses vision and orientation sensor data jointly for six degree of freedom localization, e.g., in wide-area environments. An image or video stream is captured while receiving geographic orientation data and may be used to generate a panoramic cylindrical map of an environment. A bin of model features stored in a database is accessed based on the geographic orientation data. The model features are from a pre-generated reconstruction of the environment produced from extracted features from a plurality of images of the environment. The reconstruction is registered to a global orientation and the model features are stored in bins based on similar geographic orientations. Features from the panoramic cylindrical map are matched to model features in the bin to produce a set of corresponding features, which are used to determine a position and an orientation of the camera. | 01-16-2014 |
20140320530 | APPARATUS AND METHOD FOR RADIANCE TRANSFER SAMPLING FOR AUGMENTED REALITY - Methods, systems, computer-readable media, and apparatuses for radiance transfer sampling for augmented reality are presented. In some embodiments, a method includes receiving at least one video frame of an environment. The method further includes generating a surface reconstruction of the environment. The method additionally includes projecting a plurality of rays within the surface reconstruction of the environment. Upon projecting a plurality of rays within the surface reconstruction of the environment, the method includes generating illumination data of the environment from the at least one video frame. The method also includes determining a subset of rays from the plurality of rays in the environment based on areas within the environment needing refinement. The method further includes rendering the virtual object over the video frames based on the plurality of rays excluding the subset of rays. | 10-30-2014 |
20140320593 | MONOCULAR VISUAL SLAM WITH GENERAL AND PANORAMA CAMERA MOVEMENTS - Disclosed are a system, apparatus, and method for monocular visual simultaneous localization and mapping that handles general 6DOF and panorama camera movements. A 3D map of an environment containing features with finite or infinite depth observed in regular or panorama keyframes is received. The camera is tracked in 6DOF from finite, infinite, or mixed feature sets. Upon detection of a panorama camera movement towards unmapped scene regions, a reference panorama keyframe with infinite features is created and inserted into the 3D map. When panoramic camera movement extends toward unmapped scene regions, the reference keyframe is extended with further dependent panorama keyframes. Panorama keyframes are robustly localized in 6DOF with respect to finite 3D map features. Localized panorama keyframes contain 2D observations of infinite map features that are matched with 2D observations in other localized keyframes. 2D-2D correspondences are triangulated, resulting in new finite 3D map features. | 10-30-2014 |
20140321702 | DIMINISHED AND MEDIATED REALITY EFFECTS FROM RECONSTRUCTION - Disclosed embodiments pertain to apparatus, systems, and methods for mixed reality. In some embodiments, a camera pose relative to a tracked object in a live image may be determined and used to render synthetic images from keyframes in a 3D model without the tracked object. Optical flow magnitudes for pixels in a first mask region relative to a subset of the synthetic images may be determined and the optical flow magnitudes may be used to determine pixels in each of the subset of synthetic images that correspond to pixels in the first mask. For each pixel in the first mask, a corresponding replacement pixel may be determined as a function of pixels in the subset of synthetic images that correspond to the corresponding pixel in the first mask. | 10-30-2014 |
20140323148 | WIDE AREA LOCALIZATION FROM SLAM MAPS - Exemplary methods, apparatuses, and systems for performing wide area localization from simultaneous localization and mapping (SLAM) maps are disclosed. A mobile device can select a first keyframe based SLAM map of the local environment with one or more received images. A respective localization of the mobile device within the local environment can be determined, and the respective localization may be based on the keyframe based SLAM map. The mobile device can send the first keyframe to a server and receive a first global localization response representing a correction to a local map on the mobile device. The first global localization response can include rotation, translation, and scale information. A server can receive keyframes from a mobile device, and localize the keyframes within a server map by matching keyframe features received from the mobile device to server map features. | 10-30-2014 |
20150070387 | STRUCTURAL MODELING USING DEPTH SENSORS - Techniques are presented for constructing a digital representation of a physical environment. In some embodiments, a method includes obtaining image data indicative of the physical environment; receiving gesture input data from a user corresponding to at least one location in the physical environment, based on the obtained image data; detecting at least one discontinuity in the physical environment near the at least one location corresponding to the received gesture input data; and generating a digital surface corresponding to a surface in the physical environment, based on the received gesture input data and the at least one discontinuity. | 03-12-2015 |