What is 6D.ai?
6D.ai is an AR platform for developers, that rapidly captures depth information from the real world to build a dense 3D mesh. Using the 3D mesh captured by the 6D.ai software, developers can build easy to use AR apps where assets are persistent, responsive to occlusion, and synced between multiple users on different platforms more efficiently. Most processing happens on-device in real-time, however this feature set is extended with cloud capabilities, allowing the device to handle world-scale data from the AR Cloud.
How does it work?
The 6D.ai API uses a standard built-in smartphone camera to build a real-time, three-dimensional semantic, crowd sourced map of the world, all in the background. No depth camera is needed to capture the world in 3D.
Using the 3D mesh generated by the 6D.ai API, developers build world-scale apps with a powerful feature set:
Persistence - objects stay in the world where you left them
Multiplayer - devices recognize other devices without needing complicated calibration or synching procedures
Occlusion - assets can be hidden by real-world objects, and peek out from behind them
Physics - assets can interact physically with the real world (bounce, collide etc). Because our model is persistent (not calculated per-frame using a depth-from-cnn estimation), assets will correctly physically interact with objects the camera can't see now that have been previously seen.
Essentially, spatial computing apps that were only possible to build on expensive Head Mounted Displays will now be possible on ARCore and ARKit smartphones. In the future, the 3D spatial data captured by 6D.ai will provide the foundation for 6D.ai AI neural-networks to help developers’ applications understand the world in 3D. We will also be implementing cloud services that will drastically increase capture speed and size.
What can I do with it?
The use cases of such a technology expand far beyond phone-based AR games. We’re seeing interest from enterprises, artists, musicians, OEMs, tool builders, drone-manufacturers and robotics companies to name a few. Having a 3D mesh at your disposal compared to a few sparse anchor points and planes makes a huge difference in realism, interaction, and usability.
In short, the goal is to ensure that anything a developer can do in a high-end AR HMD system can be done on a smartphone.
What platforms does 6D support?
Our beta program will start with support the iPhone X and 8. We will add support for leading ARCore devices shortly thereafter, and expand to support all ARKit and ARCore devices at general public launch. 6D.ai features are cross platform across ARKit and ARCore.
HMD devices and other software eco-systems will be supported over time.
When will you support WebXR, Unreal, Microsoft, Magic Leap?
We already have customers and partners requesting all of the above (and more). We wish we build could everything in parallel at once, but we're a seed stage company with limited resources, and have to prioritize. Our beta program will support a C++ / C# API with a sample Unity App, for both IOS and Android. After that it's up to you to tell us what's most exciting. There is a lot of interest in supporting WebXR before the other platforms, but we haven't made any investments in porting (yet).
How soon can I get a hold of the beta?
We began a measured roll out of the beta beginning in April 2018. As we gather feedback and improve the platform, we expect for each wave of beta access to include more people. Make sure to follow @6D_ai on twitter for updates!
Early Access is available for folks that help us spread the word about the beta. Rack up those shares and referrals to be bumped up on the list!
What is the size limit of the captured mesh?
Your phone’s memory will be able to store the three-dimensional geometry of around 100 sq meters in walking distance, while the on-device storage can hold city blocks of information. Each captured 3D block of space (scene) is saved to disk or the cloud as you move around, and is loaded dynamically as required, similar to the way Google Maps handles tiled sections of a 2D map.
Does 6D just capture the raw mesh, or does it get textures too?
6D.ai captures a dense mesh model of the world based on the depth of every pixel that the camera sees. We have successfully applied textures to the mesh, however the use case we are supporting (AR) only needs a transparent mesh. We felt that the work required to build a system where the texturing quality was high enough to compete with dedicated scanning apps was a lower priority than supporting AR apps. We're always willing to admit we read the market wrong, and if this is an important feature for you, please educate us by reaching out to email@example.com with details.
Does the mesh need to be pre-scanned?
We have technology that allows the meshing to build in real-time, seamlessly synching meshes across multiple devices also in real-time, while other applications run in the foreground. However the load this activity places on a GPU and Network makes it unsuitable for even today's fastest mobile devices. While we know our mesh will build in real-time over very large areas on a current top of the range phone, we are still working to tune the system to play nicely with rendering engines. We expect our initial release of meshing to request a dedicated pre-scan phase, and eventually become a background task. We will of course give developers complete control over when meshing runs, and if you have a lightweight application, meshing may work just fine for you as a background task.
Is the mesh metric scale?
Yes. We take scale from the underlying tracker, so it is as accurate as ARKit or ARCore.
Is network access required for 6D.AI to run?
No. The system will work fine (though at limited scale) without a network. All our processing is done locally on-device. The cloud is only used for data storage, stitching data from multiple sessions, multi-player, and cross-platform adaptations.
Do you have any example of a generated Mesh that I can download?
We do not have examples of 3D meshes captured via the 6D.ai API available for download at this time. Stay tuned, we want to provide developers with all the resrouces you need from us to design, prototype and build your vision.
How is 6D handling multiplayer?
What makes multiplayer AR gaming so difficult is that both phones generally need to see the world from the same vantage point in order to sync up, meaning you literally need to put a phone next to another user’s to sync your maps before joining a game or sharing an experience.
The additional dense 3D data & neural-networks (AI) that the 6D.ai software applies makes AR multiplayer a lot easier to handle. Currently, 6D.ai allows users to sync up much more easily over a wide range of relative positions between the players. Soon users won’t have to deal with this repositioning at all and will be able to complete the process from any angle at any distance.
What is the AR Cloud?
AR systems need an operating system that lives partially on device, and partially in the cloud. Network/Cloud data services are as critical to AR apps as the network is to making mobile phone calls. Think back before smartphones… your old Nokia mobile phone without the network could still be a calculator and you could play snake, but its usefulness was pretty limited. The network and AR Cloud are going to be just as essential to AR apps. We believe we will come to view today’s simple tabletop ARKit/ARCore apps as the equivalent to just having offline “Nokia Snake” vs a network connected phone.
How does the cloud implementation work?
Each captured 3D block of space (scene) is saved to disk or the cloud as you move around. This means that as more and more devices running 6D.ai’s engine hit the streets, a web of phones will begin building up a cloud map of the world’s ground-level three-dimensional data, in machine readable form. As other users open up apps in areas that have previously been mapped, 6D.ai will download that information and allow the new user to further refine the precision of the 3D model while pushing 3D interactions further into the distance than their devices can sense.
What is the licensing model?
We do not have plans to charge any licensing fees while the software is in beta. We will always offer a free tier for developers to experiment, and eventually intend to charge per transaction in a similar manner to Stripe, AWS or Twilio.
What is your stance on data privacy?
Privacy is an important and delicate subject that we take very seriously. We are putting a lot of thought into how data will be gained, accessed, and stored through 6D.ai. We also expect to learn quite a bit from what developers build - certain use cases will almost certainly have unique approaches to privacy and security. Currently, the mesh captured by 6D is texture-less and not photo-realistic, providing some degree of obfuscation. We have a number of technical programs of work underway (some common sense, some quite innovative) to responsibly manage 3D data. Many of these social issues have been solved for regular 2D photography (eg Flickr, Google photos) and we expect to listen & learn from our stakeholders to figure out the new aspects to address.
What's Next for the 6D Platform?
After beta program, we will be working on automatically segmenting meshes so that its system can identify different 3D real-world objects and passing that knowledge back to developers. This means roads and sidewalks can be distinguished outdoors, chairs and walls recognized indoors.
Are you hiring?