Exploring the Spatial Computing Spectrum: Digitization to Digital Twin

Introduction

Reality capture (RC) is the process of digitizing real-world objects or environments. As this process can be complex, this post provides context to the current state of associated technologies. RC has vast implications in many industries, including Automotive, Aerospace, Oil and Gas, Engineering, Construction, and Real Estate.

There is a fundamental shift taking place across these industries related to accurately capturing and digitizing the real world and objects within it. This technology has been improved over the last several years. What was previously a very expensive and time-consuming process that led to potentially inaccurate results, is now faster, less expensive, and more accurate. With the initial value of RC now being validated in enterprise, an emphasis on enablement is critical to usher in the next wave of adoption. In order to uncover this technology’s full potential, it is necessary to decrease barriers to entry and implementation.

This post will cover several aspects of digitization, including methods, an overview of digital twins, and a case study to show a real-world example with an AWS customer. You will also hear from the Reality Capture Network (RCN), an online education center that focuses on digitization. Matthew Byrd, Founder of RCN, will dive into trends he sees and use cases of RC around the world.

Benefits and Challenges

The benefits of RC have been validated from enterprise companies around the world. Common themes among these use-cases include surveying conditions, operational improvements, safety analysis, security analysis, and core business process advancements. The benefits can be substantial and so can the challenges. To realize these benefits, a company must understand the associated key Spatial Computing topics and challenges. These associated challenges include the 3D capture process (Digitization Methods), 3D model manipulation, and final asset implementation. These challenges will be discussed throughout this post.

Digitization Methods

In this post I will discuss three methods for digitizing the real world: LiDAR based scanning, Photogrammetry, and Neural Radiance Fields (NeRFs).

LiDAR Scanning

LiDAR based solutions send out light rays from a scanning device and calculate the time it takes to return. This is referred to as a Time of Flight (ToF) calculation. This method creates a point cloud of the real world and objects that the rays come into contact with. Point clouds can vary greatly in density depending on the settings selected and device limitations. A denser point cloud will result in a proportionally higher resolution end product. These point clouds can then be converted into 3D meshes that can be consumed by industry standard 3D software. Solutions like Autodesk’s ReCap can help with this conversion.

Conceptual image of point cloud

LiDAR scanners can be expensive and in some cases need a technician to operate. There have been great advancements in decreasing these barriers, and now there are also simpler scanners that are very easy to use. Handheld LiDAR scanners and newer high end smart phone models with LiDAR built in, can allow users to capture environments and objects quickly and easily.

Photogrammetry

Photogrammetry is a fundamentally different process from LiDAR based solutions. This process takes a set of images of an object or area as an input, and uses an algorithm to build a 3D model recreation from those images. To keep processing times reasonable, typically these image sets range roughly from 40 to 250 photographs, but can use more or less. These input images can also be extracted from video, which makes the capture process very fast, under two minutes in some cases. This makes it easy for anyone to quickly capture an environment or object. The more images included, the longer it takes for processing and recreation. Depending on the quality of the input images, this can result in a 20-30% higher resolution asset.

Left: Screenshot of the photogrammetry process. Each white triangle above the building is one input photograph camera location. Right: Example of one of the input photographs. Thomas Jefferson Memorial source data from cyark.org, license CC BY-NC-SA.

Common uses of photogrammetry include cultural heritage (art, museum relics), media and entertainment (human face and body scanning for CGI character creation), gaming (hyper-realistic game asset creation), and site surveying (capturing buildings and areas of land, commonly with use of drones).

Neural Radiance Field (NeRF)

NeRF technology is the most infant of the three techniques. NeRF uses a process that is similar to photogrammetry in that it uses photographs or extracted video frames for 3D reconstruction. A NeRF is a unique asset type which can only be viewed in specialized software. The result can be difficult to convert into a clean 3D mesh to then be used with other standard 3D modeling software and tools.

NeRF technology, including Nvidia’s Instant NeRF is improving rapidly, however early results can have substantial artifacts in their raw output. These artifacts can make the resulting 3D asset difficult to work with and less desirable for certain use cases. However, it is important to note that artifacts are also typical of raw LiDAR and photogrammetry-based scans. With LiDAR and photogrammetry having been around longer, the tools available to refine, convert and integrate these models into other software are more mature as well. For this reason, NeRF based assets have not been used heavily within enterprise use cases yet. This is sure to change as the technology and tools become more mature.

Neural Radiance Field (NeRF) Scan of a street in New York City. Using Nvidia’s Instant-ngp. Notice the blurry artifacts in the resulting output.

LiDAR, photogrammetry, and NeRF models can all be very accurate and high-resolution solutions, all aimed at different use cases. With LiDAR and photogrammetry having the ability to result in millimeter accuracy, and NeRF in its infancy, these are different tools for different jobs. As this post is focused on enterprise use cases, I will not expand further on NeRF technology and will instead dive deeper into LiDAR and photogrammetry.

3D Model Optimization

3D model resolution is important to acknowledge for the following reasons. Digital Twins are only as valuable as the information that can be gleaned from them. If the information is not high enough resolution, it may not be adequate to make necessary business decisions. If the resolution is too high, the application can become slow and difficult to use, or completely unusable.

Thus, it is important to note that higher resolution does not always mean a more valuable resulting asset or digital twin. In order to understand how high of a resolution scan you want, you need to understand the use case for which you are scanning, and how end users will be interacting with the 3D model. High resolution scans can take substantial processing power and time to create, and the resulting scan can end up being a very large files size on the scale of gigabytes to terabytes of data. These can be expensive to process and store, and can be difficult to work with and manipulate downstream.

One of the largest challenges by industries in 3D content creation is delivering high-resolution content that is also lightweight. The term “lightweight” in computer graphics is commonly used to describe a 3D asset that does not take a lot of computational processing power to visualize. Alternatively, “heavy” 3D models are complex and take a substantial amount of processing power to visualize and manipulate.

Image of two 3D model spheres rendered with lighting and shadows

Image of the same 3D model spheres. The sphere on the right is 16x heavier than the sphere on the left with hardly a noticeable difference when rendered

Having lightweight 3D assets is beneficial for several reasons. They are easier to manipulate and work with, more efficient to render and visualize, cost less to store in data storage, and are proportionally faster to send and receive between applications. Using 3D assets that have been optimized allows 3D experiences to run more smoothly and allows the creators to add additional complexity to the experience elsewhere. These lighter weight assets free up compute resources for other functionality or processing.

It can be difficult to determine the right digitization method for a specific use-case. Important questions to ask include: Are there reflective or shiny surfaces? Is the lighting of the environment changing frequently? Does the solution need to be portable? How large of an area do you need scanned? Will the render happen on the cloud/server or on a mobile device? Is the target object or area indoors or outdoors? How important is the texture resolution? How quickly do you need to scan it? What is your budget? How do you want to use the resulting 3D asset? Is there already an existing dataset?

These questions are necessary to pick the right tool for the job and validate that the resulting assets will meet business objectives downstream and bring value. One powerful way to leverage these resulting assets is within a Digital Twin.

Digital Twins

A Digital Twin is an interactive application that combines 3D models of existing physical facilities and an overlay of additional data to provide insights and enable efficient decision making. Common Digital Twin environments include factories, industrial facilities, and manufacturing plants. Digital Twins have proven to help with building operations, increase production output, and improve equipment performance.

Regardless of which method is used to digitize an object or environment, the result is typically a simplified standard 3D model format. A standard model format is critical because it is typically necessary to bring that model into other 3D software. For any of the previously mentioned industries, Automotive, Aerospace, Real Estate, each have their own industry specific 3D software. These standard model formats can also be brought into other non-industry specific tools such as web applications, Realtime engines, or Digital Twin environments.

The purpose of a Digital Twin is to give users a comprehensive understanding of the asset(s) in subject. The overlaying of additional information on top of the 3D model is critical for exactly that reason. Digital Twins allow organizations to bring together critical pieces of information to a single environment in order to gain a comprehensive view, obtain actionable insights and reach business objectives. If Information is not aggregated from multiple sources, for example a single static 3D model asset, it is not a digital twin. A Digital Twin is a living asset.

A Digital Twin solution, AWS IoT TwinMaker allows developers to easily build Digital Twins of real-world environments.

The “Scene” is a key feature of AWS IoT TwinMaker. The scene is a 3D canvas that allows you to import one or multiple 3D models. These models are organized into a hierarchy panel on the left side that allows you to control visibility of objects and overlay live data points.

Grafana is a key integration to AWS IoT TwinMaker. Grafana is a third-party interactive data visualization application that integrates nicely with AWS IoT TwinMaker. This integration allows you to quickly create charts, graphs, and dashboard layouts leveraging your existing AWS IoT data. It also allows developers to bring in their full AWS IoT TwinMaker 3D scene and pull data from the data streams within those scenes. This allows for a robust and flexible Digital Twin dashboard solution.

AWS IoT TwinMaker 3D viewer with associated live data stream in a Grafana dashboard

Another way to create a similar robust Digital Twin dashboard that can be distributed on the web is by way of AWS IoT Application Kit, or leveraging AWS Partners like Prevu3D.

IoT Application Kit is an open-source UI component library which allows developers to create web applications that integrate with AWS IoT TwinMaker quickly and securely. IoT Application Kit gives developers the flexibility of the web while also allowing the security, scalability, and reliability of AWS IoT TwinMaker and AWS IoT infrastructure under the hood.

KCI – Case Study

KCI is a global professional services firm that focuses on land development, preservation and infrastructure. KCI needs accurate surveys of existing and potential work sites to facilitate their projects. With a focus on infrastructural work, these sites can range in scale. KCI needed to create a pipeline to obtain high resolution 3D models of specific geographic areas. The areas in question ranged from 1 acre to 20 acres in size.

At this scale, typical LiDAR based scanning methods from ground level do not scale well. Ground level LiDAR scans have limits on area sizes. Depending on desired scan resolution, to cover an area larger than an acre, multiple scans must be captured and then combined. This splicing of multiple LiDAR scans together is called the “registration process”. This registration process can be difficult to automate, especially at a large scale, and can become heavy and difficult to use. Another common issue with LiDAR scanning from the ground level over large areas is the perspective of the scans. Depending on the use-case of the resulting scan, with obstacles and interference (trees, cars, hills, and buildings) the resulting assets can be incomplete and unusable.

If images are taken from the sky, by plane, satellite or drone, it can lead to a more complete recreation of the subject area.

Example of aerial based photogrammetry process

A scalable pipeline was built that allows KCI to process images using photogrammetry and receive back a resulting 3D model of the subject area. Benefits of this pipeline include not having to use local computer resources, as engineers need their local machines for other tasks. It also gives the ability to scale up and down resources on demand to never over or under utilize infrastructure, and the data is backed up on the cloud. This pipeline is built to be flexible.

An exciting feature added to this pipeline because of this flexibility, was the integration of Amazon Rekognition. This gave KCI the ability to use machine learning to remove any unwanted aspects of the images sent into the photogrammetry process. For instance, KCI did not want cars in the resulting 3D model. The team was able to use Rekognition to find all cars within the input image set and remove them as a pre-processing step. This allowed a clean resulting 3D scan model without any cars.

Left: Original input image. Right: Illustration of Amazon Rekognition finding all cars to cull from the image.

These resulting 3D model assets can be used throughout the lifecycle of the project for different use cases. These use cases include topographic analysis, feasibility studies, excavation volume takeoffs, project planning, coordination, and visualization. This can lead to improved process efficiency and decreased potential for costly mistakes. With projects at this scale, this impact can be substantial. KCI recorded a reduction of processing time from 2 days down to roughly an hour. When it comes to digitization best practices and understanding industry sentiment, experts in the field can help provide guidance and context.

Reality Capture Network

Reality Capture Network (RCN) is a podcast, online education center, and annual conference that focuses on sharing knowledge and spreading awareness on new technologies and innovation in the built environment. Matthew Byrd, Founder and Host of RCN shares his thoughts and lessons learned below:

“The demand for the digitization of existing assets is experiencing exponential growth. This trend encompasses various industries, from global oil and gas companies seeking to create digital twins of their facilities for simulations, training, capital improvement projects, airports searching for more efficient asset management, security planning, and facility maintenance programs. The use-cases of the technology are vast, and I believe [the world] is just beginning to witness the onset of a significant upward growth curve. It is an opportune time to explore these technologies.”

Matthew expands,

“There are several reasons behind the growing adoption of Reality Capture. Firstly, [the world] is witnessing advancements in both hardware and software capabilities. Simultaneously, the cost of the technology is decreasing, making it better, faster, and more affordable. Additionally, an increasing number of companies are discovering innovative ways to leverage this data, offering advanced analytic tools, drafting and modeling capabilities, and cloud-hosted online viewers. These developments enable firms across diverse industries to utilize the data in ways that were previously impossible.”

Conclusion

As digitization technologies evolve and improve, the benefits continue to grow, and the impact of these technologies has been proven. As the tools become easier to use and can handle more data, they will continue to become more valuable to their associated industries. Decreasing the barriers for users to consume and interact with these assets will greatly increase adoption across all associated industries.

Understanding the different key technologies in the space and how they can be leveraged is important to creating an effective Digital Twin or 3D asset strategy. Picking the right tool for the job to obtain actionable data is as important as knowing how to leverage that data downstream. If the end product does not provide actionable insights, then it was likely not a good investment.

There is a growing number of companies exploring within this digitization space. With hardware and software solutions improving quickly and companies investing more deeply, adoption will follow. AWS is working hard to enable these workloads and help usher the next wave of Spatial Computing.

—–