AWS Partner Network (APN) Blog
Optimize Spatial Data Management and Analytics with Ellipsis Drive and Amazon S3
By Rosalie van der Maas, CEO & Co-Founder – Ellipsis Drive
By Akshay Modak, Digital Marketing Manager – Ellipsis Drive
By Frank Mullerat, EMEA Partner Solutions Architect – AWS
 |
| Ellipsis Drive |
 |
Let’s start off with some hard truths. Having spatial data (any data type that has a location or spatial component) at your disposal means you’re faced with a data management problem.
What’s worse, when you deliver spatial analytics to clients, you are invariably creating a data management problem for them, too! Sure, the scale of the problem varies depending upon your proactiveness in building a structural solution, but only a handful of companies (at best) can confidently say they are acing the spatial data management conundrum.
What’s the crux of this challenge?
- Pain point 1: No scalable and automated ingestion of spatial data into a unified and standardized data lake or warehouse.
- Pain point 2: No scalable and interoperable spatial data searchability, indexability, and usability in downstream analytics workflows and products.
- Pain point 3: No on-demand, high performance, and real-time spatial data rendering.
So, what are the implications of these pain points from a quantitative standpoint?
- On operational workforce alone, spatial data-driven enterprises with 1,000+ employees have an estimated overspending of $2-6 million on spatial data management annually, according to Ellipsis Drive research. This represents 15-45 full-time employees (FTEs) doing in-house data management grunt work).
- Third-party data deployment and system integration services ($1-3 million).
- Unnecessarily high opportunity cost as a result of sub-optimal productivity of data science teams.
In this post, we will share how you can mitigate these issues by building a fully scalable spatial data infrastructure with Ellipsis Drive and Amazon Simple Storage Service (Amazon S3). This solution helps optimize spatial data operations and create value across the entire ecosystem/data chain.
Ellipsis Drive is an AWS Partner and AWS Marketplace Seller that’s a fully interoperable, cloud-based, spatial data management solution that simplifies and automates spatial data transformation, management, and integration. It provides fast and secure access to spatial content from any workflow by converting uploaded geodata files into live maps and web services that can be accessed and queried via your endpoints of choice.
Simplified and Automated Spatial Data Management
While Amazon S3 is a great tool for storing data, hosting data lakes for analytics and backups, its functionality is limited when it comes to working with spatial data. To successfully host spatial data, which means making it searchable and usable for any downstream audience, you need a solution that’s specialized to manage your existing spatial data S3 bucket.
You can launch Ellipsis Drive on top of an existing S3 bucket, which allows you to solve your data management problems without any data migration. On launch, files will be added to the Ellipsis Drive index and ED will from that moment on act as your bucket manager. This means you can address the above pain points and take advantage of full functionality without manual restructuring or re-indexing.
Figure 1 – Spatial data ecosystem.
Ellipsis Drive features include:
- Instantly render spatial data in endpoints of your choice, such as ArcGIS, QGIS, Power BI, and Folium via simple plugins.
- Access spatial data as open geospatial consortium (OGC) web services, tiles services, and via SpatioTemporal Asset Catalog (STAC)-compliant API.
- Scale from 1MB to infinity at log-n performance on all fronts (processing, rendering, sharing).
Many analytics tools can be deployed in conjunction with Amazon S3 and offer high-performance and cloud-native data warehousing, thus making them essential choices for diverse data analytics needs. However, they have one thing in common—they’re not tailored for the ingestion and management of spatial data. Thus, in order to broaden your solution suite within AWS and adhere to the requirements of hosting and using spatial data, you can deploy Ellipsis on top of your data lake/buckets.
The figure below shows what your network can look like when deploying fit-for-purpose data management solutions on your data lake. Here, Ellipsis Drive functions as your spatial data warehouse while Amazon Redshift may function as your non-spatial data warehouse, and both are connected via Python, API, or plugin.
Figure 2 – Fully scalable spatial infrastructure using Ellipsis Drive.
Business Impacts of Building Scalable Spatial Infrastructure
It’s important to establish the fundamental benefits this integration has to offer to a company’s workflow. Below is a summary of the positive impacts of building out a fully scalable spatial data warehouse:
- Time efficiency: +95% time saved on data management and transformation by automating spatial data ingestion, structuring, and integration.
- Data querying: 100X faster data querying by using patent-pending tile tree archives and paged vector files, providing high-performance spatial data use for data scientists, modelers, and developers compared to existing workflows.
- Seamless access: Instant access to data to high performance spatial data to feed your modeling and apps.
- Interoperability: Instant spatial data transmission from one team or organization to another.
- Scalability: 100% scalable spatial data ingestion and management pipeline for existing and new 2D/3D spatial data vendors.
With this understanding, let’s take an industry-specific approach and see how this solution translates into tangible outcomes.
Property and Casualty Insurance
By optimizing spatial data ingestion and management, insurers can efficiently process spatial data and include this in their (climate) risk modeling and claims processes, enabling more accurate risk assessment (reducing over- and under-exposure), claims processing, and fraud detection. This can lead to cost savings, better underwriting/risk pricing, and improved customer service.
Earth Observation and ClimateTech
The integration of Ellipsis Drive and Amazon S3 aids the Earth observation (EO) and ClimateTech industry by centralizing and scaling storage of vast datasets, ensuring real-time data handling, and providing robust data security. This empowers EO missions to efficiently process, host, and analyze real-time and historical data. It also streamlines climate modeling, trend analysis, and the development of sustainable solutions.
The integration reduces risk of churn by taking care of the last mile of delivery and ground operation services for EO satellite operators by ensuring scalable delivery and seamless consumption of data.
User Personas That Benefit from the Integration
The greatest benefactors of Ellipsis Drive with Amazon S3 are end users that deal with spatial data on a regular basis. The asset that’s being treasured here is time, and freeing up a company’s FTEs by relieving them of their data wrangling duties using Ellipsis Drive results in a return on investment (ROI) in three months.
Here’s a quick representation of how this calculation works:
- Invest 0.2 FTE for weeks 1-3 (Ellipsis Drive team takes care of setup and configuration)
- Invest 0.3 FTE for weeks 4-6
- Break even on your FTE allocation at the start of third month
- Save around 50% on all spatial data engineering, data science, and analytics time from month four onwards
Below is a deep dive on the specific personas this integration impacts:
- Data scientists: 70% faster project completion rate for data scientists. By automating data preparation and wrapping the API with Python and R packages, data science teams can focus on analysis right away while using tools of choice.
- Developers: Developers can flexibly render vector and raster data in software with a single line of code. The result is that all spatial data are made available on a drag-and-drop basis or simple command line basis for high performance and interoperable use by modelers, data scientists, engineers, and analysts via Python, R, API, GIS software, and Power BI.
- GIS specialists: GIS experts can set up a fully scalable and interoperable cloud-native GIS database that structures and indexes spatial data in a single upload. As a result, GIS specialists are unburdened from database creation and maintenance.
Architecture Overview
Ellipsis Drive is deployed on top of your existing file storage (Amazon S3 in this case). The solution’s infrastructure takes care of access management, querying, interoperability, and scalability. Below is a comprehensive depiction of the architecture and implementation of ED on top of S3.
Figure 3 – Ellipsis Drive and Amazon S3 integration.
Conclusion
The challenges posed by spatial data mismanagement can be daunting, and their repercussions on operational costs and workforce efficiency are severe. By seamlessly integrating Ellipsis Drive with existing Amazon S3 buckets, organizations can bypass the hurdles of data migration whilst unlocking countless cascading benefits.
The optimized spatial data warehouse yields time savings in data management and transformation, empowering the data science team with a significant boost in data querying speed. In the realm of spatial data, where time is a treasured asset, the integration of Ellipsis Drive and S3 can revolutionize the entire spatial data ecosystem.
If you’d like to start a conversation about how Ellipsis Drive can be tailored to fit your business needs, schedule a free demo. You can also learn more about Ellipsis Drive on AWS Marketplace.
Resources:
.
.
Ellipsis Drive – AWS Partner Spotlight
Ellipsis Drive is an AWS Partner and fully interoperable, cloud-based, spatial data management solution that simplifies and automates spatial data transformation, management, and integration.