HyperTrack: Managed Service for Live Location Tracking at Scale

By Alexander Kishinevsky, VP of Engineering, HyperTrack

Alexander Kishinevsky, VP of Engineering, HyperTrack

We live in a world where it is easier than ever to build software, but harder than ever to operate it. Let me explain.

Suppose you are building location-aware applications to make the mobile workforce more productive and logistics networks more efficient. To do this, you would start by getting the live location from a bunch of devices and then show it to the customer on a map in real-time. But in order to do that successfully, you also need to operate complex infrastructure to ingest, process, store, provision, and manage this data. The process gets even more complex as you go from lab to production, scale up devices in production, and add more use cases to service various teams that want to use this data. Before you know it, you are spending valuable engineering resources managing the infrastructure that you wish just worked.

Enter HyperTrack. When we first started the company in late 2015, we spun up Heroku instances for a Django-Python app built as a monolith. We made the mistake of building an API product as an application and later moved the party to Amazon EC2 and Amazon RDS instances, and de-constructed the monolith to a microservices architecture. The bills racked up and our engineers were spending time managing servers, databases, deployments, and migrations. We made the mistake of operating our own infrastructure instead of focusing on building the product.

In early 2019, our Architect Thomas Raffetseder and I started building a serverless architecture with AWS managed services. The idea was to build a platform that scaled up and down while we were asleep, even as customers from across the globe ramped their usage up and down at whim. This involved high-precision craftsmanship to wire up hundreds of resources from across a dozen of AWS managed services – Amazon API Gateway, Amazon Kinesis, AWS Lambda, Amazon DynamoDB, Amazon DynamoDB Streams, Amazon SNS, Amazon RDS, AWS Glue, AWS Step Functions, Amazon S3, Amazon Cognito, Amazon Athena, Amazon CloudWatch, and AWS AppSync. This complexity is only manageable if we have infrastructure-as-code that team members with remote offices could automatically deploy and build on top of. Therefore, we built IaC and continuous integration/deployment in parallel.

Read more about our serverless architecture in this blog post that trended on Hacker News earlier this year.

The complexities of the device-to-server infrastructure were known, but the complexities of server-to-frontend were evasive.

We knew that millions of devices would stream up time-series data from our SDK to servers that ingest this data reliably, process the peculiar data set for accuracy, store processed data in real-time data stores for applications to consumer, and archive it in a data lake for analytics. Applications would use this real-time data through REST APIs, webhooks, and embeddable views.

We operated under the assumption that getting the data into the platform is the hard part and discovered in the last days of each release that getting data out to the front-end was more complex than we gave it credit for.

We could use sockets to push data to the front-end, and private APIs for the front-end to query at load. Or we could manage our own GraphQL servers to streamline the queries and subscriptions between the platform and front-end. The front-ends could be native or hybrid mobile apps. They could be web apps used on mobile or desktop. Widgets built by HyperTrack might be embedded by customers as-is, or used as open source libraries that are customized by customers to build their own app experiences. There were more questions than answers and our heads got a little dizzy. It was not fun to run into this architecture challenge later in the cycle with deadlines staring us in the face.

To tackle this, I took a bite off the falafel wrap with my left hand and searched Google for “connect dynamodb with react graphql” with my right. Out came AWS AppSync. The next ten minutes at the lunch table swung between excitement, fear, doubt, relief, and curiosity. Fortunately, curiosity dominated over the other emotions and after a few hours of playing around, we knew that AWS AppSync would be the answer, at least for now. The data was sitting there in Amazon DynamoDB and the React components were sitting there in the front-end, looking for GraphQL schemas to drink from. A managed service to do just that was sitting in a different room of the same house that we lived in. We flipped that door open.

Six months later and with our fair share of teething pains, we have a dozen front-ends integrated into a diverse set of applications in the hands of millions of users. Initially, AWS AppSync was difficult to learn, adopt, and troubleshoot. We spent countless late nights figuring out how to set up Terraform as our primary IaC framework to help create and maintain schemas, various data sources types, and resolvers. However, the challenge of operating our own GraphQL servers at scale to achieve the same would have been much higher and consume much of our precious time.

Overall, going serverless is 25-30% cheaper per unit than the time we managed our own Amazon EC2/RDS. In hindsight, the extra cost was due to over-provisioning that led to more costs than we were utilizing, or under-provisioning that led to reactive logging and sudden scale up when servers got choked. The bigger win is that our customers are happier even as our 50+ man hours per month on operating servers has now shrunk to near zero. There are 99 problems that keep us up at night but operating servers ain’t one.

We live in a world where it is easier than ever to build software, but harder than ever to operate it. Building with AWS managed services helps us focus on building our product while leaving the compute, store, and real-time messaging infrastructure to them—just as building with HyperTrack helps our customers focus on building their product while leaving the live location to us.