HERE Technologies Scales Up Binary-Artifact Repository Using Amazon EFS

2017

Today, consumers and businesses alike take digital maps and navigation services for granted. That wasn’t the case in 1985, when HERE Technologies was founded. The company has since become a world leader in location services. It powers four out of five in-car navigation systems in North America and Europe, enterprise-grade fleet-management solutions, and the real-time traffic information utilized by transportation agencies across the world. The company is now building the HERE Reality Index and HERE Open Location Platform, which will enable all industries to access and contribute to a digital location index of the physical world.

“We are looking far beyond traditional location data and mapping to support emerging use cases such as the Internet of Things, mobile payments, and autonomous vehicles,” says Suresh Prem, principal systems engineer at HERE. “Our IT team must be agile and adaptive to keep up with the pace of innovation.” In the company’s constant search for improved DevOps self-service and automation, it has turned increasingly to Amazon Web Services (AWS).

This is part of a broader cloud-first strategy for HERE Technologies. “Since moving to the cloud, our time-to-market has greatly improved,” says Prem. “When we were largely using on-premises data centers, it took a minimum of three months to set up development infrastructure. Using AWS, teams can roll out the environment they need in one week—without having to get storage and networking teams involved."

start a python tutorial
kr_quotemark

Prior to using Amazon EFS, we were experiencing timeouts for 5–10 percent of uploads over 100 MB. Now, we have zero timeouts, which is a huge win.”

Suresh Prem
Principal Systems Engineer, HERE

Keeping Artifact Management under Control

Managing software artifacts for hundreds of development projects is a massive effort. “Initially, development teams at HERE used a range of third-party and homegrown tools for binary repositories,” says Murty Chitti, principal systems engineer at HERE. “This impacted our ability to manage build artifacts effectively and get products to market quickly. To address this problem, we chose JFrog Artifactory to centralize storage of binary artifacts, starting with an on-premises version running in our own data center.”

It wasn’t long before the company began considering a move to AWS. “We found Artifactory challenging to run on premises,” says Rajesh Sivaraman, senior systems engineer at HERE. “We were continually needing to deploy new hardware to keep up with demand, so we decided to migrate our Artifactory deployment to AWS.”

Storage Innovation Drives Efficiency

Artifactory runs on Amazon Elastic Compute Cloud (Amazon EC2) instances. For storage, the company initially kept the Network File System (NFS) storage model it was using on premises, using Amazon Elastic Block Store (Amazon EBS) for storing data. “We found ourselves having to add volumes every week,” says Chitti. “We wanted to eliminate that management burden.”

The team also wanted to improve the performance, availability, and disaster-recovery capabilities of the solution. “The way our solution was architected, NFS was a single point of failure,” says Prem. “Additionally, teams were having timeout issues when uploading large binaries.”

When Amazon released Amazon Elastic File System (Amazon EFS), HERE saw an opportunity to address these issues all at once. Amazon EFS provides a simple, scalable, fully managed file system for use with Amazon EC2. “Because Amazon EFS uses standard NFS protocols, integration with our Artifactory solution was seamless,” says Chitti. Using Amazon Virtual Private Cloud (Amazon VPC), each team can set up its own accounts and easily connect to Artifactory, which runs in its own Amazon VPC.

The solution also uses Amazon Relational Database Service (Amazon RDS) for database functionality, Amazon CloudWatch for monitoring, and Amazon Simple Storage Service (Amazon S3) for retaining data long-term.

Multiple Amazon EC2 instances can access an Amazon EFS file system at the same time. “Because Amazon EFS presents a unified namespace to all instances, we don’t have to manage the individual DNS names for each storage volume, which greatly simplifies administration,” says Chitti. “In addition, even if a server terminates, Amazon EFS persists the data for use by other servers.”

Eliminating Timeouts, Reducing TCO

Adopting Amazon EFS has eliminated timeout issues. “Prior to using Amazon EFS, we were experiencing timeouts for 5–10 percent of uploads over 100 MB,” says Prem. “Now, we have zero timeouts, which is a huge win. Production deployments depend on the data stored in Artifactory, so it needs to work as expected every time.”

In fact, the solution is faster than ever, according to HERE’s own benchmarks. The speed of transferring a 1 GB file from a cloud-based build server to Artifactory has increased by 38 percent since adopting Amazon EFS. For similar uploads from on-premises tools, speeds have increased by 33 percent. Features such as AWS Direct Connect and Amazon VPC peering optimize connectivity for faster performance.

With a centralized binary-artifact repository in the AWS Cloud, HERE developers can collaborate effectively and get new innovations to market faster. “Artifactory and Amazon EFS enable us to support a wide range of use cases globally,” says Chitti. “It doesn’t matter if we build in Europe and test in the United States, or which build tools or formats we use—all our build artifacts are in one place, from infrastructure-as-code components to Docker images.”

The elasticity of Amazon EFS has helped HERE reduce total cost of ownership compared to managing its own NFS servers and storage. “Using Amazon EFS, we don’t have to spend time or money overprovisioning to meet growing demand,” says Prem. “We use exactly as much as we need, and we only pay for what we use.”

The company has also implemented disaster recovery by replicating its environment in another AWS region. In case of a regional disaster, the company can simply change the DNS and be up and running again.

HERE uses Amazon EFS to store 1.2 million artifacts in more than 300 repositories, adding up to more than 20 TB. The system serves 750,000 downloads and 50,000 uploads each day from more than 1,000 users and continuous-integration systems. “Because of our success with AWS so far, we are planning to move all feasible DevOps tools in that direction,” says Sivaraman. In the rapidly evolving world of digital location data, AWS is helping HERE get to its strategic destinations faster than ever.


About HERE

HERE Technologies enables customers to harness the power of location to achieve better outcomes. Its services help cities manage infrastructure, empower enterprises to optimize their assets, and guide drivers to their destinations safely. The company has 8,000 employees in 200 offices and 54 countries.

Benefits of AWS

  • Increased upload reliability for large files while increasing speeds 38 percent
  • Centralized binary-artifact repository for development teams worldwide
  • Easily implemented disaster-recovery configuration
  • Stores more than 20 TB of data in the cloud
  • Reduced time to set up infrastructure from 12 weeks to 1 week

AWS Services Used

Amazon EC2

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers.

Learn more »

Amazon Elastic Block Store

Amazon Elastic Block Store (EBS) is an easy to use, high performance block storage service designed for use with Amazon Elastic Compute Cloud (EC2) for both throughput and transaction intensive workloads at any scale.

Learn more »

Amazon EFS

Amazon Elastic File System (Amazon EFS) provides a simple, scalable, fully managed elastic NFS file system for use with AWS Cloud services and on-premises resources.

Learn more »

Amazon VPC

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the AWS Cloud where you can launch AWS resources in a virtual network that you define.

Learn more »

Amazon RDS

Amazon Relational Database Service (Amazon RDS) makes it easy to set up, operate, and scale a relational database in the cloud.

Learn more »

Amazon CloudWatch

Amazon CloudWatch is a monitoring and observability service built for DevOps engineers, developers, site reliability engineers (SREs), and IT managers.

Learn more »

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance.

Learn more »

AWS Direct Connect

AWS Direct Connect is a cloud service solution that makes it easy to establish a dedicated network connection from your premises to AWS.

Learn more »


Get Started

Companies of all sizes across all industries are transforming their businesses every day using AWS. Contact our experts and start your own AWS Cloud journey today.