Indigo Ag Uses Amazon Aurora to Scale Sustainable Agriculture Solutions

Indigo Ag (Indigo) is an agricultural technology company that enhances farmers’ ability to sustainably produce crops, increase their profits, and improve consumer health using microbiology and digital tools. Founded in 2014, the startup was named number one on CNBC’s Disruptor 50 list in 2019. As Indigo grew rapidly, it launched additional self-managed microservices to match demand. As the company worked toward finding a product-market fit, engineers needed to keep the boundaries of the company’s services permeable, rather than locking themselves into premature optimizations.

Born exclusively on Amazon Web Services (AWS), Indigo decided to stay in-house and migrate to Amazon Aurora, a fully managed MySQL- and PostgreSQL-compatible relational cloud database with the performance and availability of commercial-grade databases at 10 percent the cost. On Aurora, Indigo enabled its grain market, Indigo Marketplace, to grow and provide high availability and reliability to farmers and buyers who depend on the service for six-figure transactions.

In September 2018, Indigo introduced Indigo Marketplace, a technology service that brings key stakeholders together for a more efficient way of selling grain; for example, growers gain access to a larger market and increase profits while buyers can more easily source the type of grain they want. “Grower profitability is near all-time lows,” says Andy Luther, senior director of software engineering at Indigo. “We’re giving growers a central place to transact crops more efficiently, manage risk, and make more money in the process.”

To support Indigo’s fast-paced growth, its DevOps teams began building new applications and standing up microservices in the process. At Indigo, DevOps engineers provision all data resources, giving product teams the adaptability and agility to deliver on customer needs. Because each Indigo team builds and operates its own services, in short order, the startup was managing many separate microservices, each with its own database—a siloed structure that would create more work for the DevOps team as the company grew. “We had a very small development team, and it spent a lot of time writing or needing data that existed in other services,” says Luther. “We either had to write APIs just to consume that data or do things like reach into read replicas and query the raw data itself—a big no-no.” 

Consolidating to a single database cluster would enable Indigo to make better architectural decisions as the business scaled. By migrating to Amazon Aurora with PostgreSQL compatibility, which is up to three times faster than standard PostgreSQL databases, Indigo could run its existing PostgreSQL applications and tools without modification. “That kept our velocity going without pulling everything apart and making us start from zero,” says Luther.  

Indigo was particularly attracted to Aurora for its automatically scaling storage. “Databases get bigger,” says Steve Huff, senior software engineer of DevOps at Indigo. “If your storage doesn’t scale, you’ve signed up for a never-ending operational treadmill of adding more and more storage. A huge added value of Aurora is that we don’t have to worry about that problem.”

In the first quarter of 2020, Indigo created a proof of concept and a slow rollout strategy. Indigo prioritized the migration of high-impact services, first moving its most interconnected service—a database containing user metadata that most of its apps share—to achieve the greatest benefit with the least effort. It took 4 months to fine-tune the migration procedure; once the teams hit their stride, however, they migrated five more services over the course of the next 4 months. “All scoped migrations have been successful and have improved over time,” says Kelsey Steinbeck, director of software engineering for DevOps at Indigo. “Now they’re no-ops: the last one, a product migration, took about an hour and a half. It was boring, which is great.” As of November 2020, just under half of Indigo’s services have migrated to Aurora. 

Indigo’s Marketplace service has seen 20–30 percent month-over-month growth in 2020, and the high availability offered by Aurora keeps it always online. The marketplace can also reliably scale to handle spikes in user volume. For example, when prices change, Indigo Marketplace sees large batches of transactions in a short period of time. Since September 2018, the service has processed $1 billion in cumulative transactions. “We need to be reliable enough for folks to stake their livelihood on, so we need to have reliable data storage and fix issues quickly,” says Huff. “That’s why we need to work on problems that are higher up in the stack rather than struggle with our database. The value we get out of AWS is our ability to work on problems that we are best suited to fix.” The low-latency read replicas offered by Aurora further boost service scalability, enabling services to offload read queries. 

Using Aurora, which is fully managed by the Amazon Relational Database Service (Amazon RDS), Indigo’s DevOps team can outsource to AWS various administrative tasks it doesn’t have the bandwidth to complete. This had especially become a growing pain as the startup’s separate self-managed databases increased in number. But the DevOps team is now able to focus on enabling developers to ship features faster by putting more focus on the internal software development lifecycle tooling and process. When Huff arrived at Indigo, the company was using PostgreSQL 9.6.11 even though PostgreSQL 11 had just been released. “It should not be the job of our application developers to stress over major database upgrades,” says Huff. “That’s not the best use of their time and energy. Consolidating on a single system enables us to do the painful thing once as opposed to 20 small times.” Hosting the company’s data on Aurora makes it easier for Indigo to meet regulatory and compliance obligations. “We don’t need to worry about making sure everything is compliant across databases,” says Steinbeck. “Aurora gave us an easier operational model for our small DevOps team." 

Indigo’s big win from this migration is its ability to maintain separation of concerns among its services’ data stores—by means of individual PostgreSQL schemas—while also having the ability to easily share data among services with fine granularity and control by means of structured query language (SQL) views. Data owners have precise control over the data that is exposed to other services. When it comes time to firm up service boundaries, the process of converting from data sharing through views to data sharing through APIs will be more straightforward because the contracts between services have already been captured in the definition of views. 

Now Indigo’s development teams can take advantage of cross-schema join functionality, which enables a developer to look up a piece of information across Indigo’s many services without having to implement new APIs. For example, the agricultural industry revolves around the crop-trading calendar. Before consolidation, each Indigo service had metadata noting these dates separately, whether in its application code or database, and sometimes the dates didn’t align between different services’ local metadata. Now Indigo has a centralized shared dataset that everyone can access rather than many versions of contradictory data. “With a single line, we can hop in and get read access to this data,” explains Luther. “It makes things that should be easy actually easy.”

Indigo has adopted an all-in Aurora strategy and plans to use Aurora as the database backend for its current and future applications. Its current seasonal focus is Indigo Carbon, a program that supports farmers in adopting beneficial growing practices that reduce farming emissions, restore soil health, and remove carbon dioxide from the atmosphere. By adopting beneficial farming practices, farmers can improve the health of agricultural soils to capture and store carbon dioxide from the atmosphere and earn additional income through verified agricultural carbon credits. 

On Aurora, Indigo achieved scalability that has enabled month-over-month growth on its marketplace while decreasing the amount of work for its small DevOps team and enabling its product engineers to focus on delivering business value. Now Indigo can better focus on its critical pursuit of helping farmers and others across the agricultural ecosystem sustainably feed the planet.