Leveraging Boomi iPaaS for AWS Integration
Organizations that cling to manual systems often experience a decline in innovation and efficiency. Investing in automation may be identified as an unnecessary particularly when the organization is already accustomed to manual processes, however this approach usually leads to many hidden expenses and loss of potential revenue.
Poor customer satisfaction, data inaccuracies, and frequent delays can ultimately cost businesses significantly more than their perceived savings. Nevertheless, there is a way to reach a balanced solution that doesn't require choosing between basic manual processes and complex automation: Integration Platform as a Service (iPaaS). Using an iPaaS offers a more practical and accessible way to connect various systems without the need for extensive engineering. The right solution provides the following advantages:
Reduced development time compared to building custom integrations
No infrastructure management overhead
Pre-built connectors for the AWS services you rely on
Visual workflow designer for faster implementation
In this article, we’ll dive into a common example scenario where you will learn how to streamline your integration workflows using Boomi, an iPaaS that’s available for AWS users to try free with their AWS account in AWS Marketplace.
The scenario
Imagine you are an enterprise architect who recently joined a growing e-commerce business that sells various products nationwide. The company uses an electronic order management system that processes customer orders and sends those details to the nearest warehouse. The delivery partner is chosen based on how much shipping the user is willing to pay. If the customer chooses the premium option, the package is marked for DelExpress delivery. If the package is heavy, HeavyDels is chosen as a delivery partner. All remaining orders are dispatched via postal service.
Due to the overwhelming demand for push notifications that include the order delivery status, your company implemented a temporary solution. Warehouse operators were asked to take on the additional responsibility of manually extracting and adding order status reports into an Amazon S3 bucket daily.
Nobody is really happy with the temporary solution. Some customers have reported inaccuracies in delivery information, and the overhead caused by the manual entry of data results in downstream impact to other more critical activities that must be performed by the same staff in charge of manually managing delivery data. Management has asked you to architect a solution that solves this problem.
The task is simple: Unify data from multiple delivery partners so the accurate tracking information is available on the website so it can push notifications to the customer automatically.
Current system workflow and architecture
Here is some background on how the current system is set up. The order management system runs on AWS. The website follows a typical serverless implementation pattern. It uses Amazon API Gateway and Lambda for dynamic APIs, static content is served from an Amazon Simple Storage Service (Amazon S3) bucket, Amazon DynamoDB is used for storing orders, and Amazon Simple Email Service (Amazon SES) is used for sending order details to the warehouse. Once the warehouse receives a request, the operator sends the package via an appropriate delivery partner and adds the delivery partner’s details to the order. The operator also maintains an additional sheet with tracking IDs so they can manually add the tracking updates to the order.
Here is how the operator goes about manually extracting tracking details:
DelExpress delivery partner provides a Bulk REST API behind a traditional user-ID/password-based authentication system. The API accepts comma-separated tracking IDs and returns status for all. The operator uploads the response into the Amazon S3 bucket.
HeavyDels delivery partner has an SFTP server. The operator queries the SFTP server to get the tracking details.
Postal Service requires that the operator opens a website, manually enters tracking IDs, and then downloads the PDF generated.
By automating the order tracking-related work that warehouse operators have to do, you can avoid manual errors and provide a better customer and employee experience.
So, how do you automate it? Well, simple – you code. Right? Using APIs and code, you can make the export happen; then you write the logic for retries in case of failures, then you do extensive testing; then you deploy the code, and you schedule it so it runs on a regular basis.
One month – that’s all you would need to automate! Is it really the best way forward, though?
Why traditional (expensive) integration approaches don’t work
When architecting an extensible and maintainable solution, there are many scenarios that must be considered:
Introduction of new delivery partners
Migration and replacement of existing systems
Integration with ERP, CRM and other centralized enterprise data management systems
Eventual integration with Marketing and Content Management Systems
At a typical enterprise, tens or perhaps hundreds of systems need to be integrated. If you build an integration for each, you will need a large, full-time software development team that manages all these integrations. Bugs in any of the systems would get direct leadership visibility due to the nature of these integrations.
Clearly, the solution is not to build but to buy, and the solution is called Integration Platform as a Service (iPaaS).
Introducing Integration Platform as a Service (iPaaS)
iPaaS provides all the tools and infrastructure needed to build, deploy, and manage integration flows between different applications, data sources, and business processes. With iPaaS, you do not have to write custom code. Instead, you use prebuilt connectors and visual tools to build integration flows. For example, if you need to sync order data from your e-commerce platform to your warehouse management system, you can use iPaaS's drag-and-drop interface to create a flow. The iPaaS platform handles all the complexities, such as authentication, data transformation, errors, and retries, for you. It supports business-to-business (B2B) and application-to-application (A2A) workflows. Its low-code/no-code development approach makes it simple to use and deploy.
To meet an enterprise organization’s complex iPaaS requirements, you need a solution that supports:
Leading systems
Provides diverse, large numbers of prebuilt connectors
Native AWS support, meaning integration with multiple AWS services
Compliance with industry standards such as Service Organization Control (SOC2), General Data Protection Regulation (GDPR), and Health Insurance Portability and Accountability Act (HIPAA)
Exploring the capabilities provided by an iPaaS solution
While researching solutions, the Gartner Magic Quadrant for iPaaS is a good place to start where you’ll find Boomi, which has been in the Leader’s quadrant for the past eight years. Boomi supports hundreds of prebuilt connectors for multiple systems and offers simple integration, which you will see in a minute. It also supports a rich ecosystem of AWS services. As shown in the following diagram, Boomi supports popular architectural components such as Amazon S3, Amazon Simple Queuing Service (Amazon SQS), Amazon Simple Notification Service (Amazon SNS), Amazon EventBridge, Amazon Relational Database Service (Amazon RDS), and Amazon Redshift.
You decide to do a quick proof of concept (POC) to explore Boomi’s capabilities and how it may help keep your customers happy with more accurate tracking notifications. From the documentation and some how-to videos, here are some basic concepts you learn:
A Boomi Integration simplifies the process of combining various software and hardware components into a unified system with a user-friendly visual interface and tools to connect applications and data, whether they're on-premises or in the cloud. At the core of an integration is a Boomi Process.
A Boomi Process is a series of steps that define the workflow and integrations between systems.
Boomi Steps are building blocks of a process. There are three types of steps:
Connect: These steps retrieve data into the process or send data out of the process. These steps establish a connection between systems. For establishing a connection, Boomi provides built-in connectors such as Amazon S3 connector and an Amazon Redshift connector.
Execute: These steps manipulate data. For example, a step might perform functions like data mapping or data transformation.
Logic: control the flow of data through the process. For example, a logic step might handle branching, exception throwing, or process control.
In short, you combine these steps to build a process based on your requirements. To build a process, Boomi provides a user-friendly canvas, and once you have built it, you use a Boomi Atom to execute and test the Boomi process.
Boomi Atom is a lightweight execution environment that runs integration processes. It contains all the necessary components for process execution. There are two types of atoms:
Local Atom: Used for connecting to local systems or those with strict firewall rules.
Cloud-based Atom: Used when all applications are internet-accessible.
Molecules are clusters of Atoms used for intensive tasks.
Boomi Environment: A workspace for testing, deploying, and executing integrations. Typically, separate environments are created for testing and production.
Packaging Manager: Creates and versions deployment packages. After packaging, the software is deployed to the environment.
Okay, now that you are up to date on the basic Boomi concepts, you are ready to do a POC to see if you can automate tracking data download from DelExpress. Since the DelExpress API endpoint is behind auth, your POC will test these two requirements: 1) download data from an API behind auth and 2) upload data to an S3 bucket.
Build a Boomi Integration that downloads data and uploads it to Amazon S3
Step 1: Create a Boomi Process
You first create a Boomi Account and then create a new Boomi Process. The UI displays a visual editor where you configure your start step.
Once you create the start step, to add a step, on the canvas, you choose the + icon.
Step 2: Connect to DelExpress to GET a file
Since your process should download a file using a REST API, you choose to configure the HTTP Client connector and call it DelExpressConnector. Here is a screenshot of the configuration screen of a connector that has a URL and authentication.
Once you have configured the connector, you can choose an Action. Since you are downloading a file from an endpoint, you select GET and, under Operation, configure request parameters.
Step 3: Connect to Amazon S3 to upload a file
Step 3.1: Create an Amazon S3 connector step
Next step is to upload the data into the data lake in S3.
To add an Amazon S3 connector step, from the canvas, you choose the + icon and information such as name, etc.
Step 3.2: Configure AWS authentication
Boomi needs to connect to your AWS account, so under Connection, you configure an access key and secret access key so the process can connect to your AWS account.
Step 3.3: Configure S3 bucket into which data will be uploaded
As part of the Amazon S3 bucket operation, you define the S3 bucket into which object needs to be uploaded, including the type of operation and encryption.
You add a Stop process to end the workflow. Now, you are ready to run your workflow. To run or test a process, you create an Atom by following this process to create a local Atom on your computer.
Finally, you run the process on the local Atom based on your computer, and when you open the Amazon S3 bucket, you see the tracking file.
To summarize: in a matter of hours, you set up an Integration using Boomi that downloaded the latest tracking report from DelExpress and uploaded it into your data lake in Amazon S3. You also learn that Boomi has connectors such as SFTP V2 connector that can be used for extracting data from HeavyDels SFTP server. In short, with Boomi, you just automated an end-to-end integration for tracking and sending notifications for orders. Here is how the final architecture looks.
With this integration, all manual processes are eliminated. Your customers receive accurate tracking updates for their packages, whether they are shipped using DelExpress, HeavyDels, or government post.
Key takeaways and next steps
An iPaaS solution such as Boomi can help your business streamline and automate data integration workflows. This ability enables you to automate manual tasks and easily unite data flowing in from different systems into your data lake. By implementing Boomi as your integration platform, you can reduce errors and improve efficiency. Once data has landed in your data lake, you can use powerful services provided by AWS to extract intelligent insights and add new AI/ML-backed features to your applications.
If you're starting out, here are usual first steps that can get you moving in the right direction:
Identify manual dependencies that are slowing down the rest of your processes.
Use Boomi Connectors to identify integrations you can use to automate manual tasks.
Use Boomi pre-built templates to accelerate your development.
Use Boomi monitoring capabilities to ensure your integration flows perform optimally.
To get started, try Boomi Integration Platform as a Service (iPaaS) free with your AWS account in AWS Marketplace.
Why AWS Marketplace for on-demand cloud tools
Free to try. Deploy in minutes. Pay only for what you use.