AWS Partner Network (APN) Blog
Best Practices from Pragma for Navigating the API Economy with Observability and AWS
By Nicolás Ricardo Archila Gómez, Architecture Chapter Leader – Pragma
By Omar Ashton, Partner Solutions Architect – AWS
By Neha Wadhwani, Partner Solutions Architect – AWS
 |
| Pragma |
 |
The term “API economy” refers to businesses delivering digital services to end users, other company services, or partners. But what is the API economy?
You could use APIs to improve productivity with services already built and exposed by other companies, or you could create internal services and expose them through APIs to help you reuse these services, reduce coupling, and improve maintainability. You could also develop services and expose them to a general audience or partners who need them.
All of these options could be delivered via API-as-a-service, which gives you financial benefits and strategic advantages. We call this the “API economy,” which allows you to establish many business models to develop and publish APIs.
This post discusses the necessary aspects to achieve an observability model in the API economy, including a practice example with an architecture design and related technologies.
Pragma is an AWS Advanced Tier Services Partner and AWS Marketplace Seller with Competencies in DevOps and Digital Customer Experience. Pragma has worked with big retail, banking, and insurance companies in LATAM for over 30 years. Its approach is to work alongside organizations and stakeholders to assess their business and technology needs and recommend tailor-made solution roadmaps that are focused on the end-user experience.
Exploring the API Economy: API-as-a-Service
API-as-a-service can be consumed by a company’s internal services to transform and add value before going to end users or a partner who, having the required user authorization, works with that information and delivers their value proposition to end users.
Amazon Web Services (AWS) is one of the examples of how API economy can be established. AWS uses APIs to expose core service capabilities and customers, partners, and internal AWS employees make use of these APIs, creating an API economy.
A little background on how it came into existence. As Amazon was growing, the team started transitioning to a service-oriented architecture and built services in a decoupled, API-access fashion. Internal teams inside of Amazon were expected to be able to consume peer internal development team services through these APIs. As more developments were made, AWS made the APIs available to external users as well.
To have a thriving API economy model, a well-defined observability architecture is required to optimize the business or improve external quality attributes. Here are some considerations that should be taken into account:
- Percentage of successful and failed transactions.
- Number of requests by second, minutes, hours, months, or year.
- Configuration changes and their impact on the architecture.
- API consumption plan per customer.
- Infrastructure cost.
- Most and least used APIs.
- When an error happens, detect the error, its cause, and how to solve it.
Enabling the API Economy
Based on Pragma’s extensive experience, consider the following steps to establish an API economy model.
Step 1: Understanding and Definitions
Conducting thorough market research and understanding the target consumer is crucial to create an API economy model. Define the value proposition of the API while keeping in mind legal and regulatory restrictions.
API architecture should consider business goals, processes, technical restrictions, and the current architecture context. Quality metrics should be assessed to ensure the API is reliable, secure, and efficient.
Establishing a propagation and deployment strategy includes identifying the first customers who will consume the APIs and determining how they’ll do so. Careful consideration should be given to the load the API will receive, taking into account the projected customer base. By projecting the load, it’s possible to ensure the API can handle the expected volume of requests without experiencing performance issues.
Step 2: Business Model Selection
When devising an API economy model, consider diverse business models that can be utilized to maximize profits and benefits. Here are some of the most significant ones that have proven effective in the industry:
- Free: The benefit is not monetary. It could be strategic, as when the company wants end users to take advantage of their service while tracking how they use it.
- Freemium: Users can consume the service for free with some limitations and then pay for added features. For example, the company delivers a free tier for users to see their credit card expenses, but if users want to create a budget that sends an alert when they exceed it, they have to pay an additional cost.
- Subscription: Customers subscribe to a standard service that could have extra payment options if the limit is exceeded. For example, users could request to know their credit score three times every month, but if the number of requests exceeds this limit they have to pay for the additional requests.
- Pay-per-use: The price is directly based on usage, with a specific driver such as the number of requests to the API or MB consumed. Depending on their quantity, users pay a unit/value.
- Revenue share: There could be a partnership between two companies, and the profits related to the API usage are distributed depending on the number of accesses to the service.
Step 3: Design Integration and Observability
Prioritize designing an architecture that lays the foundation for building a scalable and reliable API economy. This involves creating a cohesive structure, defining design patterns, and adopting best practices that enable seamless communication and data exchange between components.
Pragma uses dashboards that offer business and technical perspectives to ensure it can observe all APIs. These dashboards allow Pragma to analyze data and take actions related to service pricing and API consumption.
A critical aspect of this architecture is observability design, which focuses on gaining insights into the system’s behavior and performance. Observability involves configuring services and APIs to collect logs, metrics, and traces which can be analyzed to detect issues and make informed decisions.
An example of how APIs can be shared is AWS Data Exchange, which allows customers to find third-party datasets. The exchange happens through Amazon API Gateway which integrates with provider APIs, allowing data providers to distribute and monetize API-accessible data and subscribers to find, subscribe to, and use third-party APIs in the cloud.
It’s imperative to know which API you’ll expose and who will use it; such as:
- Public API – exposed to the general public.
- Private API – exposed to the company’s internal users.
- Partners API – exposed to third parties that have or will have an agreement with the company.
Step 4: Technology Selection to Implement API Lifecycle
To create successful APIs, you must establish the technology stack at this stage, choosing the appropriate tools and frameworks to develop, deploy, and manage APIs efficiently. The technology stack should align with the organization’s requirements and scalability needs.
Additionally, defining a baseline for API codification ensures consistent and standardized API development, making it easier for developers to collaborate. The necessary infrastructure should also be put in place to support API development.
A testing strategy ensures the quality of the API economy model, aligning with quality attributes metrics, strategy, and business goals. On top of that, a user-centric approach ensures a seamless and effective user experience, as the goal is to deliver reliable, user-friendly, and responsive APIs.
Step 5: Implementation
Taking a strategic approach to implementing an effective API economy model is important, and you first need to create APIs that align with the overall strategy. This involves identifying key functionalities and services offered through the APIs, and once the APIs have been developed the next step is to publish them.
To streamline development, CI/CD pipelines should be established to automate the building, testing, and deployment of APIs. Embracing an API-oriented DevSecOps approach drives agility, faster development cycles, and improved reliability in software development and delivery.
Finally, it’s essential to establish governance to ensure the API economy model is continuously evolving based on user feedback and observability. This involves monitoring the performance of the APIs and making adjustments to enhance the user experience and improve overall strategy.
Pragma’s Approach
Observability in the API economy is critical because a company needs to know if the execution is effective according to their chosen business model. Based on the insights gained, recommendations must be made to improve both profits and customer experience.
Observability has three mainstays—logs, traces, and metrics—but we should consider other critical aspects related to monitoring, such as costs, infrastructure, application performance management), transactional auditory, log management, synthetic monitoring, real user monitoring, and security.
Based on Pragma’s experience, we’ll now showcase an example of observability in the API economy and explain step by step how to implement it and how you could cater to business and technical needs.
Building the Architecture
Clone this repository, which was done with AWS CloudFormation.
Now, change in the template api_economy_template.yaml, the domain PaymentUserPoolDomain (section for the domain you want), and username in ExampleUser (section for your email).
Once you have run the template, you’ll have in your AWS console the next deployed architecture.
Figure 1 – High-level architecture.
This high-level architecture works with APIs that can be exposed to multiple customers with one API key and is connected to Amazon Cognito to manage the authorization process. In addition, the architecture has different services, which we’ll use to answer some business and technical questions.
One of the relevant services of this architecture is Amazon DevOps Guru, a machine learning (ML) service designed to detect anomalous behavior patterns by identifying problems and recommending how to solve them.
Amazon EventBridge is scheduled to run an AWS Lambda function every minute, allowing DevOps Guru to detect whether the service is healthy. Next, uncomment the third line (time.sleep(120)) to modify the latency of this service, and after 10-20 minutes DevOps Guru will start to detect problems and make recommendations to improve the latency.
In the following image, DevOps Guru shows an insight with high severity due to invocations timing out.
Figure 2 – DevOps Guru high severity.
On the same page, DevOps Guru shows recommendations to solve the problem, such as a rollback (remember the error was caused by uncommenting the line in payment service), which is the right solution to the problem. After the rollback, wait a few minutes and DevOps Guru will show the insight was closed.
Figure 3 – Recommend rollback.
Now, we’ll see how to answer other technical and business questions. For that, uncomment the third line to affect the latency again. Then, get a token—you can get one by using the URL:
Change REGION for the region in which you are working in the AWS console; example us-east-1. Next, change YOUR_DOMAIN and YOUR_CLIENT_ID for the data you can find in Cognito > User pools > App Integration (tab).
Note the token generated above should be used by the authenticated user only, as this token is tied to an individual user account. Hence, it should be kept as a secret.
If the API is called from a web or mobile application, AWS Amplify can simplify the overall authentication, including the token lifecycle management. Additionally, you can use AWS WAF and AWS Shield to block malicious requests and prevent from other web exploits and attacks.
The next step is to log in and get the URL and paste in notes to get id_token. Take into account the URL contains id_token, access_token, expires_in, token_type,and you must choose just id_token.
Take into account that to log in, the email address is what you put in the CloudFormation stack and the password was sent to that email.
Next, import TestAPIEconomy.postman_collection.json of the GitHub in Postman, fill in the next variables, and save:
- TOKEN: id_token you got previously
- X-API-KEY: API key you can find in API Gateway > Select your API > API Keys. Example: G4qilyc7EF3Yj7S904Sib9m6dwE6LxKAcavpeRI6
- URL: API Invoke URL in stage prod. Example: https://6yysr6ff7g.execute-api.us-east-1.amazonaws.com/prod/payment/
Send the request from the Postman, and you should receive the following text as a response with status 200: “Your payment has been made successfully.” Create run collections with 20 iterations and a delay of 60,000, run them manually, and wait until they finish.
After the previous process, observe the request trace, percentage of successful transactions, percentage of error transactions, and he transaction quantity per minute in AWS X-Ray for the API Gateway and AWS Lambda service.
Figure 4 – AWS X-Ray percentage of successful and error transactions.
The error we can see now in the API Gateway is because we configured a usage plan that allows just 20 requests; after that, the limit will be exceeded and the response code will be 429: “Too many requests.”
Figure 5 – Too many requests.
Alternatively, with the last requests DevOps Guru also detects the anomalous latency and 4XX Error of “Too many requests,” as shown in the following image. Besides, it gives some recommendations to solve this problem.
Figure 6 – Behavior anomalous in API Gateway.
Next, you could see the API consumption plan per customer in API Gateway > Payment > Usage Plans > Payment_Usage_Plan > API Keys (tab) > Usage.
Figure 7 – API consumption plan per customer.
Cleanup
Delete all cloud formation stacks and the associated cloud watch logs group to avoid incurring additional costs.
Conclusion
In this post, we covered the concepts of API economy and the steps you should take to establish an API economy model effectively.
You have learned the mainstays of observability and the importance of having well-defined observability architecture to have a thriving API economy model at scale. We have used AWS services such as Amazon DevOps Guru and AWS X-Ray to provide answers to improve the business.
Learn more about Pragma in AWS Marketplace.
.
.
Pragma – AWS Partner Spotlight
Pragma is an AWS Partner that has worked with big retail, banking, and insurance companies in LATAM for over 30 years. Its approach is to work alongside organizations and stakeholders to assess their business and technology needs and recommend tailor-made solution roadmaps that are focused on end-user experience.