This Guidance shows how to obtain insights, like sentiment, entities, locations, and topics based on social media posts, customer reviews, or other short form content. With the accompanying sample code, you are given a code base to serve as an information extraction system. This system extracts information across various social media platforms, including X, Facebook, and Instagram using a large language model (LLM), providing you with actionable insights about your products and services.
Architecture Diagram
Step 1
An Amazon Elastic Container Service (Amazon ECS) task runs on serverless infrastructure managed by AWS Fargate and maintains an open connection to the social media.
Step 2
The social media access tokens are securely stored in AWS Systems Manager Parameter Store, and the container image is hosted on Amazon Elastic Container Registry (Amazon ECR).
Step 3
When a new social media post arrives, it’s placed into an Amazon Simple Queue Service (Amazon SQS) queue.
Step 4
The logic of this Guidance resides in AWS Lambda function microservices, coordinated by AWS Step Functions.
Step 5
The post is processed in real-time by one of the large language models (LLMs) supported by Amazon Bedrock.
Step 6
Amazon Location Service transforms a location name into coordinates.
Step 7
The post and metadata (insights) are sent to Amazon Simple Storage Service (Amazon S3).
Step 8
Amazon Athena queries the processed tweets with standard SQL.
Step 9
Amazon Lookout for Metrics looks for anomalies in the volume of mentions per category. Amazon Simple Notification Service (Amazon SNS) sends an alert to users when an anomaly is detected.
Step 10
We recommend setting up a Amazon QuickSight dashboard so that users can easily visualize insights.
Step 1
An Amazon Elastic Container Service (Amazon ECS) task runs on serverless infrastructure managed by AWS Fargate and maintains an open connection to the social media.
Step 2
The social media access tokens are securely stored in AWS Systems Manager Parameter Store, and the container image is hosted on Amazon Elastic Container Registry (Amazon ECR).
Step 3
When a new social media post arrives, it’s placed into an Amazon Simple Queue Service (Amazon SQS) queue.
Step 4
The logic of this Guidance resides in AWS Lambda function microservices, coordinated by AWS Step Functions.
Step 5
The post is processed in real-time by one of the large language models (LLMs) supported by Amazon Bedrock.
Step 6
Amazon Location Service transforms a location name into coordinates.
Step 7
The post and metadata (insights) are sent to Amazon Simple Storage Service (Amazon S3).
Step 8
Amazon Athena queries the processed tweets with standard SQL.
Step 9
Amazon Lookout for Metrics looks for anomalies in the volume of mentions per category. Amazon Simple Notification Service (Amazon SNS) sends an alert to users when an anomaly is detected.
Step 10
We recommend setting up a Amazon QuickSight dashboard so that users can easily visualize insights.
Get Started
Deploy this Guidance
Well-Architected Pillars
The AWS Well-Architected Framework helps you understand the pros and cons of the decisions you make when building systems in the cloud. The six pillars of the Framework allow you to learn architectural best practices for designing and operating reliable, secure, efficient, cost-effective, and sustainable systems. Using the AWS Well-Architected Tool, available at no charge in the AWS Management Console, you can review your workloads against these best practices by answering a set of questions for each pillar.
The architecture diagram above is an example of a Solution created with Well-Architected best practices in mind. To be fully Well-Architected, you should follow as many Well-Architected best practices as possible.
-
Operational ExcellenceRead the Operational Excellence whitepaper
Amazon CloudWatch keeps logs of the operations performed in the text processing workflow, allowing for efficient monitoring of the applications status. Amazon CloudFormation allows for reproducibility of the deployment and can also be rolled back to a stable state in case deployment fails. Additionally, Amazon Bedrock is a managed service to use LLMs through a simple interface. This combination of monitoring, reproducible deployments, and AWS managed LLMs usage offers powerful natural language processing capabilities without having to manage the underlying infrastructure.
-
SecurityRead the Security whitepaper
The data stored in Amazon S3 is encrypted at rest using AWS Key Management Service (AWS KMS) keys, and AWS Identity and Access Management (IAM) is utilized to control access to the data. Specifically, AWS KMS assists in the creation and management of the encryption keys used to securely encrypt the data stored in Amazon S3. Whereas IAM provides the capability to configure granular permissions based on roles for least privilege access control to that data.
-
ReliabilityRead the Reliability whitepaper
The data is stored in Amazon S3, an object storage service that offers 99.999999999% (11 nines) durability. The LLMs are invoked using Amazon Bedrock through a simple and efficient API interface that can automatically scale up and down. Athena, QuickSight, and AWS Glue are used to query and visualize the data at scale without the need to provision infrastructure.
-
Performance EfficiencyRead the Performance Efficiency whitepaper
Through the use of various serverless and managed AWS services, this Guidance is designed for your workloads to achieve high performance efficiency, automatically scaling resources to meet the demands of the workload, and providing a seamless experience for you to access insights from your social media platforms. For example, Lambda, a serverless compute service, automatically scales up and down based on demand, ensuring the compute capacity is optimized for the workload. With Amazon Bedrock, you can invoke LLMs from an extensive catalogue without the need to provision and manage the underlying servers.
-
Cost OptimizationRead the Cost Optimization whitepaper
Lambda is used in this architecture to process events and initiate the batch transformation analysis, removing the need for a continuously running server. Moreover, AWS Glue jobs are used to perform extract, transform, load (ETL) on batches of user data, rather than individual records. By aggregating the data and processing in larger chunks, the overall compute and storage requirements are reduced, leading to lower costs compared to handling each record individually. Lastly, Amazon Bedrock allows for the use of the LLN that best fits your budget requirement so you do not incur unnecessary expenses associated with more powerful, but potentially over-provisioned, models.
-
SustainabilityRead the Sustainability whitepaper
Lambda, AWS Glue, Athena, and QuickSight are all serverless services that operate on-demand, adjusting their resource use to match the current workload. This helps ensure that the performance and use of resources are maximized, as the services scale up and down automatically to accommodate the required demand. By using these serverless offerings, this architecture can efficiently utilize the necessary resources, avoiding over-provisioning or under-utilization of compute, storage, and other infrastructure components.
Related Content
Build a news-based real-time alert system with Twitter, Amazon SageMaker, and Hugging Face
Disclaimer
The sample code; software libraries; command line tools; proofs of concept; templates; or other related technology (including any of the foregoing that are provided by our personnel) is provided to you as AWS Content under the AWS Customer Agreement, or the relevant written agreement between you and AWS (whichever applies). You should not use this AWS Content in your production accounts, or on production or other critical data. You are responsible for testing, securing, and optimizing the AWS Content, such as sample code, as appropriate for production grade use based on your specific quality control practices and standards. Deploying AWS Content may incur AWS charges for creating or using AWS chargeable resources, such as running Amazon EC2 instances or using Amazon S3 storage.