Docebo logo

Docebo Improves System Observability Using Amazon Data Firehose and AWS Glue

2022

When software-as-a-service learning technology company Docebo determined its team was spending too much time identifying bugs within its multitenant structure, it wanted a single source of truth to drastically reduce the time required for those tasks. Docebo’s fragmented structure required team members to search logs individually when they encountered issues, a process that often took days.
 
The company sought a solution that would facilitate implementation of agnostic trace logs that different teams within the organization could use with a minimum time expenditure. Docebo opted to build a solution using Amazon Web Services (AWS) to unify the logging features of all workloads. As a result, the company has reduced time spent on troubleshooting and monitoring, improved operational efficiency, and expanded analysis opportunities.
Perspective view on digital glowing magnifier in technological circle surrounded by microcircuit lines, global search concept
kr_quotemark

Using AWS, we can improve our product in many ways because we created a stand-alone library that each of our present or future components or microservices can use.”

Francesco Marchesini
Senior Backend Developer

Seeking a Single Source of Truth

Founded in 2005 as a startup, Docebo quickly accelerated to become a global supplier of e-learning technology. The company’s current client list spans the globe and includes AWS itself. Docebo successfully launched its initial public offering on the Toronto Stock Exchange in 2019 and on the Nasdaq in 2020.

Before adopting AWS solutions to create a new unified log stream, the Docebo team was spending valuable time reconstructing its customers’ actions each time a bug arose, trying to identify the problem. “I could spend at least 2 or 3 days on one bug,” says Michele Acierno, senior backend developer at Docebo. It took days to go through the onerous tasks of searching logs individually, and sometimes the team was still unable to fully understand the problem. “We needed a solution to avoid the huge, tedious, daily effort that we were facing,” says Alberto Roli, software development manager at Docebo.

Docebo needed a single source of truth for its notification components for the Docebo log stream. Rather than having disparate log streams for each of the different microservices that the company uses, Docebo required something that could provide a snapshot of each state of the notification flow from its creation to the job. “Our goal was to gather normalized data such that we could understand the flow for each solution on a specific logging group and be able to debug as necessary,” says Acierno.

Implementing Agnostic Trace Logs Using AWS

Docebo began a straightforward implementation in August 2021 and spent approximately 1 month putting the solution into practice. The company’s solution uses a container on Fluent Bit—a logging and metrics processor and forwarder—to handle the tracing log requests. Fluent Bit sends the requests to Amazon Data Firehose—an extract, transform, load service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services. “We use Fluent Bit to send the data with specific information, like logging type and platform domain,” says Acierno. “We use that information as partition keys for Amazon Data Firehose.” Docebo organizes the data using Amazon Simple Storage Service (Amazon S3), an object storage service offering industry-leading scalability, data availability, security, and performance. It then extrapolates that data using Amazon Athena, an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL.

Docebo’s solution also uses AWS Glue, a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. Using AWS Glue as a data catalog helped Docebo to collect the new static fields—such as logging type, platform domain, and timestamp—for logging messages. “We’re using AWS Glue to normalize the data that we have already structured with files for Amazon Athena,” says Acierno.

The team began to benefit from the solution’s precision right away, and it can now use agnostic trace logs to discover misconfigurations or lack of configurations and resolve the root issue. Before, combing through more than 100,000 notifications using different log streams made this kind of discovery nearly impossible. By deploying one Fluent Bit daemon in every region and using dynamic partitioning from Data Firehose, the team can avoid a bottleneck. And if the team needs to implement new workloads, it can do so by using an infrastructure-as-code approach, which makes it possible to install a PHP module or node within the microservice. The module wraps the logs and provides them to Fluent Bit, creating a consistent data structure.

Roli’s team gathered metrics on notification-handling performance by setting up queries on the traced data. It discovered that the system was unnecessarily processing notification jobs due to nonoptimized logics on the management of initiated events. “Millions of jobs resulted from just a few hundred actual submissions,” says Roli. With trace logs, Docebo became aware of this problem and was able to optimize code to avoid useless jobs.

Saving time is a huge benefit for Roli’s team. Docebo has reduced the time that it takes to detect and resolve issues from days to minutes, decreasing troubleshooting time by up to 90 percent. “The only thing I have to do now is insert a specific tracing into an Amazon Athena query, and I have details on when the post was created, if it was delivered, and if there was an error. I have everything, and I can understand everything,” says Acierno. “That was something we couldn’t do before. What used to take 3 days can now take as little as 5 minutes.” In the past, 70–80 percent of the team’s time was spent resolving bugs. With the trace logs in place, the team now spends less than 15 percent of its time on resolving bugs.

Using AWS Glue along with other AWS services and Fluent Bit, the company now has a consistent data structure, offering new benefits to clients. “We can improve our code, and we can improve the future development of the input for the new notification stream,” says Francesco Marchesini, senior backend developer at Docebo. “Using Fluent Bit alongside AWS services, including AWS Glue, is a great solution. Our customers will benefit, too, because we can solve their issues and offer a better, faster product.”

Facilitating Growth on AWS

On AWS, Docebo is improving its business and better supporting its product. Docebo benefits from the agnostic trace logs provided by its combined solution of AWS services, including AWS Glue, and Fluent Bit. And using solutions, such as Amazon Athena, supports the company in querying data for specific payload information, something it couldn’t do before. The developer teams now have a consistent data structure and can integrate its new solution to provide logs. The infrastructure can then be deployed using an infrastructure-as-code approach. 

In the future, the Docebo team plans to use AWS solutions to implement unified tracing along with automatic alerts and notifications. “Using AWS, we can improve our product in many ways because we created a stand-alone library that each of our present or future components or microservices can use,” says Marchesini. “We have a more stable, maintainable, extendable, and performable solution.”


About Docebo

Founded in 2005 in Biassono, Italy, Docebo is a software-as-a-service digital learning company that has grown into a global brand. Docebo is growing rapidly and offers innovative learning technologies.

Benefits of AWS

  • Improved efficiency for problem-solving and debugging
  • Reduced troubleshooting time by 90%
  • Reduced the amount of time to identify bugs from 3 days to as little as 5 minutes
  • Implemented the new solution in 1 month
  • Reduced time spent resolving bugs from 70–80% to less than 15%
  • Facilitated querying of data

AWS Services Used

AWS Glue

AWS Glue is a serverless data integration service that makes it easy to discover, prepare, and combine data for analytics, machine learning, and application development. AWS Glue provides all the capabilities needed for data integration so that you can start analyzing your data and putting it to use in minutes instead of months.

Learn more »

Amazon Athena

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. Athena is serverless, so there is no infrastructure to manage, and you pay only for the queries that you run.

Learn more »

Amazon Data Firehose

Amazon Data Firehose is an extract, transform, and load (ETL) service that reliably captures, transforms, and delivers streaming data to data lakes, data stores, and analytics services

Learn more »

Amazon S3

Amazon Simple Storage Service (Amazon S3) is an object storage service offering industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps.

Learn more »


Get Started

Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.