AWS Partner Network (APN) Blog

Category: Amazon RDS

Why Next-Generation MSPs Need Next-Generation Monitoring

We wrote a couple of months ago about how ISVs are rapidly evolving their capabilities and products to meet the growing needs of next generation Managed Service Providers (MSPs), and we heard from Cloud Health Technologies about how they are Enabling Next-Generation MSPs with cloud management tools that span the breadth of customer engagements from Plan & Design to Build & Migrate to Run & Operate and to Optimize. Today we are sharing a guest post from APN Advanced Technology and SaaS Partner, Datadog, as they address the shift from traditional to next gen monitoring and how these capabilities elevate the level of value that an MSP can deliver to their customers.

Let’s hear from Emily Chang, Technical Author at Datadog.

Why Next-Generation MSPs Need Next-Generation Monitoring

To stay competitive in today’s ever-changing IT landscape, managed service providers (MSPs) need to demonstrate that they can consistently deliver high-performance solutions for their customers. Rising to that challenge is nearly impossible without the help of a comprehensive monitoring platform that provides insights into customers’ complex environments.

Many next-generation MSPs team with Datadog to gain insights into their customers’ cloud-based infrastructure and applications. In this article, we’ll highlight a few of the ways that MSPs use Datadog’s monitoring and alerting capabilities to proactively manage their customers’ increasingly dynamic and elastic workloads with

  • Full visibility into rapidly scaling infrastructure and applications.
  • Alerting that automatically detects abnormal changes.
  • Analysis of historical data to gain insights and develop new solutions.
  • Continuous compliance in an era of infrastructure-as-code.

Full visibility into rapidly scaling infrastructure and applications

As companies continuously test and deploy new features and applications, MSPs need to be prepared to monitor just about any type of environment and technology at a moment’s notice. Whether their customers are running containers, VMs, bare-metal servers, or all of the above, Datadog provides visibility into all of these components in one place.

Datadog’s integration for Amazon Web Services (AWS) automatically collects default and custom Amazon CloudWatch metrics from dozens of AWS services, including Amazon Elastic Compute Cloud (Amazon EC2), Elastic Load Balancing, and Amazon Relational Database Service (Amazon RDS). In total, Datadog offers more than 200 turn-key integrations with popular infrastructure technologies. Many integrations include default dashboards that display key health and performance metrics, such as the AWS overview dashboard shown below.

 

 

MSPs need the ability to monitor every dimension of their customers’ modern applications—as well as their underlying infrastructure. As customers continuously deploy new features and applications in the cloud, MSPs can consult a global overview of the infrastructure with Datadog, and then drill down into application-level issues with Application Performance Monitoring (APM), without needing to switch contexts. Datadog APM traces individual requests across common libraries and frameworks, and enables users to identify and investigate bottlenecks and errors by digging into interactive flame graphs like this one:

 

Infrastructure-aware APM gives MSPs full-stack observability for their customers’ applications, which is critical for troubleshooting bottlenecks in complex environments.

Alerting that automatically detects abnormal changes

Because today’s dynamic cloud environments are constantly in a state of flux, MSPs can benefit immensely from sophisticated alerts that can distinguish abnormal deviations from normal, everyday fluctuations. As customers’ infrastructure rapidly scales to accommodate changing workloads, what constitutes a normal/healthy threshold often will need to scale accordingly. Customers may also wish to track critical business metrics, such as transactions processed, which often exhibit normal, user-driven fluctuations that correlate with the time of day or the day of the week.

Both of these scenarios explain why threshold-based alerts, while helpful for many types of metrics, are not ideal solutions for detecting more complex issues with modern-day applications. To accommodate these challenges, next-generation MSPs need a monitoring solution that uses machine learning to automatically detect issues in their customers’ metrics. Datadog’s anomaly detection algorithms are designed to distinguish between normal and abnormal trends in metrics while accounting for directional trends, such as a steady increase in transaction volume over time and seasonal fluctuations.

Datadog also uses machine learning for outlier detection—algorithms that determine when a host or group of hosts behaves differently from its peers. This effectively enables MSPs to make sense of how resources are being used within a customer’s infrastructure, even as it rapidly scales to accommodate varying workloads. Whenever an outlier monitor is triggered, MSPs can consult the monitor status page, like the one shown below, to quickly understand when the outlier was detected, and which component(s) of the infrastructure it may impact.

 

Analyzing historical data to gain insights and develop new solutions

As their customers’ environments scale and grow increasingly complex, MSPs need an effective way to visualize how all of those components change over time. For historical analysis, all data is retained at full granularity for 15 months. This allows MSPs to analyze how their customers’ infrastructure and applications have evolved and develop strategies that help them make strategic decisions going forward. In addition to visualizing AWS services and other common infrastructure technologies in default dashboards, MSPs can create custom visualizations that deliver deeper insights into their metrics. These visualizations include:

  • Trend lines: Use regression functions to visualize metric trends
  • Change graphs: Display how a metric’s value has changed compared to a point in the past (an hour ago, a week ago, etc.)
  • Heat maps: Use color intensity to identify patterns and deviations across many separate entities. In the example below, a Datadog heat map shows Docker CPU usage steadily trending upward across a large ensemble of containers

 

Ensuring continuous compliance in an era of infrastructure-as-code

Infrastructure-as-code has revolutionized the way that organizations deploy new assets and manage their existing resources, enabling them to become more agile, continuously deploy new features, and quickly scale resources to respond to changing workloads. However, as these tools are more widely adopted, they also require organizations to monitor their assets more carefully, in order to meet compliance requirements.

Datadog integrates with key infrastructure-as-code tools like Chef, Puppet, and Ansible to provide MSPs with a real-time record of configuration changes to each customer’s infrastructure. Datadog also ingests AWS CloudTrail logs to help MSPs track API calls made across AWS services and aggregates them in the event stream for easy reference. In the example below, you can see that CloudTrail reports any successful and failed logins to the AWS Management Console, as well as any EC2 instances that have been terminated—and who terminated them.

 

With all of this data readily available, MSPs can track critical changes as they occur in real time and set up monitors to proactively audit and enforce continuous compliance of their customers’ AWS environments. They can also search and filter for specific types of changes in the event stream and then overlay them on dashboards for correlation analysis, as shown below.

 

Event-based alerts help MSPs automatically detect unexpected changes and/or immediately notify their customers about events that may endanger compliance requirements. These alerts can also be configured to trigger actions in other services through custom webhooks. By making all of this information available in one central location, Datadog prepares MSPs with the data they need to respond quickly to compliance issues.

Next steps for next-generation MSPs

Datadog is pleased to be able to provide monitoring capabilities that help MSPs navigate the challenges of delivering high-performance solutions for dynamic infrastructure and applications. To learn more about how Datadog helps fulfill AWS MSP Partner Program checklist items needed to apply for the AWS Managed Service Program, download our free eBook. You can also view a recording of our recent webinar with AWS and CloudHesive, “What is Means to be a Next-Generation Managed Service Provider” here.

How We Built a SaaS Solution on AWS, by CrowdTangle

The following is a guest post from Matt Garmur, CTO at CrowdTangle, a startup and APN Technology Partner who makes it easy for you to keep track of what’s happening on social media. Enjoy!

Horses were awesome.

If you had a messenger service 150 years ago, using horses was so much better than the alternative, walking. Sure, you had to hire people to take care of horses, feed them, and clean up after them, but the speed gains you got were easily worth the cost. And over time, your skills at building a business let you create systems that could handle each of these contingencies extremely efficiently.

And then cars came around, and you were out of luck.

Not immediately, of course. The first car on the street didn’t put you out of business. Even as cars got more mainstream, you still had the benefit of experience over startup car services. But once the first company grew up that was built with the assumption that cars existed, despite all your knowledge, you were in big trouble.

At CrowdTangle, we build some of the best tools in the world for helping people keep track of what’s happening on social media. We have a team of engineers and account folks helping top media companies, major league sports teams, and others find what they care about in real time (and we’re hiring!). Importantly, we started our company in 2011, which meant that AWS had been around for 5 years, and we could, and did, confidently build our entire business around the assumption that it would exist.

AWS was our car.

It may seem like an exaggeration, but it’s not. We were able to build an entirely different type of organization on AWS than we could have built five years prior. Specifically, it has impacted us in four critical ways: business model, hiring, projections and speed, which of course are all different ways of saying, “cost,” and thus, “survival.”

First is the business model. When we started developing our company, we didn’t consider producing physical media to hold our software, nor did we consider installing it on-premises. By making our model Software as a Service (SaaS), we got a lot of immediate benefits: we were able to allow users to try our product with no more effort than going to a website; we could push features and fixes dozens of times a day; and we could know that everyone would get the same controlled experience. But by taking on the hosting ourselves, we would need to have a significant capital outlay at the start in order to simply deliver our product. Having AWS to begin on without those initial costs made SaaS a viable option for our growing startup.

Next is hiring. AWS has Amazon Relational Database Service (Amazon RDS), a managed database service, which means I don’t need to hire a DBA, since it’s coder-ready (and on Intel Xeon E5s, so we’re certainly not sacrificing quality). AWS has Elastic Beanstalk, a service that makes it simple for us to deploy our application on AWS, which means I can set up separate environments for front- and back-end servers, and scale them independently at the push of a button. Amazon DynamoDB, the company’s managed noSQL database service, helps alleviate me of the need to have four full-time engineers on staff keeping my database ring up and running. We keep terabytes of real-time data, get single-digit millisecond response times, and from my perspective, it takes care of itself. My team can be focused on what matters to our driving the growth of our business, because we don’t need to spend a single hire on keeping the lights on.

Third is projections. If you’re in the horse world, your purchasing model for computers is to run as close to capacity as possible until it’s clear you need a capital outlay. Then you research the new machine, contact your supplier, spend a lot of money at once, wait for shipping, install it, and when it goes out of service, try to resell it and recover some of the cost. In the car world, if I think we might need more machinery, even for a short time, I request an instance, have it available immediately, and start paying pennies or dollars by the hour. If I’m done with that instance? Terminate and I stop paying for it. If I need a bigger instance? I simply provision a bigger instance on the spot.

Finally, I want to talk about speed. Because of our choice to build our solution on AWS, we have a lean team that can provision resources faster, and can constantly work on fun projects rather than having to focus on simple maintenance. Not only can we move quickly on the scoped projects, but we can do cheap R&D for the moonshots. Every new project could be a bust or our next million-dollar product, but they start the same — have an idea, clone an existing environment, put your project branch on it, trot it out for clients to play with, and spin it down when done.

We recently decided that an aggregation portion of our system was slower than we liked, and we researched moving it to Amazon Redshift. To do so, we spun up a small Redshift instance (note: no projections), did initial testing, then copied our entire production database into Redshift (note: R&D speed). “Production” testing proved the benefits, so now we have an entire secondary Amazon Kinesis-Redshift managed pipeline for our system (note: no hiring, despite adding systems), and the speed increase has opened the door for new products that weren’t possible for us under the prior method. How much would that experimentation cost in the horse world? What would it have taken to execute? Would any of those projects have been small enough to be worth taking a chance on? We place small bets all the time, and that’s what helps us remain a leader in our field.

Your next competitor will have grown up in the age of cars. How can you compete when you have horses?

To learn more about CrowdTangle, click here.

 

 

 

 

 

The content and opinions in this blog are those of the third party author and AWS is not responsible for the content or accuracy of this post.
 

Oracle Database Encryption Options on Amazon RDS

As a solutions architect at AWS, I get opportunities to answer customer and partner queries. Many queries require extensive research. This blog post is an outcome of my research on various encryption options such as Oracle Transparent Data Encryption (TDE) and Oracle Native Network Encryption (NNE) and SSL options on Amazon RDS. It explains how Amazon RDS supports Oracle TDE, Oracle NNE, and SSL. If you are an architect or a developer, this post will help you plan and configure storage and network encryption on Amazon RDS. You should be aware of the need to encrypt data at rest and how Oracle TDE, Oracle NNE, and SSL can help you achieve your encryption goals.

Introduction

Oracle Database uses authentication, authorization, and auditing mechanisms to secure data in the database, but not in the operating system data files where data is stored. To protect these data files, Oracle Database provides Transparent Data Encryption (TDE). TDE encrypts sensitive data stored in data files. To prevent unauthorized decryption, TDE stores the encryption keys in a security module external to the database.

TDE enables you to encrypt sensitive data, such as credit card numbers, stored in tables and tablespaces. Encrypted data is transparently decrypted for a database user or application that has access to data. TDE helps protect data stored on media in the event that the storage media or data file gets stolen.

Database users and applications do not need to manage key storage or create auxiliary tables, views, and triggers. An application that processes sensitive data can use TDE to provide strong data encryption with little or no change to the application.

TDE supports the Advanced Encryption Standard (AES-256, AES-192, and AES-128), and the Triple Data Encryption Algorithm (3DES).

Amazon RDS provides two distinct ways to perform Oracle DB instance encryption at rest:

Oracle Native Network Encryption (NNE) and SSL protect the confidentiality of Oracle data as it is transmitted across the network. Encrypting Oracle network traffic safeguards sensitive data such as social security numbers, credit card numbers and other personally identifiable information against packet sniffing. From Oracle 10.2.0.1 onward, Native Network Encryption and TCP/IP with SSL are no longer part of the Advanced Security Option. Amazon RDS for Oracle provides these options on all editions.

How to perform TDE using Option Groups for Amazon RDS for Oracle

Oracle TDE supports two encryption modes: TDE tablespace encryption and TDE column encryption. TDE tablespace encryption is used to encrypt entire application tables. TDE column encryption is used to encrypt individual data elements that contain sensitive data. You can also apply a hybrid encryption solution that uses both TDE tablespace and column encryption.

Option Groups

Amazon RDS uses option groups to enable and configure additional features that make it easier to manage data and databases, and to provide additional security for your database. The TDE option is a permanent option that cannot be removed from an option group. Once associated, that option group cannot be removed from a DB instance. You cannot disable TDE from a DB instance once that instance is associated with an option group with the Oracle TDE option.

Amazon RDS manages the Oracle Wallet and TDE master key for the DB instance. You do not need to set the encryption key using the command ALTER SYSTEM set encryption key.

The process for using Oracle TDE with Amazon RDS is as follows:

  1. You can create an option group and add the TDE option or modify the associated option group to add the TDE option if the DB instance is not associated with an option group that has the TDE option enabled. For information about creating or modifying an option group, see Working with Option Groups. For information about adding an option to an option group, see Adding an Option to an Option Group.
  2. Associate the DB instance with the option group with the TDE option. For information about associating a DB instance with an option group, see Modifying a DB Instance Running the Oracle Database Engine.

If you no longer want to use the TDE option with a DB instance, you must decrypt all your data on the DB instance, copy the data to a new DB instance that is not associated with an option group with TDE enabled, and then delete the original instance. You can rename the new instance to be the same as the previous DB instance if you prefer.

How to perform encryption using AWS CloudHSM for Amazon RDS for Oracle TDE

AWS CloudHSM is a service that lets you use a hardware appliance called a hardware security module (HSM) for secure key storage and cryptographic operations. You can use AWS CloudHSM with an Oracle Enterprise Edition DB instance to store TDE keys when using Oracle TDE. You enable an Amazon RDS DB instance to use AWS CloudHSM by setting up an HSM appliance, setting the proper permissions for cross-service access, and then setting up Amazon RDS and the DB instance that will use AWS CloudHSM.

To use AWS CloudHSM with an Amazon RDS Oracle DB instance, you must complete the following tasks:

When you complete the entire setup, you should have the following AWS components:

  • An AWS CloudHSM control instance that will communicate with the HSM appliance using port 22, and the AWS CloudHSM endpoint. The AWS CloudHSM control instance is an EC2 instance that is in the same VPC as the HSMs and is used to manage the HSMs.
  • An Amazon RDS Oracle DB instance that will communicate with the Amazon RDS service endpoint, as well as the HSM appliance, using port 1792.

 

How to perform encryption with AWS KMS

The AWS Key Management Service (AWS KMS) makes it easy for you to create and control the keys used to encrypt your data. AWS KMS is integrated with other AWS services, including Amazon Elastic Block Store (Amazon EBS), Amazon Simple Storage Service (Amazon S3), Amazon Redshift, Amazon Elastic Transcoder, Amazon WorkMail, and Amazon Relational Database Service (Amazon RDS), to make it simple to encrypt your data with encryption keys that you manage.

AWS KMS is the default option used to perform encryption in Amazon RDS for Oracle databases. While creating an Oracle 11g Enterprise Edition Database instance, you can optionally choose encryption.

How to enable Oracle Native Network Encryption for Amazon RDS for Oracle

Native network encryption (NNE) gives you the ability to encrypt database connections, without the configuration overhead of TCP/IP and SSL/TLS and without the need to open and listen on different ports. NNE not only encrypts the connections but also checks the integrity of the communication by comparing checksums.

Amazon RDS for Oracle uses option groups to enable and configure NNE. The NNE is not a permanent option, so the option can be removed if you no longer want to use the option. To use Oracle Native Network Encryption option on an Amazon RDS Oracle DB instance, you must follow the Oracle Native Network Encryption task.

How to enable Secure Socket Layer (SSL) connection for Amazon RDS for Oracle

You enable SSL encryption for an Amazon RDS Oracle DB instance by adding the Oracle SSL option to the option group associated with the DB instance. Amazon RDS uses a second port for SSL connections which allows clear text communication and SSL-encrypted connections to establish between an Amazon RDS Oracle DB instance and a client.

To establish SSL connection on an Amazon RDS Oracle DB instance, you must complete the following task:

You can use various clients such as Oracle SQL*Plus and JDBC client to connect to an Amazon RDS Oracle instance. You can reference the following sections to configure those clients.

Conclusion

In this post, I described how you can implement Oracle TDE, Oracle NNE, and SSL successfully on Amazon RDS, and also touched upon some important security and encryption services like Amazon KMS and AWS CloudHSM. If you’d like more information, you may find the security and compliance track at re:Invent useful.

 

 

Now Available: Splunk App for AWS 4.2

On Tuesday, Advanced all-in APN Technology Partner and AWS Big Data and Security Competency Partner Splunk announced the availability of its Splunk App for AWS 4.2. The Splunk App for AWS 4.2 adds a number of new features for users, including (but not limited to):

Splunk Principle Product Manager Randy Young wrote a great blog post walking through many of the features of the app, which you can read here.

To learn more about Splunk, visit the company’s AWS Partner Directory listing.