AWS Open Source Blog

User uploads data in BIDS format to S3 and starts the Lambda function → Lambda parses the uploaded data and launches a cluster of EC2 instances → EC2 instances run fMRIprep which preprocesses the data → preprocessed data are saved to S3.

fMRI data preprocessing on AWS using fMRIprep

A typical fMRI study often produces imaging data of terabytes or more. Storing and preprocessing this data can be challenging on a single computer because it often has neither enough disk space to store the data nor enough computing power to preprocess it. Traditionally, researchers use a combination of cloud-based storage and on-premises high-performance clusters […]

VMD over NICE DCV.

Deploying an HPC cluster and remote visualization in a single step using AWS ParallelCluster

Since its initial release in November 2018, AWS ParallelCluster (an AWS-supported open source tool) has made it easier and more cost effective for users to manage and deploy HPC clusters in the cloud. Since then, the team has continued to enhance the product with more configuration flexibility and enhancements like built-in support for the Elastic […]

Open Distro for Elasticsearch logo.

Introducing real-time anomaly detection in Open Distro for Elasticsearch

There is an enormous increase in real-time streaming applications across a wide range of industries such as finance, health, information technology, retail, and the Internet of Things (IoT). Organizations depend on log analytics solutions to detect aberrations in the data and identify critical situations. Examples include finding fraudulent behavior in financial transactions, discovering suspicious IP addresses […]

AWS Fargate container logs collection and analysis with AWS FireLens and Sumo Logic

AWS Fargate is a compute engine for Amazon ECS that allows you to run containers without having to manage servers or clusters. Fargate manages provisioning, configuration, and scaling of the clusters. With Fargate, you can focus on your application design and implementation instead of worrying about the infrastructure. In this post, we’ll provide an overview […]

Figure 1: Sumo Logic EKS Integration

The Sumo Logic integration with Amazon EKS

Amazon Elastic Kubernetes Service (Amazon EKS) makes it easy to deploy, manage, and scale containerized applications using Kubernetes on AWS. In this post, we’ll provide an overview on how Sumo Logic’s integration with Amazon EKS works using the open source tools Helm, Fluent Bit, Fluentd, Prometheus, and Falco, and how to use it to: Monitor the […]

Splitting an application’s logs into multiple streams: a Fluent tutorial

Not all logs are of equal importance. Some require real-time analytics, others simply need to be stored long term so that they can be analyzed if needed. In this tutorial, I will show three different methods by which you can “fork” a single application’s stream of logs into multiple streams which can be parsed, filtered, […]

Gearing up for re:Invent 2019 with Open Distro for Elasticsearch sessions

re:Invent 2019 has a new track this year and it’s all about Open Source! There are lots of great sessions coming up on Open Distro for Elasticsearch and its components such as Alerting, Security, and Performance Analyzer. Join in to learn more and participate in hands-on workshops! Keep a lookout for our sessions on machine […]

Robotics at re:Invent 2019.

Robotics at re:Invent 2019

  For the first time, at re:Invent 2019, we will have dedicated Robotics and Open Source tracks! We’re excited to be working closely with the open source community to make robotics easier than ever for everyone. Throughout the week, we are offering a diverse variety of content and learning opportunities about the open source Robot Operating System […]

Overview of the infrastructure pipeline.

Using AWS CodePipeline and open source tools for at-scale infrastructure deployment

AWS offers a rich set of developer tools to host code, build, and deploy your application and/or infrastructure to AWS. These include AWS CodePipeline, for continuous integration and continuous deployment orchestration; AWS CodeCommit, a fully-managed source control service; AWS CodeBuild, a fully-managed continuous integration service that compiles source code, runs tests, and produces software packages; […]

Continuous delivery of container applications to AWS Fargate with GitHub Actions

At the day two keynote of the GitHub Universe 2019 conference on Nov 14, Amazon Web Services announced that we have open sourced four new GitHub Actions for Amazon ECS and ECR. Using these GitHub Actions, developers and DevOps engineers can easily set up continuous delivery pipelines in their code repositories on GitHub, deploying container workloads to Amazon Elastic Container […]