AWS Open Source Blog

Category: Amazon Simple Storage Services (S3)

User uploads data in BIDS format to S3 and starts the Lambda function → Lambda parses the uploaded data and launches a cluster of EC2 instances → EC2 instances run fMRIprep which preprocesses the data → preprocessed data are saved to S3.

fMRI data preprocessing on AWS using fMRIprep

A typical fMRI study often produces imaging data of terabytes or more. Storing and preprocessing this data can be challenging on a single computer because it often has neither enough disk space to store the data nor enough computing power to preprocess it. Traditionally, researchers use a combination of cloud-based storage and on-premises high-performance clusters […]

Read More
Setup of the centralized logging demo app

Centralized Container Logging with Fluent Bit

by Wesley Pettit and Michael Hausenblas AWS is built for builders. Builders are always looking for ways to optimize, and this applies to application logging. Not all logs are of equal importance. Some require real-time analytics, others simply need to be stored long-term so that they can be analyzed if needed. It’s therefore critical to […]

Read More
s5cmd in action

Parallelizing S3 Workloads with s5cmd

This open source project comes from our customer community. It was developed by Peak Games to assist with their own S3 workflow, and includes features such as tab completion and built-in wild card support for files in S3 commands. Enjoy! – Deirdré Background Up until now, working on multiple objects on Amazon S3 from the […]

Read More