LightningFlow comes pre-integrated with all the required Airflow libraries namely, webserver, scheduler and worker configurations, local Spark cluster, Apache Livy, and a postgres database. When you start the EC2 instance, all the required services are instantiated with standard configurations, thus enabling to start running and testing the DAGs instantaneously. The DAGs run on a local Spark cluster, thus eliminating the need to create an EMR cluster. However, appropriate permissions (e.g. S3 and RedShift access) need to be added to the LightningFlow EC2 through an IAM role. Alternatively, jobs can be submitted to an EMR cluster communicating via Livy REST API, which would require additional configurations. In case you need any assistance for configuring, deploying and running scalable jobs in the Production environment, please reach out to us at info@lightning-analytics.com for more details.