Overview
Visual Flow for Databricks is a cloud-native open-source ETL tool developed for Databricks environment with a handy Graphical User Interface. It uses best Databricks practices and combines them into a single system with the following features: Portability, flexibility and multi-cloud compatibility Increased developer productivity High availability, performance and fault tolerance Cost-effective Open-source Visual Flow for Databricks provides a Graphical User Interface to create ETL jobs and combine them in Data Processing Pipelines, run/schedule them, and monitor execution. Individual cluster setup is available for every Job. It is a multi-tenant system. You can create multiple projects in a single cluster. Each project is represented as Databricks workspace. Visual Flow for Databricks supports all existing Databricks Cloud systems and auto-detects them from the project settings. Our solution provides the ability to securely manage parameters and connections and reuse them in Jobs. Import/Export functionality allows to migrate data between projects with ease. Why Visual Flow for Databricks? Supports all existing Databricks Cloud environments Graphical User Interface speeds up learning and jobs development Good monitoring and alerting capabilities out of the box Error Handling out of the box Organizing jobs into Data pipelines with the ability to run them on schedule Unlimited parallelism Unlimited scalability Easy to customize and extend Easy to test automatically Can be installed on-premise as well as on cloud provider of your choice Can be packaged and provided as SaaS solution on your own Cloud platform
REQUIRED SERVICES & APPLICATIONS: Before deployment: GitHub account for OAuth. After deployment: Databricks account. Guide to connect to Databricks
Highlights
- Unique open-source ETL tool with a Graphical User Interface Developed for Databricks environment Unlimited parallelism and scalability
- Visual Flow landing page https://visual-flow.com/
- Visual Flow on YouTube https://tinyurl.com/VisualFlow-playlistst
Details
Pricing
Vendor refund policy
Open source software is freely available to use and made by the people, for the people.
Legal
Vendor terms and conditions
Content disclaimer
Delivery details
Helm chart to deploy Visual Flow for Databricks containers into EKS
- Amazon EKS
Container image
Containers are lightweight, portable execution environments that wrap server application software in a filesystem that includes everything it needs to run. Container applications run on supported container runtimes and orchestration services, such as Amazon Elastic Container Service (Amazon ECS) or Amazon Elastic Kubernetes Service (Amazon EKS). Both eliminate the need for you to install and operate your own container orchestration software by managing and scheduling containers on a scalable cluster of virtual machines.
Version release notes
0.2 (2024-08-19)
Improvements and new features:
- Added support of OAuth M2M as an authentication mechanism for Databricks service principles.
- Added ability to configure OAuth while creating a project.
Additional details
Usage instructions
- Create EKS cluster
- Install AWS Load Balancer Controller and AWS EBS CSI driver on top of your EKS cluser
- Create GitHub OAuth .
Homepage URL
= https://visual-flow-dummy-url.com/vf/ui/ andAuthorization callback URL
= https://visual-flow-dummy-url.com/vf/ui/callback - Install Helm chart
# Admin users for this app. You can have more than 1 GitHub user, see example below
export GITHUB_USER_LIST=GitHubUser1,GitHubUser2
# use GITHUB_APP_ID and GITHUB_APP_SECRET from step #3
export GITHUB_APP_ID=YOUR_GITHUB_APP_ID_HERE
export GITHUB_APP_SECRET=YOUR_GITHUB_APP_SECRET_HERE
# set the latest available version
export VF_HELM_VERSION=0.2
# helm install command
helm upgrade -i vfdbricks-aws-app oci://709825985650.dkr.ecr.us-east-1.amazonaws.com/iba-group/visual-flow-databricks -n default \
--set vfdbricks-aws-services.databricks.configFile.superusers={${GITHUB_USER_LIST}} \
--set vfdbricks-aws-services.frontend.deployment.secretVariables.GITHUB_APP_ID=${GITHUB_APP_ID} \
--set vfdbricks-aws-services.frontend.deployment.secretVariables.GITHUB_APP_SECRET=${GITHUB_APP_SECRET} \
--debug \
--version ${VF_HELM_VERSION}
Find your autogenerated URL and replace visual-flow-dummy-url.com
from step #2 with your autogenerated URL:
kubectl get svc vfdbricks-aws-app-frontend -o yaml | grep hostname | cut -c 17-
Resources
Vendor resources
Support
Vendor support
Support options:
Essential support - EU Business hours Advanced support - 24x5 Premium support - 24x7 + Technical Account Manager
Additionally on demand:
Professional Services Architect's consult Initial installation Feature requests
Contact details for pre-purchase support: info@visual-flow.com Contact details for post-purchase support: info@visual-flow.com
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.