AWS Big Data Blog
Author: Manabu McCloskey
Manabu is a Solutions Architect at Amazon Web Services. He works with AWS strategic customers to help them achieve their business goals. His current focus areas include GitOps, Kubernetes, Serverless, and Spinnaker.
Introducing MCP Server for Apache Spark History Server for AI-powered debugging and optimization
Today, we’re announcing the open source release of Spark History Server MCP, a specialized Model Context Protocol (MCP) server that transforms this workflow by enabling AI assistants to access and analyze your existing Spark History Server data through natural language interactions. This project, developed collaboratively by AWS open source and Amazon SageMaker Data Processing, turns complex debugging sessions into conversational interactions that deliver faster, more accurate insights without requiring changes to your current Spark infrastructure. You can use this MCP server with your self-managed or AWS managed Spark History Servers to analyze Spark applications running in the cloud or on-premises deployments.