Chalk

The data platform for machine learning. Tired of Spark? So are we. Just-in-time data + Hot-reload + Rust compute

Chalk is a data platform that powers machine learning and generative AI. Chalk's best-in-class developer experience enables data teams to declare features and their dependencies with idiomatic Python in online, streaming, and batch environments. Chalk compiles these definitions into parallel pipelines that run on a Rust-based engine. These pipelines use the exact same source code to serve temporally-consistent training sets to data scientists and live feature values to models. This re-use ensures that feature values from online and offline contexts match and dramatically cuts development time. With Chalk, engineers, data scientists, and analysts can focus on their unique products while Chalk seamlessly handles data infrastructure.

Chalk's platform includes building blocks that are critical to shop production-grade machine learning:

Compute - Chalk makes it easy to integrate data rom any APO or data source to compute realtime ML features just-in-time. With Chalk, models operate on the freshest possible data and users don't pay to fetch they don't need. Chalk automatically orchestrates compute, caching, scheduling, and streaming infrastructure, and executes Python on a Rust-based runtime for maximum performance.

LLM Toolchain - Chalk unifies structured and unstructured data, allowing companies to incorporate deep learning and LLMs into decisions alongside structured business data. It offers a vector database and integrations with OpenAI, Cohere, and Anthropic to support Retrieval Augmented Generation (RAG) workflows.

Feature Store - Chalk is a centralized place to store, serve, and discover features for machine learning. It accelerates new model and feature development by re-using engineering work from previous models. It enables users to fetch DataFrames directly from Jupyter notebooks so production and training data is guaranteed to be identical.

Monitoring - Chalk was built with an awareness that production data often drifts from historical baselines, pipelines break, and partners change data formats. Chalk automatically monitors the execution of feature pipelines and the distributions of features to alert users when problems arise.

Branches - Chalk enables users to instantly fork feature engineering pipelines and experiment with new features. For example, users can define a new resolver in one notebook cell and use it to generate training sets in the next. They can also seamlessly iterate on definitions and visualize the impact of changes, with deployment times measured in milliseconds.