What Is LLMOps?

Large Language Model Operations (LLMOps) are the tools and practices used to manage large language model operations in production environments. Most organizations select from open source and commercial large language models (LLMs), customize them for specific use cases, and deploy them into production. This process has unique security, scale, customer support, and version control challenges. LLMOps manages and automates the LLM lifecycle, from fine-tuning to maintenance. It includes methodologies derived from DevOps to support continuous monitoring, testing, and improvement of Large Language Models.

Generative AI (GenAI) applications in business can significantly increase productivity, help gain a competitive edge, and scale operations beyond what’s currently possible. However, implementing generative AI comes with its own operational challenges.

LLMOps brings the same benefits to LLM development that DevOps brings to traditional software development. Traditionally, DevOps aims to bridge the gap between development and operations teams. DevOps helps ensure code changes are automatically deployed to production using a continuous integration and delivery (CI/CD) process. Automated testing makes code updates and new feature development efficient and reliable.

Similarly, LLMOps promotes a culture of collaboration to achieve faster release cycles, improved AI application quality, and more efficient resource use in LLM development. Organizations can produce quality LLM apps at scale with model and pipeline development supporting the LLM lifecycle.

Versioning

Application code updates can create errors that require rollbacks and troubleshooting. Similarly, LLM fine-tuning activities require version control and management of model configurations, prompt datasets, and training datasets. LLMOps makes LLM customization and retraining versionable and traceable. Developers can reuse models and datasets, increasing the speed of AI adoption.

Observability

LLMOps enhances the visibility of LLM operations in the production pipeline so you can measure and control the performance of your AI applications. LLMOps teams can track metrics like prompt tokens, completions, LLM latency, context adherence, completeness, chunk utilization, and more. While LLMs are inherently more challenging to instrument than other ML models, new tooling is emerging rapidly in this space. Developers can use the technology to find the root cause of issues and fix them.

Control

LLMOps strengthens security, giving organizations more control over the data their LLMs use and the responses they generate. Managers can map their LLM projects' observability and performance metrics to business objectives and report on AI outcomes. Organizations can better control operational costs associated with running large-scale models and track the returns on their AI investments.

MLOps, FMOps, and LLMOps are different terms used in AI application development. They have all derived from DevOps and have the same goal— enhance efficiency and quality using automation throughout the development cycle. However, implementation details vary because of different project requirements.

Classic MLOps help you productize your ML use cases. However, generative AI use cases require extending MLOps capabilities to meet more complex operational requirements. That’s where FMOps and LLMOps become essential.

MLOps

Machine learning is the science of developing algorithms and statistical models that computer systems use to perform tasks without explicit instructions, relying on patterns and inference instead. Machine learning models and systems gradually evolved into the modern generative AI applications we see today.

ML and operations (MLOps) is the combination of people, processes, and technology to productionize ML solutions efficiently. It includes automated model training and building, testing, deployment, and serving. All the produced models and code automation are stored in a centralized tooling account using the capability of a model registry. Teams can abstract, templatize, maintain, and reuse ML models for different use cases.

FMOps

A foundation model (FM) is a very advanced and massively scaled ML model that can be used to create a wide range of other AI models. FMs are trained on terabytes of data and have hundreds of billions of parameters so they can predict the next best answer for three main generative AI categories:

• Text-to-text—given text input, predict the next best word or sequence of words.
• Text-to-image—given text input, predict the next best image.
• Text-to-audio or video—given text input, predict the next best audio or video clip.

Given the generic nature of foundation models, FMOps has a broader scope. Its goal is to build and commercialize new FM models and train them from scratch. FMOps includes technologies and processes to manage large-scale multimedia training data and review complex output. Beyond model versioning, it requires capabilities for parameter tuning and result evaluation at scale.

LLMOps

Large language models (LLMs) are text-to-text FM models. Most organizations do not have the capabilities to build a new FM from scratch. Instead, they customize an existing LLM for specific use cases. LLM model selection and customization come with a set of new concerns. LLMOps is the toolset and practices of deploying large language models in a repeatable, stable, managed environment, regardless of the business use case.

LLMOps includes technologies, processes, and people that enhance efficiency at every step of the LLM lifecycle. The broad stages of the LLM lifecycle are described below.

Model discovery

Productionizing LLM models begins with model discovery and selection. Initially, teams perform exploratory data analysis (EDA). They identify the datasets the LLM will work with and perform tasks like data collection, cleaning, and preprocessing for model consumption. After that, they select and evaluate the best models for their use case.

LLMOps platforms offer a selection of large language models. Engineers and decision-makers can perform a model review on different parameters before deciding on the best fit for their data. Due to sensitive data, model performance is examined alongside other parameters such as cost and security. Evaluation frameworks can be used to support the model discovery phase.

Model customization

Once a model is chosen, it is customized to fit the organization’s specific dataset. Customization includes:

Fine-tuning

Changes the LLM model parameters. It requires a larger upfront amount of computing power but allows for LLMs to be trained to do tasks they previously could not.

RAG

Retrieval Augmented Generation (RAG) is an alternative to fine-tuning a model where model parameters are not changed. Instead, domain data is converted to vector embeddings indexed in a vector database. The application performs a similarity search of the prompt embedding against the index and uses the result to provide a context within the LLM prompt.

LLMOps provides tooling that supports prompt engineering, chaining, and agents. DevOps-style tooling for telemetry, security, and testing can also be integrated. Tooling to support new LLM releases is critical here, as these updates can significantly improve model performance.

Deployment and monitoring

Once the LLM is customized, it is prepared for production. Continuous model monitoring is essential to ensure users make appropriate requests and that the LLM only responds with authorized and accurate data. You must continuously deploy new data so the LLM always has up-to-date information.

LLMOps provides automation and processes that support the continuous deployment of LLM apps to production. It ensures the observability and monitoring of all LLM apps in production for performance and issue tracking.

Similar to DevOps, LLMOps comprises three broad phases.: Continuous Integration (CI), Continuous Deployment (CD), and Continuous Tuning (CT). The people involved in this process include data scientists, prompt engineers, machine learning engineers, model owners, and generative AI consumers.

Continuous integration

CI consists of merging all working copies of an application’s code into a single version and running system and unit tests on it. When working with LLMs, unit tests often need manual tests of the model’s output. For example, with a game character backed by an LLM, tests could ask the character questions about their background, other characters in the game, and the setting.

Continuous deployment

CD consists of deploying the application infrastructure and model(s) into the specified environments once the models are evaluated for performance and quality with metric-based evaluation or with humans in the loop. A typical pattern consists of deploying into a development and quality assurance (QA) environment before deploying into production(PROD). By placing a manual approval between the QA and PROD environment deployments, you can ensure the new models are tested in QA before deployment in PROD.

Continuous tuning

CT is the process of fine-tuning the large language model with additional data to update its parameters, which optimizes and creates a new version of the model. This process generally consists of data pre-processing, model tuning, model evaluation, and model registration. Once the model is stored in a model registry, it can be reviewed and approved for deployment.

Various technologies work together in large language model operations.

Prompt engineering

Prompt engineering is the process of designing the most appropriate formats, phrases, and words to guide the LLM in generating an optimal output. However, creating new prompts is just the first step. Prompt engineering technologies allow you to track prompt changes and the resulting responses. You can determine high-quality vs. low-quality prompts and measure model inference.

Embeddings and vector databases

Embeddings are numerical representations of languages. They allow the LLM to understand and process languages like humans do. Organizations must create new embedding from internal knowledge bases so LLMs can respond to user queries with internal data. Vector databases are essential for storing, processing, and updating embeddings. Management of embeddings and these vector databases is essential for the delivery of customized large language models.

Chains and agents

Chains are where an LLM response is input to another or the same LLM. Agents are autonomous or semi-autonomous systems that interact with the language models to perform specific functions. Agents handle LLM activities within an application. Management of these two different technologies can be difficult, as there may be a lot of debugging. However, both are essential for large-scale AI adoption within an organization.

Evaluation tools

Large language model output has to be checked for bias, coherence, accuracy, model drift, and reliability. LLMOps includes various tools that set benchmarks and measure LLM output quality. For example, evaluation tools can assess the quality of summarizations, perform cross-LLM checks, or evaluate any retrieval augmented generation (RAG) pipelines. They also support human evaluation and feedback of LLM output.

Observability tools

LLMOps includes technologies that support instrumenting an LLM service to collect logs, traces, and metrics. You can use them to identify and solve issues in production and produce performance and reliability measurements. The tools assess model output and can alert team members if specific metric thresholds are unmet.

Amazon SageMaker Pipelines is a purpose-built workflow orchestration service that automates all stages of the LLM lifecycle, from data pre-processing to model monitoring. With an intuitive UI and Python SDK, you can manage repeatable end-to-end LLM pipelines at scale. You can easily train, test, troubleshoot, deploy, and govern LLM models at scale to boost productivity while maintaining model performance in production.

Amazon Bedrock is a fully managed service that offers a choice of high-performing LLMs and a broad set of capabilities you need to build generative AI applications with security, privacy, and responsible AI. Using Amazon Bedrock, you can:

• Experiment with and evaluate top LLMs for your use case
• Privately customize LLMs with your data using fine-tuning or RAG
• Build agents that execute tasks using your enterprise systems and data sources.

Since Amazon Bedrock is serverless, you don't have to manage any infrastructure. You can securely integrate and deploy generative AI capabilities into your applications using the AWS services you are already familiar with.

Get started with LLMOps on AWS by creating a free account today.