Sold by: CCNets
Deployed on AWS
Train and deploy reinforcement learning policies from offline trajectory data. Use existing logs for cloud-based training while keeping your simulator, game build, or robot in your own environment. Built for game AI, robotics, and automation.
Overview
CCNets Causal GPT-RL Training Algorithm trains a reinforcement learning policy from trajectories you have already recorded, for teams building game AI, robotics, simulation, and automation agents.
Training is entirely offline. The job runs in your own AWS account and learns from the dataset you upload, so your simulator, game build, or robot never has to run inside the training container.
You are not left waiting in the dark until it finishes. Runnable policies are written to the checkpoint S3 location you configure for the job, at regular points through the run and again whenever the policy improves, so you can pull each one as it lands, score it in your target environment, and stop early once you have what you need.
A transformer looped through its own output drifts, and reinforcement learning has no internet-scale corpus for those errors to average out, so we built the reinforcement learning algorithm that holds a GPT stable across a full episode instead. Because the model outputs actions rather than language it stays small, and because it stays autoregressive you can choose the history window at deployment without retraining.
That structure also changes what setup costs you. Reinforcement learning is usually built on hand-tuned MLP stacks, so every new environment turns into its own tuning campaign. Posing it as a sequence-modeling problem instead moves it onto ground that language models are already known to be forgiving on: the same training configuration carries across environments, and you choose the model capacity that fits the deployment you have in mind, so the policy is built at the size you intend to run.
Trained policy bundles and trajectory datasets for MuJoCo and Unity ML-Agents environments are published publicly, so you can see what the algorithm produces before committing your own data.
Resources
-
Start here, the four-part walkthrough from data to a policy you can prompt https://github.com/ccnets-team/causal-gpt-rl/blob/main/training/docs/journey/01-bring-your-own-data.md
-
GitHub repository and public inference runtime https://github.com/ccnets-team/causal-gpt-rl
-
AWS Marketplace and SageMaker training guide https://github.com/ccnets-team/causal-gpt-rl/blob/main/training/docs/aws/aws-marketplace-training.md
-
MuJoCo policy bundles and benchmark results https://huggingface.co/ccnets/causal-gpt-rl
-
MuJoCo training runs, the public W&B logs behind those results https://wandb.ai/causal-gpt-rl/mujoco
-
Unity ML-Agents policies and benchmark results https://huggingface.co/ccnets/causal-gpt-rl-unity
-
Unity ML-Agents trajectory datasets https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-datasets
-
Unity ML-Agents environments https://huggingface.co/datasets/ccnets/causal-gpt-rl-unity-envs
-
CCNets https://ccnets.org
Highlights
- Offline reinforcement learning from trajectory data you already have. Your simulator, game build, or robot never runs inside the training container, so there is nothing to port to the cloud and nothing to license.
- Runnable policy bundles land in your own S3 while the job is still training, delivered at five points across the run and again whenever the policy improves. Score them in your target environment and stop the job early once they are good enough.
- One training configuration carries across environments, with no per-environment tuning campaign. The GPT-style structure keeps the policy small, and you choose the model size you intend to deploy before training starts.
Details
Sold by
Categories
Delivery method
Latest version
Deployed on AWS
New
Introducing multi-product solutions
You can now purchase comprehensive solutions tailored to use cases and industries.
Features and programs
Financing for AWS Marketplace purchases
AWS Marketplace now accepts line of credit payments through the PNC Vendor Finance program. This program is available to select AWS customers in the US, excluding NV, NC, ND, TN, & VT.
Pricing
Pricing is based on actual usage, with charges varying according to how much you consume. Subscriptions have no end date and may be canceled any time.
Additional AWS infrastructure costs may apply. Use the AWS Pricing Calculator to estimate your infrastructure costs.
Dimension | Description | Cost/host/hour |
|---|---|---|
ml.g5.xlarge Training Recommended | Algorithm training on the ml.g5.xlarge instance type | $4.00 |
ml.m5.xlarge Inference (Batch) Recommended | Model inference on the ml.m5.xlarge instance type, batch mode | $1.00 |
ml.m5.xlarge Inference (Real-Time) Recommended | Model inference on the ml.m5.xlarge instance type, real-time mode | $1.00 |
ml.m5.2xlarge Inference (Batch) | Model inference on the ml.m5.2xlarge instance type, batch mode | $1.00 |
ml.m5.2xlarge Inference (Real-Time) | Model inference on the ml.m5.2xlarge instance type, real-time mode | $1.00 |
ml.m5.4xlarge Inference (Batch) | Model inference on the ml.m5.4xlarge instance type, batch mode | $1.00 |
ml.m5.4xlarge Inference (Real-Time) | Model inference on the ml.m5.4xlarge instance type, real-time mode | $1.00 |
ml.m5.large Inference (Batch) | Model inference on the ml.m5.large instance type, batch mode | $1.00 |
ml.m5.large Inference (Real-Time) | Model inference on the ml.m5.large instance type, real-time mode | $1.00 |
Dimensions summary
You pay by the hour based on the task you run and the instance you choose. One dimension covers training on a GPU instance (ml.g5.xlarge), where you build a policy from your offline trajectory data. The remaining eight dimensions cover inference on four CPU instance sizes (ml.m5.large, ml.m5.xlarge, ml.m5.2xlarge, ml.m5.4xlarge). Each inference size offers two modes: batch, for processing grouped data, and real-time, for live responses. Costs scale with the instance size you select and the number of host-hours you use. There is no upfront commitment.
Top-of-mind questions for buyers
Training on the ml.g5.xlarge GPU instance meters each host-hour your job runs. You upload an offline trajectory dataset and the job builds a Policy Bundle from it. This is a decision-making model trained from your recorded logs, not from live simulation. You keep the trained weights for later inference.
Batch mode runs inference on grouped data you submit together, useful for scoring stored logs. Real-time mode responds to live requests as they arrive, suited for interactive agents. Both meter host-hours on the same CPU instance type, so cost tracks how long the instance runs.
Charges apply per host-hour while the instance runs. When a training job completes and the instance stops, software charges stop. Inference instances meter running time, so an active instance accrues charges even between requests. Underlying AWS storage or compute fees may apply separately.
ccnets.org
Helpful?
Vendor refund policy
CCNets may review refund requests case by case for duplicate charges, AWS Marketplace billing errors, inability to access the product due to a CCNets-side issue, or confirmed defects in the Marketplace container or product configuration. Refunds are not provided for customer configuration errors, unsupported datasets, dataset quality issues, unmet performance expectations, or successfully completed usage. Requests must be sent within 14 days to support@ccnets.org .
Legal
Vendor terms and conditions
Upon subscribing to this product, you must acknowledge and agree to the terms and conditions outlined in the vendor's End User License Agreement (EULA) .
Content disclaimer
Vendors are responsible for their product descriptions and other product content. AWS does not warrant that vendors' product descriptions or other product content are accurate, complete, reliable, current, or error-free.
Delivery details
Amazon SageMaker algorithm
An Amazon SageMaker algorithm is a machine learning model that requires your training data to make predictions. Use the included training algorithm to generate your unique model artifact. Then deploy the model on Amazon SageMaker for real-time inference or batch processing. Amazon SageMaker is a fully managed platform for building, training, and deploying machine learning models at scale.
Deploy the model on Amazon SageMaker AI using the following options:
Version release notes
CCNets Causal GPT-RL Training Algorithm, SageMaker Algorithm version cgrl-training-algorithm-v0-0-13.
What is new
The final model package now carries more than the selected bundle. It also contains the full training state that bundle was saved from, and the candidate bundles preserved during the run, so you can pick training back up from what you were delivered and compare the candidates after the job has finished. Neither of those depends on having configured a checkpoint location while the job ran.
The job now records the batch size and context length it actually used, together with the runtime version it ran on, in the startup log and in the run summary delivered with the model.
Additional details
Inputs
- Summary
You start from trajectories you have already recorded. The training channel points to an Amazon S3 prefix holding one or more Minari-compatible dataset directories, and the dataset_ids hyperparameter selects which of them to train on as paths relative to that channel root.
The dataset is the contract. A Minari dataset already stores trajectories as ordered state and action steps and carries the Gymnasium space definitions for those fields, so the algorithm reads the policy's input and output structure directly from your data, with nothing separate to declare or map. A dataset you already have trains as it stands, and if you are recording new data, the spaces you choose to record are the input and output interface you will get.
That interface can follow the environment you already have. Observations and actions may combine continuous and categorical fields in the same structure, with Dict and Tuple containers nested as needed, so continuous control and discrete choices sit side by side in one hybrid space rather than being flattened into a single numeric vector.
The same contract can be pushed further, because what you put in the observation is also how you steer the policy. Action masks, goals, modes, task IDs, and encoded instruction vectors are learned as part of the state, so a single trained policy can be redirected at run time by what you feed it rather than retrained for each new objective.
If you have a simulator rather than a dataset, that is where the data comes from. A game build or physics simulator can be rolled out to record trajectories, and a controller you already run, scripted or learned, can drive that rollout to produce exactly the behavior you want the policy to learn from.
- Limitations for input type
- Training input must be ordered offline reinforcement learning trajectories stored as Minari-compatible dataset directories containing observations, actions, rewards, terminations, and truncations. dataset_ids must reference paths under the training channel root. Supported leaf spaces are one-dimensional Box, Discrete, MultiDiscrete, and MultiBinary, nested freely inside Dict and Tuple containers. Multidimensional image Box, Text, Sequence, Graph, OneOf, and variable-length structures are not supported. Encode raw text, images, audio, or variable-length inputs as fixed-size numeric fields before recording them. Trajectories from a simulator must be collected outside the training container. The training job never launches the Unity Editor or your game build, and SageMaker trains only from the uploaded dataset.
- Input MIME type
- application/json
{
"observations": [
[0.12, -0.04, 0.31, 0.08, -0.15, 0.22],
[0.14, -0.02, 0.28, 0.09, -0.13, 0.20]
]
}
{"observations":[[0.12,-0.04,0.31,0.08,-0.15,0.22]]}
{"observations":[[0.14,-0.02,0.28,0.09,-0.13,0.20]]}
Input data descriptions
The following table describes supported input data fields for real-time inference and batch transform.
Field name | Description | Constraints | Required |
|---|---|---|---|
dataset_ids | Comma-separated Minari dataset IDs to use for training. Each dataset ID is interpreted relative to the SageMaker training channel root. Example: mujoco/humanoid/simple-v0,mujoco/humanoid/medium-v0 | - | Yes |
max_steps | Maximum number of training update steps, where a step is one training-loop update and not an episode count. This value also sets the delivery schedule: bundles are delivered at 20, 40, 60, 80, and 100 percent of it. | - | No |
batch_size | Training minibatch size. The value is moved to the nearest of 32, 64, 128, 256, and 512, and may be lowered further if the dataset cannot fill it. The effective value is recorded in the exported bundle. | - | No |
context_length | How many trajectory steps the policy reads as history while training. The value is moved to the nearest of 16, 24, 32, 48, and 64, and may be lowered further with batch size if the dataset cannot fill it. The effective value is recorded in the exported bundle and becomes the policy's default history window when it is served. | - | No |
archive_steps | Optional extra delivery points, given as comma-separated training steps, in addition to the five scheduled across max_steps. Use it when you want a finer trend to judge early stopping against. Example: 25000,75000. The hyperparameter guide describes how requested steps are selected. | - | No |
Resources
Support
Vendor support
For product support, contact CCNets at support@ccnets.org .
Website: https://ccnets.org
CCNets provides email-based technical support for this product. Support includes guidance for Amazon SageMaker training job configuration, container usage, supported hyperparameters, CloudWatch log review, S3 model artifact output, and general troubleshooting related to running the algorithm.
This product is intended for technical users familiar with Amazon SageMaker and reinforcement learning workflows. Support does not include custom model development, customer-specific dataset engineering, production system integration, or guaranteed performance tuning unless separately agreed with CCNets.
For enterprise deployment, custom training workflows, or integration support, please contact CCNets to discuss a separate support arrangement.
AWS infrastructure support
AWS Support is a one-on-one, fast-response support channel that is staffed 24x7x365 with experienced and technical support engineers. The service helps customers of all sizes and technical abilities to successfully utilize the products and features provided by Amazon Web Services.
Customer reviews
No customer reviews yet
Be the first to review this product . We've partnered with PeerSpot to gather customer feedback. You can share your experience by writing or recording a review, or scheduling a call with a PeerSpot analyst.