CCNets Causal GPT-RL Training Algorithm

This release updates CCNets Causal GPT-RL Training Algorithm to SageMaker Algorithm version cgrl-training-algorithm-v0-0-9 and provides updated training and inference containers for GPT-style offline reinforcement learning.

This release supports fixed-shape Gymnasium observation and action schemas built from one-dimensional Box, Discrete, MultiDiscrete, and MultiBinary leaf spaces. Dict and Tuple containers may be nested and may combine continuous and categorical fields within the same observation or action structure.

Verified exported-bundle combinations include Dict observations that combine Box and Discrete fields, and Tuple or Dict action spaces that combine continuous Box branches with Discrete branches. The same schema system also supports structured Unity ML-Agents datasets such as Tuple observations with Discrete or MultiDiscrete action branches.

Unity ML-Agents integration follows an offline training workflow. Users collect trajectories from a Unity ML-Agents environment, package the trajectories as Minari-compatible datasets, upload the datasets to Amazon S3, and train the policy through SageMaker. The SageMaker training container does not launch the Unity Editor or execute the customer’s Unity build during training.

The public repository includes Unity trajectory collection examples and an ONNX deployment example for running a trained policy against a Unity ML-Agents build. ONNX deployment is a downstream integration example and is separate from the primary SageMaker model.tar.gz output.

Public benchmark resources are available for MuJoCo policy bundles and a Unity ML-Agents Crawler ONNX policy. Benchmark results are environment-, runtime-, and protocol-specific and should be used as reproducible references rather than guarantees for customer datasets or environments.

The default training configuration is designed to provide a practical starting point for common MuJoCo and Unity ML-Agents workflows, reducing the need for extensive initial hyperparameter tuning. Additional adjustment may still be useful depending on dataset quality, reward scale, trajectory coverage, observation and action design, and environment-specific behavior.

This release supports SageMaker training jobs, Minari-compatible dataset input through the SageMaker training channel, configurable training hyperparameters, CloudWatch training logs, checkpoint generation, S3 model artifact export, and compatible inference artifact loading.

Buyer-provided observation and action spaces define the trained policy interface. This allows the policy to reflect the buyer's own sensors, game state, application state, action choices, control vectors, action masks, goals, modes, task IDs, or encoded instruction vectors when these fields are included in the dataset.

The trained policy can be used through a reset, act, and observe rollout loop. The workflow supports state-based behavior control, prompt-like control through buyer-provided instruction vectors, batched rollout handling with per-episode resets, and stable episode boundary handling for cached inference.

CloudWatch Logs may display offline metrics such as action negative log-likelihood and Forecast/StepReward. Forecast/StepReward is an experimental model-estimated per-step reward metric intended to help buyers monitor offline training progress direction. These offline metrics are training progress indicators and do not guarantee final performance in the buyer's target environment.

The final model artifact is exported as model.tar.gz and may include reports/summary.json, a canonical inference bundle, and snapshot candidate bundles depending on the training configuration.

SageMaker checkpoint outputs are separated from final inference artifacts. Checkpoints are intended for resume or retraining workflows, while bundle and snapshot outputs inside model.tar.gz are intended for downstream evaluation or compatible inference workflows.