PyTorch 2.12 on Ubuntu 24.04 with Support by Easycloud

Once launched, SSH into the instance. The default username is 'ubuntu'. You can switch to the root user environment by running: sudo su -

This product must be launched on a GPU-backed instance type (G4dn, G5, G6, P4, P5, or P6 family). On a CPU-only instance type such as t3 the driver has no device to bind to and torch.cuda.is_available() will return false.

• OS: Ubuntu 24.04 LTS (x86_64, Minimal Installation)

• NVIDIA Driver: 595.71.05 (open kernel module build)

• PyTorch: 2.12.1 built for CUDA 13.0, with torchvision 0.27, torchaudio 2.11, and Triton 3.7

• Environment: self-contained Python 3.13 virtual environment at /opt/pytorch (about 7.8 GB)

• GPU Libraries: CUDA 13.0 runtime, cuDNN 9.20, and NCCL 2.29, all installed inside the virtual environment

• Included Tooling: JupyterLab 4.6, TorchServe 0.12, tritonclient, SciPy, Pillow

• Containers: Docker 29 with the NVIDIA Container Toolkit 1.19

• Fabric: EFA with libfabric-aws 2.4.0 and the NCCL OFI plugin 1.18 for multi-node training

• System Python 3.12 is kept clean and does not contain PyTorch

• Note: there is no system-wide CUDA installation under /usr/local. The CUDA runtime is provided inside the /opt/pytorch environment, which is what PyTorch uses. If you need a system CUDA Toolkit with nvcc for compiling custom CUDA extensions, use our GPU Base CUDA AMI instead.

1. Launch on a GPU instance type. Recommended: g5.xlarge (A10G, 24 GB GPU memory) for single-GPU training, fine-tuning, and inference, or a larger g6, p4, p5, or p6 instance for multi-GPU work. Root volume is 30 GB with about 7.8 GB used by the environment; increase it at launch if you plan to store datasets or checkpoints.

2. SSH in as 'ubuntu' and activate the PyTorch environment (the login banner also shows this): source /opt/pytorch/bin/activate

3. Verify GPU access: python -c 'import torch; print(torch.version, torch.cuda.is_available(), torch.cuda.get_device_name(0))' Expect the PyTorch version, True, and your GPU model. You can also check the driver level with: nvidia-smi

4. Start JupyterLab for interactive work. With the environment activated, run: jupyter lab --ip 0.0.0.0 --port 8888 Then open http://<YOUR_IP>:8888 and paste the access token printed in the terminal. Bind to a private address or use an SSH tunnel if the notebook should not be reachable from the internet.

5. Serve models with TorchServe. With the environment activated, run torchserve against your model archive; the inference API listens on port 8080 and the management API on port 8081.

6. Run GPU containers when you prefer your own image: sudo docker run --rm --gpus all <your-image>

7. For multi-node distributed training, the EFA stack and the NCCL OFI plugin are already installed; place your instances in the same cluster placement group and use EFA-capable instance types.

8. Manage services with systemctl start/stop/restart/status. JupyterLab and TorchServe are not enabled as services by default - you start them on demand from the activated environment as shown above.

• SSH (Port 22): Required for administration, activating the PyTorch environment, and running training or inference jobs.

• TorchServe Inference API (Port 8080): Serves model predictions once you start TorchServe; closed until you do.

• TorchServe Management API (Port 8081): Registers, scales, and unregisters models in a running TorchServe instance.

• JupyterLab (Port 8888): Interactive notebook interface, available once you start JupyterLab manually.

• Security Recommendation: For production environments, strictly limit access to these ports to trusted IP addresses only via cloud Security Groups or the local firewall.