Qwen3.8-27B Multimodal Model (INT4)

Runs inside your own AWS account: your prompts and images are processed on a SageMaker endpoint you control, in network isolation, and are never sent to a third-party API.

This SageMaker model package serves Qwen3.8-27B, an open-weight (Apache-2.0) multimodal language model from Alibaba Cloud's Qwen team, as an OpenAI-style chat service. It accepts text and images (and one short video) in the same request and answers in text.

What it does for each supported content type

• application/json, real-time endpoint: a chat request (a list of messages, or just {"inputs": "..."}) returns a chat completion. Messages can include images. A request can stream its answer.
• application/json or application/jsonlines, batch transform: one chat request per line, one response line per input line, in order. Tested with a 24-record job on ml.g6e.xlarge.

Typical uses are question answering and summarisation over long documents, reasoning and code assistance, extracting information from screenshots, charts and scanned pages, and agent workflows that call tools. The model thinks before it answers by default; thinking can be switched off per request for short, fast replies.

Context: 262,144 tokens by default. The container can be started with a larger context (up to 524,288 tokens on one instance, using YaRN) by setting the environment variable MAX_MODEL_LEN on the SageMaker model; that mode was tested with one request of about 456,000 tokens and has not been evaluated for short-prompt quality.

Runs on vLLM 0.29 using the 4-bit (W4A16, group size 128) checkpoint from Red Hat AI; the vision tower is not quantised. The model is included in the container, so the endpoint needs no network access at runtime.

We welcome your feedback at support@sigmodata.com .

Known limitations

• Use streaming for long or thinking-heavy requests. A real-time InvokeEndpoint call is cut off at 60 seconds by SageMaker, and this model can think for longer; call InvokeEndpointWithResponseStream (set "stream": true) instead. A prompt of about 200,000 tokens takes about two minutes before the first token.
• On harder questions the model sometimes thinks until it reaches the output limit without answering: 10 of 56 questions in our run did so at a 4,096-token limit. Set a generous max_tokens, switch thinking off (chat_template_kwargs: {"enable_thinking": false}), or lower the reasoning effort for latency-sensitive work.
• Images must be sent inline as base64 data: URLs. The endpoint has no network access and cannot fetch http(s) image links. At most 8 images and 1 video per request by default.
• One GPU per container: up to 8 requests are processed together. For more throughput add instances.
• Answers can be wrong or invented. Test on your own data and keep a person in the loop for decisions that matter.
• Not evaluated: any language other than English, video, and any SageMaker instance type other than ml.g6e.xlarge.

Model and training data

Base model Qwen3.8-27B by Alibaba Cloud (Qwen team), Apache-2.0, used as published. Its training data is not published in detail; see the model card at https://huggingface.co/Qwen/Qwen3.8-27B . The 4-bit checkpoint is RedHatAI/Qwen3.8-27B-INT4 (Apache-2.0), pinned to revision 91bd022d5b49442a868bc35008f6c21e1860edfa. Sigmodata did not train or fine-tune either. Qwen is a trademark of its owner; this product is not made or endorsed by Alibaba Cloud or Red Hat.

Measured performance

• Accuracy, measured by us through this package on a SageMaker endpoint at a 4,096-token output limit: 37 of 40 correct on a 40-question sample of GSM8K, and 39 of 56 correct on a 56-question stratified sample of MMLU-Pro (4 per category). These are small samples (the 95 percent interval on 56 questions is about plus or minus 12 points) meant to show the package works, not a benchmark.
• Reported by the checkpoint's publisher (Red Hat AI, on its model card, for the unquantised baseline versus this INT4 build): MMLU-Pro 84.46 vs 83.45, GPQA Diamond 89.23 vs 87.88, GSM8K Platinum 95.75 vs 96.77. We have not reproduced these.
• Throughput on ml.g6e.xlarge (one NVIDIA L40S, 48 GB), measured on the same container outside SageMaker: about 60 output tokens per second for a single request, and about 175 and 386 tokens per second in total with 4 and 8 requests at the same time.
• Memory: the container holds the model in 18.5 GiB and has room for about 560,000 tokens of context in total (2.15 requests of 262,144 tokens).
• Latency on the SageMaker endpoint (one request at a time, thinking on, 4,096-token limit): the first streamed token arrived after a median of 0.2 s (95th percentile 0.4 s). A GSM8K answer took a median of 3 s (95th percentile 23 s) and an MMLU-Pro answer a median of 9 s (95th percentile 67 s).
• Start-up: a new endpoint reached InService in about 12 minutes, including pulling the 22.4 GB compressed container image.