2.5.2 Release Notes

HAI-15848

Security hardening of Admin Panel credential handling during upgrades.

Blocker

Ensuring alignment with standard password management approaches.

HAI-15281

While creating a GPUaaS instance, the Region and Scenario dropdowns were often empty, even though the API responses contained the required data.

High

Customers can reliably pick a region and scenario and create GPUaaS instances without retrying or reloading the page.

HAI-15581

An idle tenant could keep a large share of GPU memory despite repeated reclaim attempts, leaving active tenants on the same card short of VRAM.

Blocker

Idle tenants now reliably release GPU memory, so active tenants on the same card get the VRAM they need and no longer see "GPU is physically FULL" errors.

HAI-15620

A tenant that could not allocate GPU memory kept the GPU well beyond its scheduled turn, failing repeatedly and blocking co-tenants on the same card.

Critical

Tenants that cannot allocate no longer hog the GPU, so time slices are shared fairly and throughput across the pool improves.

HAI-15236

On an oversubscribed shared card, a non-vLLM tenant (e.g. Ollama or llama.cpp) loading its model while the card was busy got no working GPU service. Its model either fell back to CPU and timed out, or failed to load at all.

Critical

LLM instances started on a busy shared GPU now get working GPU service, instead of working or failing depending on when they happened to load.

HAI-10946

An idle tenant's VRAM was not freed for a waiting tenant. The result was a crash or a silent, permanent hang.

High

Idle workloads now hand GPU memory to waiting workloads as intended, preventing crashes and indefinite hangs on shared GPUs.

HAI-10915

A tenant joining a shared GPU was starved until the tenant already on the card finished its job.

High

New workloads start getting GPU time promptly instead of waiting for existing jobs to finish.

HAI-10896

Two training workloads sharing one GPU under VRAM oversubscription could deadlock permanently, with no error shown to either tenant and no recovery.

High

Concurrent training jobs on a shared GPU no longer hang forever, so customers do not lose compute time to stuck jobs.

HAI-15765

Improved isolation between tenants sharing a node in the GPU-sharing service.

High

Stronger isolation and stability for every tenant on shared GPU infrastructure.

HAI-15705

A workload that could not reach the GPU-sharing service at start-up hung forever with no error. This hit processes started with a minimal environment, such as CI jobs, sandboxes and some system services.

High

Workloads now fail fast with a clear error instead of hanging silently.

HAI-15639

A vLLM server on a shared GPU could die while idle after start-up: the next request after a pause of about 90 seconds returned HTTP 500 "CUDA error: unknown error", and vLLM had to be restarted.

High

Inference servers stay healthy through idle periods, so customers no longer come back to a dead endpoint after a short break.

HAI-15836

When GPU sharing refused a PyTorch memory allocation over the tenant's limit, PyTorch crashed with an internal assert instead of the normal out-of-memory error.

High

Users get the standard PyTorch out-of-memory error they can handle in their code, instead of a confusing crash.

HAI-15621

Under normal multi-tenant load, GPU-sharing status reports (haisharectl --status-json) could silently come back incomplete.

High

Operators and monitoring tools now get complete, reliable GPU-sharing status reports.

HAI-15341

When a pool's workloads no longer fit in GPU memory, the platform raised no signal: cards appeared healthy while tenants failed, and the logs were cluttered with routine entries.

High

The platform now signals when a pool cannot fit its workloads, and logs are much less noisy, so operators can spot and fix capacity problems quickly.

HAI-13770

Restoring the scheduler configuration turned on time-scheduling for GPUs running in spatial mode, putting them in an unexpected state.

High

Spatial-mode GPUs keep their intended scheduling behaviour after a restore, avoiding unexpected locking and performance problems.

HAI-15684

Fixed a rare error in GPU request handling for multi-threaded workloads such as PyTorch and vLLM.

Normal

Removes a source of rare, hard-to-diagnose errors in multi-threaded GPU workloads.

HAI-13280

In some cases, mainly during rolling upgrades, a tenant's processes were placed on different GPUs instead of the same card.

Normal

A workload's processes stay on the same GPU, avoiding performance loss and inconsistent behaviour during upgrades.

HAI-15567

Disabling redundancy on a vDisk failed with 422 "Deadlock in redundancy modification", and no replicas were removed.

Blocker

Operators can turn off vDisk redundancy and free the replica storage as expected.

HAI-15397

Enabling redundancy on a single-replica vDisk failed because the disk was reported as "degraded". The failed attempt also left the disk wrongly marked as redundant, which could stop it from activating.

Blocker

Operators can enable redundancy on existing disks, and a failed change no longer leaves a disk in a state that can block activation.

HAI-15396

The "Manage vDisk redundancy" action was disabled in the UI (Instance › Virtual Disks and Virtual Disks › Settings), so a disk's redundancy could not be enabled or changed.

Blocker

Operators can manage vDisk redundancy directly from the Cluster Panel again.

HAI-15516

The Create Instance wizard crashed on the 4th step with "TypeError: r.dnsIps.join is not a function", so no instances could be created.

Blocker

Users can complete the Create Instance wizard and deploy VMs again.

HAI-15364

The storage network never appeared in the networks list when the storage interface was a plain (non-bridge) interface, such as a VLAN.

Normal

All configured networks, including storage, are now visible in the Cluster Panel, so operators get an accurate view of the cluster's networking.

HAI-15579

Preparing a model with LoRA support forced eager mode, so every default model served about 6x slower.

High

LoRA-enabled models serve at full speed again.

HAI-15596

haid ran kubectl on every chat request while holding the runtime lock, adding about 0.11 s per request and queueing users behind each other.

High

Lower, more consistent chat latency when many users are active.

HAI-15806

A long pasted prompt stalled every other user's first token for the whole prefill, because there was no per-request prefill cap.

High

One user's long prompt no longer freezes responses for everyone else on the endpoint.

HAI-15597

Stream relays waited for 1,024 bytes before forwarding, holding back the first token by about 0.06 s.

Normal

Faster first token on streaming responses.

HAI-15757

The wrapper, haid and worker listened with a connection backlog of 5, so bursts of concurrent chats had connections dropped or reset (p95 of 1,061 ms at 32 concurrent requests, resets at 64).

Normal

Endpoints absorb traffic bursts without dropped connections.

HAI-15798

The worker's state endpoint walked the whole metrics registry twice on every chat (1.35 ms idle, 25–159 ms under contention).

Normal

Less overhead on every request, especially under load.

HAI-15802

A model on a dedicated GPU now starts with 95% GPU memory utilisation instead of 75% (+34% KV cache measured on a 9B model).

Normal

More concurrent requests and longer contexts on the same GPU.

HAI-15751

MTP speculative decoding for Qwen3.5 can now be enabled from the model card (off by default).

Normal

Operators can turn on faster generation for supported models.

HAI-15241

Certified models now auto-fill every Prepare parameter.

High

Faster set-up and fewer misconfigured deployments.

HAI-15577

Deploy rejected a service whose target GPU name was written differently from the pool's GPU name.

High

Deploys succeed regardless of how the GPU name is formatted.

HAI-15759

Multimodal models served with the 8,192-token fallback context, or failed to load, because the context window was only read from the top level of config.json.

High

Multimodal models get their full context window.

HAI-12525

haid's cleanup deleted a model blob but left the registry entry pointing to it, so the next Prepare saw a corrupt model instead of a missing one.

High

Prepare works after cleanup without manual repair.

HAI-12728

A serving setting could never be cleared: leaving a field out kept its old value.

High

Operators can reset serving settings back to their defaults.

HAI-13983

A single-GPU snapshot survives a reboot on the same driver. This is now explicit (TF_K1_GOLDEN_SURVIVES_REBOOT), and the misleading "cannot be restored" message is no longer logged.

High

Faster recovery after a reboot and no false alarms in the logs.

HAI-15576

Token Factory nodes now use the 580 long-term-support NVIDIA driver and reboot after a driver change.

High

A consistent, supported driver on every Token Factory node.

HAI-12680

Unprepare reclaimed no disk space (478 GB of snapshots remained on a node reporting not_prepared).

Normal

Disk space is freed when a model is unprepared.

HAI-12691

The first deploy after a Prepare always discarded the snapshot and captured it again.

Normal

Faster first deploy.

HAI-12701

The context length of multimodal models was missed because it lives in config.text_config.

Normal

The correct context length is shown and used.

HAI-12727

The Prepare form offered parser names that the target engine image cannot load.

Normal

Only valid options are offered, so fewer Prepares fail.

HAI-12738

The model catalog moved to a remote asset and now includes per-model runtime requirements.

Normal

The catalog can be updated without a platform release.

HAI-15593

Every CUDA-graphs Prepare recompiled from scratch because compile caches were not kept.

Normal

Faster repeat Prepares.

HAI-15594

Prepare now picks the fastest weight downloader by measurement.

Normal

Faster model downloads.

HAI-15595

Prepare's snapshot job took 69 s even though the snapshot itself takes about 7 s.

Normal

Shorter Prepare times.

HAI-15618

The runtime image check looked for the :dev tag, so a version bump never reached a node that already had the image.

Normal

Nodes pick up new runtime versions reliably.

HAI-15707

The fit advisor refused FP8 on A100 (it works there as W8A16 Marlin) and had no catalog entry for 43 of the 62 supported GPUs.

Normal

Accurate model-fit advice across many more GPU types.

HAI-15733

Deploy discarded the Prepare snapshot when the model card named the same quantization differently.

Normal

Faster deploys that reuse the prepared snapshot.

HAI-15735

With the compile cache enabled, a snapshot captured from seeded cache files could never be restored because of a file-permission mismatch.

Normal

The compile cache can be enabled safely.

HAI-15736

With the compile cache enabled, a restored engine that needed a new kernel could not write to its cache directory and crashed.

Normal

Restored engines keep running when they need new kernels.

HAI-15743

Prepare downloaded the latest model revision instead of the one pinned on the model card, fetched unused weight copies, and could exceed the host's RAM.

Normal

Reproducible, smaller downloads with no host memory overrun.

HAI-15746

New model card settings for the recurrent-state data type and FP8 KV cache, rejected where the engine cannot serve them.

Normal

More tuning options, with validation that blocks unsupported combinations.

HAI-15748

Memory sizing never found the shared weight file on A100, so models without a set budget were sized at 0 GB.

Normal

Accurate memory sizing and placement.

HAI-15805

Compiled kernels are now shared across model card changes for the same model (deploy time on a 9B model dropped from 332 s to 165 s).

Normal

Redeploys after a settings change take about half the time.

HAI-15816

haid accepted FP8 KV cache on workers that never passed it to vLLM, so the endpoint used the default KV type while reporting FP8.

Normal

The reported configuration matches what is actually serving.

HAI-15741

The status endpoint reported the configuration requested from vLLM, not what the engine actually resolved.

Normal

Operators see the real serving configuration.

HAI-15239

Token Factory placed training and serving workloads on GPUs already pinned by GPUaaS customer pods.

Critical

Token Factory no longer competes with or disrupts GPUaaS customer workloads.

HAI-15480

Pods that were terminating were dropped from the customer-GPU protection list, so Token Factory could place work on a card a deleting pod still held.

High

No placement collisions while customer pods shut down.

HAI-15483

GPU protection timers used wall-clock time, so a system clock change broke them.

High

GPU protection stays correct even if the node's clock changes.

HAI-12466

Seven abandoned tenant footprints held 28.7 of the A100 node's 64 CPUs while serving nothing, starving fine-tuning.

High

CPU capacity is freed for active work, including fine-tuning.

HAI-15481

A temporary, retryable "GPU protection unavailable" response was reported as a permanent restore failure (500).

Normal

Temporary conditions are retried instead of failing the request.

HAI-15482

When no GPU was placeable, admission counted every GPU in the pool, including protected ones, as capacity.

Normal

Accurate capacity checks and no over-admission.

HAI-15484

The GPU inventory call to nvidia-smi had no timeout, and the protection refresh waited on it.

Normal

A slow or hung GPU query can no longer stall placement.

HAI-15815

The GPU architecture cache never re-queried nvidia-smi once populated, and the 20 s query ran while holding the cache lock.

Normal

GPU information stays current and lookups no longer block.

HAI-13957

A fine-tune that was ever queued could never start.

High

Queued fine-tuning jobs now start when capacity frees up.

HAI-15627

Fine-tuning failed within 31 s on a single-GPU node that was also serving a model, because the trainer reserved 48 GiB of RAM but only needs about 3.6 GB.

High

Fine-tuning runs alongside serving on single-GPU nodes.

HAI-13222

The LoRA trainer used an unseeded random number generator, so identical runs produced different adapters.

Normal

Fine-tuning results are reproducible.

HAI-15242

Measured throughput and concurrency for a fine-tuned model are shown before renting serving capacity.

Normal

Customers can size and budget serving before they commit.

HAI-15797

After the inference engine died inside a running pod, the pod still reported Ready and every chat returned EngineDeadError, with no recovery.

High

Endpoints recover automatically instead of failing every request.

HAI-15721

A haid restart caused customer chats to fail with 502, because the runtime wrapper gave up on the first refused connection.

Normal

Chats ride through haid restarts.

HAI-15730

A chat that arrived while its pod was being deleted waited 90 s and then got a 500.

Normal

Faster, correct handling of requests during pod turnover.

HAI-15799

Reloading weights in GDS mode silently loaded wrong weights for some parameters.

Normal

Models always serve with the correct weights.

HAI-15800

A kubectl call that hung past its timeout was not handled, so the fallback was skipped and the caller saw a dropped connection.

Normal

Requests recover when a cluster call stalls.

HAI-12726

haid reported an unrelated clean-up error instead of the real failure, hiding every capture error.

Normal

Operators see the actual cause of capture failures.

HAI-13164

Some haid failures dropped the connection with no response, so callers saw only "RemoteDisconnected".

Normal

Callers get a proper error response.

HAI-13958

/v1/completions accepted a request, discarded the caller's prompt, and answered a hard-coded question with 200 OK.

High

The completions API answers the prompt that was sent.

HAI-15580

Qwen3 replies included the model's internal reasoning text in the answer content.

High

Clean answers, with reasoning separated as expected.

HAI-15626

Invalid chat parameters returned 502 instead of 400, so OpenAI SDKs kept retrying them.

High

Clients get a clear validation error and stop retrying bad requests.

HAI-15628

The panel throttle returned 429 to Token Factory keys below their rate-limit policy, without the OpenAI error format or a Retry-After header.

High

Keys get their full allowance, and SDKs back off correctly when limited.

HAI-15622

A constant Idempotency-Key billed only one chat call per model per day, whatever the prompts.

High

Every chat call is billed correctly.

HAI-15623

A chat request with stream set to "true" or 1 was streamed in full but recorded at zero cost.

High

Streaming usage is billed correctly.

HAI-15873

Fine-tuning ran with no fine-tuning rate configured and billed nothing.

High

Fine-tuning is priced and billed correctly.

HAI-15624

A non-string knowledge_base_id was silently ignored, so the chat answered without grounding and was still billed.

Normal

Invalid input is rejected instead of giving ungrounded answers.

HAI-12836

Deleting a document from a knowledge base did not reduce the base's vector count.

Normal

Accurate knowledge base size and usage figures.

HAI-12837

Re-uploading a document with an Idempotency-Key reused from a deleted document returned 500.

Normal

Re-uploads are handled gracefully.