Skip to main content
You are here: Releases Notes

2.5.3 Release Notes

Summary

Overview - Release V2.5.3 - is a patch release for the HOSTED.AI Platform with significant resolved issues, including 7 Blocker, 10 Critical and 68 High priority items. It focuses on platform security hardening, GPU-sharing (HAIShare) reliability, Token Factory quality and Admin Panel, Cluster Panel and KVM stability.

Key highlights

  • Security hardening: KVM nodes now ship with a host firewall, and guest VMs can no longer reach host services. VM consoles are password-protected and no longer listen on every interface. SSH password login is disabled. Default cluster panel passwords are replaced with random per-cluster passwords and rotated on upgrade. System recipes cannot be modified through the API, and the recipe runner is locked down to prevent privilege escalation. Credentials are no longer written to the admin log, playbook logs or tenant nodes. Go toolchain and library vulnerability advisories are patched.

  • GPU sharing (HAIShare): More reliable multi-GPU (NCCL) and tensor-parallel workloads. Tenant migration is safer. The scheduler no longer livelocks a card shared by two non-evictable tenants or lets an idle tenant hold memory indefinitely. Several tenant crashes are fixed, including a customer-reported eviction loop on RTX PRO 6000 dynamic pools and a crash caused by an in-place library update.

  • Token Factory: OpenAI-compatible API behaviour is corrected (n, seed, top_k, response_format on reasoning models, thinking mode and context-length errors). RAG indexing is faster and validates file types, and deleting a document or knowledge base now cleans up fully. Fine-tune and deploy lifecycles are more robust. Usage, billing and metrics figures are accurate, team isolation and error redaction are stronger, and enabling and tearing down Token Factory on a live region is more reliable.

  • Admin Panel, Cluster Panel and KVM: Nodes can be added on a management IP on a VLAN sub-interface (customer-reported). Joining a secondary node to a KVM cluster and HA on private management networks now work. Reducing redundancy on a running VM no longer destroys it, Configure Racks works over HTTP, and a wrong 2FA code shows an error instead of returning to login.

  • User Panel and instances: Running pods no longer flip to Pending in the instance list. vGPU counts and shared-pool capacity are reported correctly. A new instance is no longer deleted shortly after launch, and re-uploading a recipe template no longer breaks the services that use it. The deploy wizard respects the team's resource policy again.

  • Provisioning: Enable GPUaaS no longer deletes snapshot storage on a live region, and GPU node setup and reboots are faster.

Improvements to existing features

1. GPU Sharing (HAIShare)

Ticket

Description

Priority

Business impact / value to end users

HAI-15479

HAIShare V2 CDI now supports multiple glibc versions, so containers built on older or newer base images run under GPU sharing.

High

Customers can run a wider range of container images on HAIShare V2 pools without compatibility failures.

HAI-15562

HAIShare V1 CDI now supports multiple glibc versions.

High

The same image compatibility for existing HAIShare V1 pools.

HAI-16573

Temporal sharing switches tenants faster (delta freeze and region hand-off), and credit-based preemption is fixed so a tenant cannot keep the card at others' expense.

Normal

Less idle GPU time when tenants take turns on a shared card, giving each tenant more usable GPU time and fairer turns.

HAI-16577

Memswap disk spill now moves evicted memory chunk by chunk and keeps as much as possible off disk.

Normal

Much faster tenant switches for pods that are short of host RAM.

2. GPU node setup and provisioning

Ticket

Description

Priority

Business impact / value to end users

HAI-15591

Enable GPUaaS no longer installs Prometheus, GPU Operator and Envoy Gateway one after another.

Normal

Regions are enabled faster.

HAI-15588

GPU node reboots no longer wait about 2 minutes for a network port that has no IPv4 address.

Normal

Faster node reboots and maintenance windows.

HAI-15587

The NVIDIA driver kernel module is compiled only for the running kernel instead of every installed kernel.

Normal

Faster driver installs during node setup.

HAI-16279

Capacity can be checked for many services in one call, and a "full" answer now matches exactly when instance creation would be refused.

Normal

Partners and the UI can show accurate availability before deploying, avoiding failed instance creation.

3. Token Factory

Ticket

Description

Priority

Business impact / value to end users

HAI-15900, HAI-15903

Prepare lists HOSTED.AI-certified models first, with a badge and auto-filled settings. The Prepare dialog keeps model, placement and GPUs per replica visible, auto-selects single options, and moves engine settings into a closed Advanced section with sensible defaults.

Normal

Customers prepare models faster and with fewer configuration mistakes.

HAI-16481, HAI-16549

Prepare fails early with a clear reason instead of late and opaquely. Error messages are clearer, a lack of capacity is no longer reported as a server error, container images are pinned and RAG query gets health probes.

Normal

Fewer confusing failures, and problems are easier to diagnose and act on.

HAI-15889

RAG parsing keeps tables, scanned pages and layout, so grounded answers can cite table cells.

Normal

More accurate answers from real-world documents such as reports and scanned PDFs.

HAI-16017

Indexing shows every step live (queued, parse, layout, tables, OCR, chunk, embed, write) with counts, speed, elapsed time and ETA.

Normal

Customers can see exactly how far indexing has progressed instead of waiting on an opaque "processing" state.

HAI-15908, HAI-16100, HAI-16407

Citations include the quoted passage and page number, the original document can be downloaded and opened at the cited page, and grounded answers carry per-citation retrieval signals and timings.

Normal

Users can verify answers against the source quickly, which builds trust in grounded chat.

HAI-15909

Metrics add window-level TTFT, TPOT and end-to-end latency percentiles (p50/p95/p99).

Low

Accurate latency reporting for performance and SLA tracking.

HAI-16036

The Token Factory admin team can see customer revenue against the GPU cost of serving it, per model.

Normal

Operators can judge the profitability of each model and price it accordingly.

HAI-15786, HAI-15603

API keys verified once are cached instead of re-checked on every call (about 54 ms each), and the serving token is no longer minted inline on a customer's call.

Normal

Lower latency on every Token Factory inference request.

HAI-16228

The Token Factory load test runs on the server, next to the model.

Normal

Load-test results reflect true model performance rather than network overhead.

HAI-16482, HAI-16517, HAI-16741

Token Factory node overhead is reduced: fewer GPU status polls, the cache librarian is off by default, versions are reported, and RAG query resources are right-sized.

Low

More node CPU and memory are left for customer workloads.

HAI-16592

"Tested" and "certified" labels now claim only what a completed test actually proves.

Normal

Customers can rely on certification labels when choosing a model.

Bug fixes

1. Security – Admin Panel, recipes and credentials

Ticket

Description

Priority

Business impact / value to end users

HAI-17141

Recipe execution is locked down, and system recipes can no longer be cloned and modified to run arbitrary commands.

Blocker

Prevents privilege escalation through the recipe feature.

HAI-17142

System recipe templates are now read-only through the API.

Blocker

Built-in automation cannot be tampered with.

HAI-17144

Local copies of system recipes are now created automatically at startup.

Blocker

Platform automation keeps working after the recipe lockdown.

HAI-17175

The recipe feature in the Admin Panel is restricted: inventories and recipe runs can no longer be created through the API, and recipe edits are checked more strictly.

Critical

Removes a route to root access and to private-key leaks.

HAI-16065

Improved access controls for Ansible logs and enhanced sensitive information masking to prevent unintended exposure of confidential data.

Critical

Reduces the risk of sensitive configuration details being accessible to lower-privileged Admin Panel users..

HAI-17224

Enhanced security measures to prevent the exposure of sensitive authentication information through application logs.

High

Enhances the protection of sensitive authentication information within application logs

HAI-16765

The User Panel was built with an older Go version that contained known security advisories. It now uses the patched version.

High

Removes known standard-library vulnerabilities from the shipped package.

2. KVM and Cluster Panel

Ticket

Description

Priority

Business impact / value to end users

HAI-16231

Reducing redundancy on a running VM could delete its live disk copies and destroy the VM. The operation is now safe.

Blocker

Prevents VM data loss during storage changes.

HAI-16650

When a User Panel synced several KVM clusters, images never appeared, so no VMs could be created. Images now sync correctly.

Blocker

Customers can deploy VMs in every KVM region.

HAI-16890

The Create new rack button did nothing when the Cluster Panel was opened over HTTP. It now works.

Critical

Operators can complete cluster setup.

HAI-16722

Joining a secondary node to a KVM cluster failed with a generic error. Duplicate nodes are now caught when added, and join works.

High

Faster, clearer cluster expansion.

HAI-17234

On private management networks, enabling HA gave peers an unreachable address. The correct management address is now used.

High

HA can be enabled on private networks.

3. Admin Panel, User Panel and GPUaaS

Ticket

Description

Priority

Business impact / value to end users

HAI-16181

Adding a node whose management IP is on a VLAN sub-interface failed. VLAN interfaces are now detected.

Blocker

Customers with VLAN or bonded networks can onboard nodes.

HAI-16889

NVIDIA L4 GPU nodes failed to initialise because the pinned driver 550 could not build on newer kernels. L4 nodes now use a compatible driver.

Blocker

L4 GPU nodes can be added and come online without manual work.

HAI-16109

Entering a wrong 2FA code sent the user back to the login page. An error now appears on the 2FA page.

High

Smoother, clearer sign-in.

HAI-16697

Re-uploading a recipe template left existing services pointing at a deleted recipe. Services now follow the new template.

High

Services keep working after recipe updates.

HAI-16643

Pool capacity reported room for only one pod on every shared pool. It now shows the real free capacity.

High

Customers see accurate capacity and can place more workloads.

HAI-16749

Running pods briefly showed as Pending every 5 minutes. Status is now stable.

High

Accurate status and fewer false alarms.

HAI-17147

The deploy wizard offered regions outside the team's resource policy. Only permitted regions are shown now.

High

Teams stay within their allowed regions.

HAI-17209

Subscribing a team to a HAIShare V2 pool always failed with "Insufficient resources". Subscriptions now work.

High

Teams can use HAIShare V2 pools.

HAI-16763

Removing a pod could delete the team namespace while it still held a subscription, breaking the team's next pod. This is now prevented.

High

New pods no longer fail after a teardown.

4. GPU sharing (HAIShare)

Ticket

Description

Priority

Business impact / value to end users

HAI-16319

A GPU shared by two tenants that could not be evicted could stall for hours with nothing serving. The scheduler now recovers.

Blocker

Prevents silent outages on shared GPUs.

HAI-16079

A ComfyUI tenant crash-looped during idle memory eviction. Eviction now handles this memory correctly.

Critical

Stable image-generation workloads on shared GPUs.

HAI-16080

Updating the HAIShare library in place crashed running workloads. Updates now replace the file safely.

Critical

Upgrades no longer crash running customer pods.

HAI-16190

A node could hang when the device plugin started, crashing all shared tenants. GPU operations on evicted memory are now properly gated.

High

Prevents node-wide outages.

HAI-16718

An idle co-tenant could force a running vLLM workload into host memory every ~12 seconds. Sharing mode now selects correctly.

High

Customers get full GPU performance.

HAI-15745

Paused tenants could exhaust host RAM, killing other tenants' workloads. Host memory use is now limited.

High

Prevents cascading workload failures.

HAI-15766

A failed freeze during migration reported success but left memory behind. Migration now checks and rolls back.

High

Safe workload migration.

HAI-16104

Migration to a GPU without free memory left the workload frozen with no error. It now rolls back and reports the failure.

High

Workloads are not left stuck.

HAI-15850

If a pod's serving process died, its GPU calls hung with no error. They now fail immediately.

High

Faster failure detection and recovery.

HAI-16027

Two multi-GPU (tensor-parallel) jobs sharing GPUs could deadlock. Freezes are now coordinated across all GPUs of a job.

High

Reliable multi-GPU workloads.

HAI-16744

A pod's own multi-GPU job was not recognised as multi-GPU, so one GPU of the job could be moved alone. Such jobs are now detected.

High

Prevents multi-GPU job failures.

HAI-16194

A tenant stuck after a partial restore kept its GPU memory forever and starved others. Memory is now released.

High

Fair GPU sharing.

HAI-16039

Programs using cuDNN's CUDA-graph API crashed the tenant's serving process. These calls are now supported.

High

Wider framework compatibility.

HAI-16694

Multi-GPU communication (NCCL) in shared pods ran over TCP at 1–5% of native speed. Direct GPU-to-GPU transfer is now enabled.

High

Up to ~100x faster multi-GPU training and inference.

HAI-16733

Multi-GPU initialisation failed for some pods because of a GPU address clash. Addresses are now unique.

High

Multi-GPU pods start reliably.

HAI-14009

PyTorch's expandable memory segments setting was stripped. It is now passed through.

High

Better memory efficiency for PyTorch and vLLM.

HAI-17241

A new tenant could inherit the previous tenant's connection limit, so its server failed to start. Limits are now reset.

High

Workloads start reliably on reused slots.

5. Token Factory

Ticket

Description

Priority

Business impact / value to end users

HAI-16530

Fine-tuned adapters stayed linked to a deleted base model, so chats failed while the UI showed Ready. Adapters now follow the current base model.

Critical

Fine-tuned models keep working.

HAI-15494

Token Factory can be enabled on a live region from the controller node without a reset. Access for finished pods is also removed.

High

Enable Token Factory with no downtime.

HAI-12039

Turning Token Factory off for a region did not remove its components. They are now torn down automatically.

High

Clean region disable with no leftover endpoints.

HAI-15617

Enable GPUaaS ran steps meant to be skipped, which could delete snapshot storage on a live region. These steps no longer run.

High

Prevents data loss on live regions.

HAI-15866

Fresh installs did not store the Token Factory admin token, so the first model prepare failed. The token is now stored.

High

Token Factory works right after install.

HAI-16052

Token Factory could be installed on HAIShare V2 regions, where it does not work. This is now refused.

High

Prevents broken installs.

HAI-16053

Live enable was refused on panels with a default install. It now works with the default configuration.

High

Live enable works out of the box.

HAI-16061

Enable referenced a model image that did not exist. Image references are now correct and checked.

High

Enables complete successfully.

HAI-16380

Any manual change to a Token Factory object made the next restart fail. Restarts now succeed.

High

More robust operations.

HAI-16739

The model-engine selection setting was never applied by the installer. It is now applied.

High

The correct engine version is used for each GPU class.

HAI-15792

A streamed chat to an adapter that could not load hung for 600 seconds. It now returns a clear error immediately.

High

Faster feedback for API users.

HAI-15851

Models ignored the request to skip the thinking step, and thinking text was billed. The request is now honoured.

High

Correct behaviour and billing.

HAI-15865

Reasoning models asked for JSON returned an empty answer. JSON now arrives in the answer.

High

Structured output works on reasoning models.

HAI-16451

A healthy model engine could be wrongly declared dead and restarted mid-traffic. Health checks are now more reliable.

High

Fewer interruptions for live users.

HAI-16477

A successful fine-tune could be marked failed. Status now reflects the real result.

High

Accurate fine-tune results.

HAI-16513

Image-capable models such as Qwen 3.5 and Gemma 3/4 were refused at prepare. Their memory needs are now calculated correctly.

High

More models can be deployed.

HAI-16519

A deployment showed Ready when no engine was running. It now shows why it is not running.

High

Accurate status for customers.

HAI-16533

Deployments could leave GPU engines unclaimed after slow, cancelled or restarted deploys. GPU claims are now tracked throughout.

High

No phantom GPU usage, and accurate capacity.

HAI-15875

Deleting a model left about 14 GB of files on the server. Files are now removed and the space reported.

High

Frees disk space.

HAI-15897

The deploy wizard showed no GPU pools for new teams. Pools are now listed.

High

New teams can deploy immediately.

HAI-15868

The metrics page showed blank latency values. All latency metrics now display.

High

Customers can see model performance.

HAI-16040

The metrics page counted storage, fine-tune and RAG rows as model calls. Only real calls are counted now.

High

Accurate usage metrics.

HAI-16057

The model cost card now includes storage, fine-tuning and RAG charges.

High

Cost matches billing.

HAI-15870

Deleting a knowledge base left indexing jobs running. They are now stopped.

High

Frees CPU and memory.

HAI-15874

Indexing large PDFs was very slow (11+ minutes for 300 pages). Indexing is now faster.

High

Faster knowledge-base setup.

HAI-16044

One team could crash the shared RAG query service for every team. The cache now has a size limit.

High

Stable RAG service for all teams.

HAI-16046

Deleted RAG documents stayed on disk and could be served again. They are now fully deleted.

High

Deleted data is really gone.

HAI-16504

RAG retrieval defaulted to a less accurate mode. Hybrid retrieval is now the default.

High

Better answer quality (72% to 87% on the test set).

HAI-16748

A node lifecycle action could race with a live Token Factory enable. The lock is now re-checked.

High

Prevents conflicting operations.

HAI-16758

Editing a region with Token Factory off could re-run a failed teardown or block the edit. Teardown now runs only when needed.

High

Region edits work reliably.

HAI-16759

The Token Factory break-glass action always failed, and its audit records showed an unknown user. It now works and is audited.

High

Emergency recovery path is usable and traceable.

HAI-16764

A live Token Factory enable could run at the same time as a worker joining or leaving. These now exclude each other.

High

Prevents cluster conflicts.

Upgrade instructions

This is only an incremental upgrade and it is to be applied on 2.5.2

Panel upgrade

apt update apt install --only-upgrade -y hostedai-adminpanel-api apt install --only-upgrade -y hostedai-adminpanel-ui apt install --only-upgrade -y hostedai-userpanel-api apt install --only-upgrade -y hostedai-userpanel-ui

Recipe upgrade

Update recipes using deploy utility

./deploy --recipes

HaiDra upgrade

apt update apt install --only-upgrade -y haidra

HaiShare V2 upgrade

If there are regions running HAISHARE V2 update the debian package on all the nodes using

apt install --only-upgrade -y haishare

KVM Upgrade
** First the primary node is supposed to be upgraded; and then the secondaries **

apt update apt upgrade