Summary
Overview - Release V2.5.3 - is a patch release for the HOSTED.AI Platform with significant resolved issues, including 7 Blocker, 10 Critical and 68 High priority items. It focuses on platform security hardening, GPU-sharing (HAIShare) reliability, Token Factory quality and Admin Panel, Cluster Panel and KVM stability.
Key highlights
Security hardening: KVM nodes now ship with a host firewall, and guest VMs can no longer reach host services. VM consoles are password-protected and no longer listen on every interface. SSH password login is disabled. Default cluster panel passwords are replaced with random per-cluster passwords and rotated on upgrade. System recipes cannot be modified through the API, and the recipe runner is locked down to prevent privilege escalation. Credentials are no longer written to the admin log, playbook logs or tenant nodes. Go toolchain and library vulnerability advisories are patched.
GPU sharing (HAIShare): More reliable multi-GPU (NCCL) and tensor-parallel workloads. Tenant migration is safer. The scheduler no longer livelocks a card shared by two non-evictable tenants or lets an idle tenant hold memory indefinitely. Several tenant crashes are fixed, including a customer-reported eviction loop on RTX PRO 6000 dynamic pools and a crash caused by an in-place library update.
Token Factory: OpenAI-compatible API behaviour is corrected (n, seed, top_k, response_format on reasoning models, thinking mode and context-length errors). RAG indexing is faster and validates file types, and deleting a document or knowledge base now cleans up fully. Fine-tune and deploy lifecycles are more robust. Usage, billing and metrics figures are accurate, team isolation and error redaction are stronger, and enabling and tearing down Token Factory on a live region is more reliable.
Admin Panel, Cluster Panel and KVM: Nodes can be added on a management IP on a VLAN sub-interface (customer-reported). Joining a secondary node to a KVM cluster and HA on private management networks now work. Reducing redundancy on a running VM no longer destroys it, Configure Racks works over HTTP, and a wrong 2FA code shows an error instead of returning to login.
User Panel and instances: Running pods no longer flip to Pending in the instance list. vGPU counts and shared-pool capacity are reported correctly. A new instance is no longer deleted shortly after launch, and re-uploading a recipe template no longer breaks the services that use it. The deploy wizard respects the team's resource policy again.
Provisioning: Enable GPUaaS no longer deletes snapshot storage on a live region, and GPU node setup and reboots are faster.
Improvements to existing features
1. GPU Sharing (HAIShare)
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-15479 | HAIShare V2 CDI now supports multiple glibc versions, so containers built on older or newer base images run under GPU sharing. | High | Customers can run a wider range of container images on HAIShare V2 pools without compatibility failures. |
HAI-15562 | HAIShare V1 CDI now supports multiple glibc versions. | High | The same image compatibility for existing HAIShare V1 pools. |
HAI-16573 | Temporal sharing switches tenants faster (delta freeze and region hand-off), and credit-based preemption is fixed so a tenant cannot keep the card at others' expense. | Normal | Less idle GPU time when tenants take turns on a shared card, giving each tenant more usable GPU time and fairer turns. |
HAI-16577 | Memswap disk spill now moves evicted memory chunk by chunk and keeps as much as possible off disk. | Normal | Much faster tenant switches for pods that are short of host RAM. |
2. GPU node setup and provisioning
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-15591 | Enable GPUaaS no longer installs Prometheus, GPU Operator and Envoy Gateway one after another. | Normal | Regions are enabled faster. |
HAI-15588 | GPU node reboots no longer wait about 2 minutes for a network port that has no IPv4 address. | Normal | Faster node reboots and maintenance windows. |
HAI-15587 | The NVIDIA driver kernel module is compiled only for the running kernel instead of every installed kernel. | Normal | Faster driver installs during node setup. |
HAI-16279 | Capacity can be checked for many services in one call, and a "full" answer now matches exactly when instance creation would be refused. | Normal | Partners and the UI can show accurate availability before deploying, avoiding failed instance creation. |
3. Token Factory
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-15900, HAI-15903 | Prepare lists HOSTED.AI-certified models first, with a badge and auto-filled settings. The Prepare dialog keeps model, placement and GPUs per replica visible, auto-selects single options, and moves engine settings into a closed Advanced section with sensible defaults. | Normal | Customers prepare models faster and with fewer configuration mistakes. |
HAI-16481, HAI-16549 | Prepare fails early with a clear reason instead of late and opaquely. Error messages are clearer, a lack of capacity is no longer reported as a server error, container images are pinned and RAG query gets health probes. | Normal | Fewer confusing failures, and problems are easier to diagnose and act on. |
HAI-15889 | RAG parsing keeps tables, scanned pages and layout, so grounded answers can cite table cells. | Normal | More accurate answers from real-world documents such as reports and scanned PDFs. |
HAI-16017 | Indexing shows every step live (queued, parse, layout, tables, OCR, chunk, embed, write) with counts, speed, elapsed time and ETA. | Normal | Customers can see exactly how far indexing has progressed instead of waiting on an opaque "processing" state. |
HAI-15908, HAI-16100, HAI-16407 | Citations include the quoted passage and page number, the original document can be downloaded and opened at the cited page, and grounded answers carry per-citation retrieval signals and timings. | Normal | Users can verify answers against the source quickly, which builds trust in grounded chat. |
HAI-15909 | Metrics add window-level TTFT, TPOT and end-to-end latency percentiles (p50/p95/p99). | Low | Accurate latency reporting for performance and SLA tracking. |
HAI-16036 | The Token Factory admin team can see customer revenue against the GPU cost of serving it, per model. | Normal | Operators can judge the profitability of each model and price it accordingly. |
HAI-15786, HAI-15603 | API keys verified once are cached instead of re-checked on every call (about 54 ms each), and the serving token is no longer minted inline on a customer's call. | Normal | Lower latency on every Token Factory inference request. |
HAI-16228 | The Token Factory load test runs on the server, next to the model. | Normal | Load-test results reflect true model performance rather than network overhead. |
HAI-16482, HAI-16517, HAI-16741 | Token Factory node overhead is reduced: fewer GPU status polls, the cache librarian is off by default, versions are reported, and RAG query resources are right-sized. | Low | More node CPU and memory are left for customer workloads. |
HAI-16592 | "Tested" and "certified" labels now claim only what a completed test actually proves. | Normal | Customers can rely on certification labels when choosing a model. |
Bug fixes
1. Security – Admin Panel, recipes and credentials
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-17141 | Recipe execution is locked down, and system recipes can no longer be cloned and modified to run arbitrary commands. | Blocker | Prevents privilege escalation through the recipe feature. |
HAI-17142 | System recipe templates are now read-only through the API. | Blocker | Built-in automation cannot be tampered with. |
HAI-17144 | Local copies of system recipes are now created automatically at startup. | Blocker | Platform automation keeps working after the recipe lockdown. |
HAI-17175 | The recipe feature in the Admin Panel is restricted: inventories and recipe runs can no longer be created through the API, and recipe edits are checked more strictly. | Critical | Removes a route to root access and to private-key leaks. |
HAI-16065 | Improved access controls for Ansible logs and enhanced sensitive information masking to prevent unintended exposure of confidential data. | Critical | Reduces the risk of sensitive configuration details being accessible to lower-privileged Admin Panel users.. |
HAI-17224 | Enhanced security measures to prevent the exposure of sensitive authentication information through application logs. | High | Enhances the protection of sensitive authentication information within application logs |
HAI-16765 | The User Panel was built with an older Go version that contained known security advisories. It now uses the patched version. | High | Removes known standard-library vulnerabilities from the shipped package. |
2. KVM and Cluster Panel
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-16231 | Reducing redundancy on a running VM could delete its live disk copies and destroy the VM. The operation is now safe. | Blocker | Prevents VM data loss during storage changes. |
HAI-16650 | When a User Panel synced several KVM clusters, images never appeared, so no VMs could be created. Images now sync correctly. | Blocker | Customers can deploy VMs in every KVM region. |
HAI-16890 | The Create new rack button did nothing when the Cluster Panel was opened over HTTP. It now works. | Critical | Operators can complete cluster setup. |
HAI-16722 | Joining a secondary node to a KVM cluster failed with a generic error. Duplicate nodes are now caught when added, and join works. | High | Faster, clearer cluster expansion. |
HAI-17234 | On private management networks, enabling HA gave peers an unreachable address. The correct management address is now used. | High | HA can be enabled on private networks. |
3. Admin Panel, User Panel and GPUaaS
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-16181 | Adding a node whose management IP is on a VLAN sub-interface failed. VLAN interfaces are now detected. | Blocker | Customers with VLAN or bonded networks can onboard nodes. |
HAI-16889 | NVIDIA L4 GPU nodes failed to initialise because the pinned driver 550 could not build on newer kernels. L4 nodes now use a compatible driver. | Blocker | L4 GPU nodes can be added and come online without manual work. |
HAI-16109 | Entering a wrong 2FA code sent the user back to the login page. An error now appears on the 2FA page. | High | Smoother, clearer sign-in. |
HAI-16697 | Re-uploading a recipe template left existing services pointing at a deleted recipe. Services now follow the new template. | High | Services keep working after recipe updates. |
HAI-16643 | Pool capacity reported room for only one pod on every shared pool. It now shows the real free capacity. | High | Customers see accurate capacity and can place more workloads. |
HAI-16749 | Running pods briefly showed as Pending every 5 minutes. Status is now stable. | High | Accurate status and fewer false alarms. |
HAI-17147 | The deploy wizard offered regions outside the team's resource policy. Only permitted regions are shown now. | High | Teams stay within their allowed regions. |
HAI-17209 | Subscribing a team to a HAIShare V2 pool always failed with "Insufficient resources". Subscriptions now work. | High | Teams can use HAIShare V2 pools. |
HAI-16763 | Removing a pod could delete the team namespace while it still held a subscription, breaking the team's next pod. This is now prevented. | High | New pods no longer fail after a teardown. |
4. GPU sharing (HAIShare)
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-16319 | A GPU shared by two tenants that could not be evicted could stall for hours with nothing serving. The scheduler now recovers. | Blocker | Prevents silent outages on shared GPUs. |
HAI-16079 | A ComfyUI tenant crash-looped during idle memory eviction. Eviction now handles this memory correctly. | Critical | Stable image-generation workloads on shared GPUs. |
HAI-16080 | Updating the HAIShare library in place crashed running workloads. Updates now replace the file safely. | Critical | Upgrades no longer crash running customer pods. |
HAI-16190 | A node could hang when the device plugin started, crashing all shared tenants. GPU operations on evicted memory are now properly gated. | High | Prevents node-wide outages. |
HAI-16718 | An idle co-tenant could force a running vLLM workload into host memory every ~12 seconds. Sharing mode now selects correctly. | High | Customers get full GPU performance. |
HAI-15745 | Paused tenants could exhaust host RAM, killing other tenants' workloads. Host memory use is now limited. | High | Prevents cascading workload failures. |
HAI-15766 | A failed freeze during migration reported success but left memory behind. Migration now checks and rolls back. | High | Safe workload migration. |
HAI-16104 | Migration to a GPU without free memory left the workload frozen with no error. It now rolls back and reports the failure. | High | Workloads are not left stuck. |
HAI-15850 | If a pod's serving process died, its GPU calls hung with no error. They now fail immediately. | High | Faster failure detection and recovery. |
HAI-16027 | Two multi-GPU (tensor-parallel) jobs sharing GPUs could deadlock. Freezes are now coordinated across all GPUs of a job. | High | Reliable multi-GPU workloads. |
HAI-16744 | A pod's own multi-GPU job was not recognised as multi-GPU, so one GPU of the job could be moved alone. Such jobs are now detected. | High | Prevents multi-GPU job failures. |
HAI-16194 | A tenant stuck after a partial restore kept its GPU memory forever and starved others. Memory is now released. | High | Fair GPU sharing. |
HAI-16039 | Programs using cuDNN's CUDA-graph API crashed the tenant's serving process. These calls are now supported. | High | Wider framework compatibility. |
HAI-16694 | Multi-GPU communication (NCCL) in shared pods ran over TCP at 1–5% of native speed. Direct GPU-to-GPU transfer is now enabled. | High | Up to ~100x faster multi-GPU training and inference. |
HAI-16733 | Multi-GPU initialisation failed for some pods because of a GPU address clash. Addresses are now unique. | High | Multi-GPU pods start reliably. |
HAI-14009 | PyTorch's expandable memory segments setting was stripped. It is now passed through. | High | Better memory efficiency for PyTorch and vLLM. |
HAI-17241 | A new tenant could inherit the previous tenant's connection limit, so its server failed to start. Limits are now reset. | High | Workloads start reliably on reused slots. |
5. Token Factory
Ticket | Description | Priority | Business impact / value to end users |
|---|---|---|---|
HAI-16530 | Fine-tuned adapters stayed linked to a deleted base model, so chats failed while the UI showed Ready. Adapters now follow the current base model. | Critical | Fine-tuned models keep working. |
HAI-15494 | Token Factory can be enabled on a live region from the controller node without a reset. Access for finished pods is also removed. | High | Enable Token Factory with no downtime. |
HAI-12039 | Turning Token Factory off for a region did not remove its components. They are now torn down automatically. | High | Clean region disable with no leftover endpoints. |
HAI-15617 | Enable GPUaaS ran steps meant to be skipped, which could delete snapshot storage on a live region. These steps no longer run. | High | Prevents data loss on live regions. |
HAI-15866 | Fresh installs did not store the Token Factory admin token, so the first model prepare failed. The token is now stored. | High | Token Factory works right after install. |
HAI-16052 | Token Factory could be installed on HAIShare V2 regions, where it does not work. This is now refused. | High | Prevents broken installs. |
HAI-16053 | Live enable was refused on panels with a default install. It now works with the default configuration. | High | Live enable works out of the box. |
HAI-16061 | Enable referenced a model image that did not exist. Image references are now correct and checked. | High | Enables complete successfully. |
HAI-16380 | Any manual change to a Token Factory object made the next restart fail. Restarts now succeed. | High | More robust operations. |
HAI-16739 | The model-engine selection setting was never applied by the installer. It is now applied. | High | The correct engine version is used for each GPU class. |
HAI-15792 | A streamed chat to an adapter that could not load hung for 600 seconds. It now returns a clear error immediately. | High | Faster feedback for API users. |
HAI-15851 | Models ignored the request to skip the thinking step, and thinking text was billed. The request is now honoured. | High | Correct behaviour and billing. |
HAI-15865 | Reasoning models asked for JSON returned an empty answer. JSON now arrives in the answer. | High | Structured output works on reasoning models. |
HAI-16451 | A healthy model engine could be wrongly declared dead and restarted mid-traffic. Health checks are now more reliable. | High | Fewer interruptions for live users. |
HAI-16477 | A successful fine-tune could be marked failed. Status now reflects the real result. | High | Accurate fine-tune results. |
HAI-16513 | Image-capable models such as Qwen 3.5 and Gemma 3/4 were refused at prepare. Their memory needs are now calculated correctly. | High | More models can be deployed. |
HAI-16519 | A deployment showed Ready when no engine was running. It now shows why it is not running. | High | Accurate status for customers. |
HAI-16533 | Deployments could leave GPU engines unclaimed after slow, cancelled or restarted deploys. GPU claims are now tracked throughout. | High | No phantom GPU usage, and accurate capacity. |
HAI-15875 | Deleting a model left about 14 GB of files on the server. Files are now removed and the space reported. | High | Frees disk space. |
HAI-15897 | The deploy wizard showed no GPU pools for new teams. Pools are now listed. | High | New teams can deploy immediately. |
HAI-15868 | The metrics page showed blank latency values. All latency metrics now display. | High | Customers can see model performance. |
HAI-16040 | The metrics page counted storage, fine-tune and RAG rows as model calls. Only real calls are counted now. | High | Accurate usage metrics. |
HAI-16057 | The model cost card now includes storage, fine-tuning and RAG charges. | High | Cost matches billing. |
HAI-15870 | Deleting a knowledge base left indexing jobs running. They are now stopped. | High | Frees CPU and memory. |
HAI-15874 | Indexing large PDFs was very slow (11+ minutes for 300 pages). Indexing is now faster. | High | Faster knowledge-base setup. |
HAI-16044 | One team could crash the shared RAG query service for every team. The cache now has a size limit. | High | Stable RAG service for all teams. |
HAI-16046 | Deleted RAG documents stayed on disk and could be served again. They are now fully deleted. | High | Deleted data is really gone. |
HAI-16504 | RAG retrieval defaulted to a less accurate mode. Hybrid retrieval is now the default. | High | Better answer quality (72% to 87% on the test set). |
HAI-16748 | A node lifecycle action could race with a live Token Factory enable. The lock is now re-checked. | High | Prevents conflicting operations. |
HAI-16758 | Editing a region with Token Factory off could re-run a failed teardown or block the edit. Teardown now runs only when needed. | High | Region edits work reliably. |
HAI-16759 | The Token Factory break-glass action always failed, and its audit records showed an unknown user. It now works and is audited. | High | Emergency recovery path is usable and traceable. |
HAI-16764 | A live Token Factory enable could run at the same time as a worker joining or leaving. These now exclude each other. | High | Prevents cluster conflicts. |
Upgrade instructions
This is only an incremental upgrade and it is to be applied on 2.5.2
Panel upgrade
apt update apt install --only-upgrade -y hostedai-adminpanel-api apt install --only-upgrade -y hostedai-adminpanel-ui apt install --only-upgrade -y hostedai-userpanel-api apt install --only-upgrade -y hostedai-userpanel-ui
Recipe upgrade
Update recipes using deploy utility
./deploy --recipes
HaiDra upgrade
apt update apt install --only-upgrade -y haidra
HaiShare V2 upgrade
If there are regions running HAISHARE V2 update the debian package on all the nodes using
apt install --only-upgrade -y haishare
KVM Upgrade
** First the primary node is supposed to be upgraded; and then the secondaries **
apt update apt upgrade