Overview
Release v2.4.2 is a targeted patch release focused on stability, access-control correctness and GPU sharing reliability across the http://HOSTED.AI Platform. Key highlights include hardening of team-scoped resource isolation, multiple critical fixes in the HAIShare GPU sharing runtime and resolution of instance provisioning and lifecycle issues reported against v2.4.1.
Improvements to existing features
Ticket | Description | Area | Business Impact / Value to End Users |
|---|---|---|---|
HAI-9434 | Reduced excessive API polling in the User Panel UI that was triggering rate-limit (429) errors under normal usage patterns. | User Panel | Users no longer encounter unexpected 429 rate-limit errors during routine platform usage, resulting in a smoother, more reliable User Panel experience with fewer silent failures. |
Bug fixes
Ticket | Description | Priority | Business Impact / Value to End Users |
|---|---|---|---|
HAI-9458 | Fixed a bug where users who were re-invited to a team could not see their teams after accepting the invitation. | Blocker | Re-invited team members immediately regain access to team resources upon accepting their invitation, eliminating support escalations and lost productivity caused by invisible team memberships. |
HAI-9459 | Fixed an incorrect role display where Team Admins were shown as Members in the team roster. | Critical | Team Admins are now correctly identified in the roster, giving teams accurate visibility into who holds administrative privileges and removing confusion in access-control workflows. |
HAI-9265 | Fixed a regression in the pool statistics time-filter that returned incorrect data for teams with no active instances in the selected time window. | High | Pool utilisation statistics now accurately reflect zero-activity periods, giving administrators reliable data for capacity planning and reporting — no more misleading figures for idle teams. |
HAI-9836 | Fixed a pagination bug in the Team Members API where the page query parameter was silently ignored, causing incomplete member listings when a team has more than one page of members. | Critical | Administrators managing large teams now see complete member listings, preventing audit blind-spots and ensuring access-control decisions are based on the full team roster. |
2) GPU Instance Reliability
Ticket | Description | Priority | Business Impact / Value to End Users |
|---|---|---|---|
HAI-9052 | Resolved a deadlock that occurred when pod deprovisioning and volume deletion ran in parallel on the same node, which prevented GPUs from being released and left nodes in an undeletable state. | Critical | GPUs are now reliably freed when instances are terminated, preventing capacity lock-ups and ensuring GPU resources are promptly available for new workloads — directly improving cluster utilisation and reducing wait times. |
HAI-9296 | Fixed intermittent "No GPU available" errors shown during B200 instance launch even when GPUs were free and available. | Critical | Users can now launch B200 instances without encountering false availability errors, reducing failed provisioning attempts and improving confidence in GPU capacity reporting. |
HAI-9302 | Fixed an issue where newly launched instances appeared stuck in "Setting up" status with no IP address visible in the User Panel. | Normal | Instances now transition through setup visibly and surface their IP address as expected, allowing users to connect to their workloads immediately without needing to refresh or file a support request. |
HAI-9435 | Fixed a 404 error that occurred when closing the Instance Creation panel mid-flow. | Normal | Users can now dismiss the Instance Creation panel at any point in the flow without hitting an error, resulting in a smoother and less disruptive provisioning experience. |
HAI-9571 | Fixed a 502 Proxy Error returned when attempting to rename a running instance via the API. | High | Users and automation scripts can now reliably rename live instances via the API, enabling better workload organisation and labelling without requiring downtime or workarounds. |
HAI-8927 | Fixed a false validation error in the Admin Control Panel that incorrectly rejected GPUaaS node edits with "A node with this IP and port already exists" when editing an existing node. | Normal | Platform administrators can now update GPUaaS node configurations without hitting erroneous validation blocks, streamlining routine infrastructure maintenance tasks. |
3) GPU Sharing (HAIShare)
Ticket | Description | Priority | Business Impact / Value to End Users |
|---|---|---|---|
HAI-9284 | Fixed a GPU sharing bypass where Docker containers launched inside GPU instances skipped the HAIShare interposer entirely, causing CUDA out-of-memory errors in co-tenant workloads running on shared GPU slices. | Critical | Docker-based workloads now correctly operate within their assigned GPU memory slice, preventing one tenant's containers from exhausting GPU memory shared by others and eliminating CUDA OOM errors caused by sharing violations. |
HAI-9053 | Fixed incorrect storage quota accounting where writes to NFS-mounted shared volumes were incorrectly charged against the per-instance ephemeral storage limit, causing premature storage exhaustion errors. | Critical | Users working with shared NFS volumes no longer hit unexpected storage limits mid-workload. Shared storage writes are now correctly accounted separately from instance-local storage, preventing spurious exhaustion errors on large dataset operations. |
HAI-9656 | Fixed a kernel-level storage quota bug that incorrectly triggered "no space left on device" errors during file copy operations inside GPU instances, blocking Docker installation and AppArmor configuration workflows. | Critical | Users can now install Docker and configure AppArmor inside GPU instances without hitting false "no space left on device" errors, unblocking a common containerised workload setup path that was failing on valid configurations. |
HAI-9820 | Fixed a self-recursion loop in the HAIShare runtime when nvidia-smi was invoked inside nested containers. | Normal | Running nvidia-smi inside nested containers no longer triggers a runtime loop, ensuring reliable GPU diagnostics and monitoring tools work correctly in containerised environments. |
HAI-9866 | Fixed an issue where a GPU instance could hang indefinitely on startup when a co-tenant held a shared GPU card's memory lock. | Blocker | GPU instances now start reliably regardless of co-tenant activity on the same card, eliminating indefinite startup hangs and ensuring predictable provisioning times in shared GPU environments. |
4) Infrastructure & Networking
Ticket | Description | Priority | Business Impact / Value to End Users |
|---|---|---|---|
HAI-9295 | Fixed a non-atomic step in the baremetal provisioning flow that could leave SSH credentials out of sync between the platform database and the provisioned node, making the node credential unrecoverable without manual intervention. | Critical | Baremetal nodes are now provisioned atomically, ensuring SSH credentials remain consistent between the platform and the node at all times — eliminating the risk of unrecoverable nodes that previously required manual remediation. |
HAI-9475 | Fixed a false driver compatibility error when adding a GPU to a pool via the Admin Panel — the reconcile step was connecting to the wrong node during validation. | Normal | Platform administrators can now add GPUs to pools via the Admin Panel without encountering false driver compatibility failures, making pool expansion faster and less error-prone. |
HAI-9760 | Fixed an issue where deleting a Floating IP via the haiDra CLI did not remove the corresponding record from the admin database, leaving stale entries that could cause confusion on subsequent operations. | Critical | Floating IP deletions via the CLI now fully clean up all associated records, preventing stale database entries from causing confusion or blocking future IP allocation and management operations. |
5) Marketplace & Shared Storage
Ticket | Description | Priority | Business Impact / Value to End Users |
|---|---|---|---|
HAI-10012 | Fixed marketplace regions incorrectly advertising shared storage support regardless of actual availability, and ensured team-level shared-storage resource policies are now correctly enforced at region selection. | High | Users now only see shared storage options for regions that genuinely support it, preventing failed deployments caused by selecting an unsupported region. Team-level storage policies are also correctly enforced, reducing misconfiguration risk. |
MHA-810 | Fixed a routing bug in the Marketplace where creating a shared volume always routed to the first available storage pool, ignoring the region selected by the user. | High | Shared volumes are now created in the region the user selects, ensuring data locality is respected and preventing volumes from silently landing in an unintended region. |
6) Known Issues
Ticket | Description | Status |
|---|---|---|
HAI-10020 | The marketplace shared-storage capability indicator may not accurately reflect volume-group-specific availability in all regions. A dedicated capability flag is being introduced to address this in a subsequent patch. | In Progress |
HAI-10060 | HAIShare GPU isolation may not engage when a workload specifies runtime: nvidia directly in a docker-compose file. A fix is currently in review and will be included in the next patch. | In Review |
2.4.2 Patch Upgrade Notes
Steps to update API/UI components
Run these on admin/user panel servers
wget -q https://assets.hosted.ai/deploy && bash deploy --upgradeSteps to update recipes
Run following on panels to update recipes
./deploy --recipes --recipe-source stableSysbox patch
As part of this patch following two bugs related to sysbox are patched. Patching these will require restarting worker pods there by Downtime is required
https://hostedai.atlassian.net/browse/HAI-9053
https://hostedai.atlassian.net/browse/HAI-9656
Follow these steps to patch sysbox:
If there are pods with rootfs disable take backup of file system using https://bitbucket.org/hostedai/build-and-installer-scripts/src/main/UpgradeScripts/reboot-deployment-worker.sh
Update the hostedai-sysbox apt package on all worker nodes
Restart the worker pods
Run sysbox-persist-ctl recalc 2d26725d-7459-493b-986b-2732597bb82d/i-1fa44b36-f06e-448a-828a-65f81e6f84db (<namespace/worker-id>) on each node where the worker pods are present
Scheduler and device plugin upgrade
Bugs fixed as part of this patch:
https://hostedai.atlassian.net/browse/HAI-9866
https://hostedai.atlassian.net/browse/HAI-9820
https://hostedai.atlassian.net/browse/HAI-9284
https://hostedai.atlassian.net/browse/HAI-10060
Patching these will require restarting of scheduler, device plugin and worker pods
Get the upgrade scripts from https://bitbucket.org/hostedai/build-and-installer-scripts/src/main/SchedulerUpdateScripts/ These are generic scripts that could be used to re-render the device plugin, scheduler pods and configure the namespaces in scheduler pods.
All these steps are to be executed on master node
Step 0) Take backup
Take backup of /var/lib/hai folder
Step 1) Device-plugin:
This will re-render the device plugin, pulls the latest image and applies it
python3 update_device_plugin.pyStep 2) Scheduler:
This will re-render the scheduler, pulls the latest image and applies it
python3 update_scheduler_image.pyStep 3) Register namespaces:
This will register namespaces to respective scheduler pods by reading info from haidra
python3 register_pool_config.pyStep 4) Insert/Update haishare docker runtime for worker pods
This updates the pods yaml to include haishare docker runtime. Run the following dry-run command to see the number of pods that are going to be updated.
python3 update_haishare_runtime.py --hai-dir /var/lib/hai --dry-runTo modify the YAMLs run
python3 update_haishare_runtime.py --hai-dir /var/lib/haiOnce YAML files are updated, apply the change using kubectl apply -f <path-of-yaml> for each worker pod
Step 5) Restart all the worker pods
Restart all the worker pods that were not updated in previous steps (this will make sure existing works pickup new haishare runtime binaries). To restart just delete the worker pod, as these pods are running as deployments they will get recreated.