Skip to main content
You are here: Releases Notes

2.4.2 Release Notes

Overview

Release v2.4.2 is a targeted patch release focused on stability, access-control correctness and GPU sharing reliability across the http://HOSTED.AI Platform. Key highlights include hardening of team-scoped resource isolation, multiple critical fixes in the HAIShare GPU sharing runtime and resolution of instance provisioning and lifecycle issues reported against v2.4.1.

Improvements to existing features

Ticket

Description

Area

Business Impact / Value to End Users

HAI-9434

Reduced excessive API polling in the User Panel UI that was triggering rate-limit (429) errors under normal usage patterns.

User Panel

Users no longer encounter unexpected 429 rate-limit errors during routine platform usage, resulting in a smoother, more reliable User Panel experience with fewer silent failures.

Bug fixes

Ticket

Description

Priority

Business Impact / Value to End Users

HAI-9458

Fixed a bug where users who were re-invited to a team could not see their teams after accepting the invitation.

Blocker

Re-invited team members immediately regain access to team resources upon accepting their invitation, eliminating support escalations and lost productivity caused by invisible team memberships.

HAI-9459

Fixed an incorrect role display where Team Admins were shown as Members in the team roster.

Critical

Team Admins are now correctly identified in the roster, giving teams accurate visibility into who holds administrative privileges and removing confusion in access-control workflows.

HAI-9265

Fixed a regression in the pool statistics time-filter that returned incorrect data for teams with no active instances in the selected time window.

High

Pool utilisation statistics now accurately reflect zero-activity periods, giving administrators reliable data for capacity planning and reporting — no more misleading figures for idle teams.

HAI-9836

Fixed a pagination bug in the Team Members API where the page query parameter was silently ignored, causing incomplete member listings when a team has more than one page of members.

Critical

Administrators managing large teams now see complete member listings, preventing audit blind-spots and ensuring access-control decisions are based on the full team roster.

2) GPU Instance Reliability

Ticket

Description

Priority

Business Impact / Value to End Users

HAI-9052

Resolved a deadlock that occurred when pod deprovisioning and volume deletion ran in parallel on the same node, which prevented GPUs from being released and left nodes in an undeletable state.

Critical

GPUs are now reliably freed when instances are terminated, preventing capacity lock-ups and ensuring GPU resources are promptly available for new workloads — directly improving cluster utilisation and reducing wait times.

HAI-9296

Fixed intermittent "No GPU available" errors shown during B200 instance launch even when GPUs were free and available.

Critical

Users can now launch B200 instances without encountering false availability errors, reducing failed provisioning attempts and improving confidence in GPU capacity reporting.

HAI-9302

Fixed an issue where newly launched instances appeared stuck in "Setting up" status with no IP address visible in the User Panel.

Normal

Instances now transition through setup visibly and surface their IP address as expected, allowing users to connect to their workloads immediately without needing to refresh or file a support request.

HAI-9435

Fixed a 404 error that occurred when closing the Instance Creation panel mid-flow.

Normal

Users can now dismiss the Instance Creation panel at any point in the flow without hitting an error, resulting in a smoother and less disruptive provisioning experience.

HAI-9571

Fixed a 502 Proxy Error returned when attempting to rename a running instance via the API.

High

Users and automation scripts can now reliably rename live instances via the API, enabling better workload organisation and labelling without requiring downtime or workarounds.

HAI-8927

Fixed a false validation error in the Admin Control Panel that incorrectly rejected GPUaaS node edits with "A node with this IP and port already exists" when editing an existing node.

Normal

Platform administrators can now update GPUaaS node configurations without hitting erroneous validation blocks, streamlining routine infrastructure maintenance tasks.

3) GPU Sharing (HAIShare)

Ticket

Description

Priority

Business Impact / Value to End Users

HAI-9284

Fixed a GPU sharing bypass where Docker containers launched inside GPU instances skipped the HAIShare interposer entirely, causing CUDA out-of-memory errors in co-tenant workloads running on shared GPU slices.

Critical

Docker-based workloads now correctly operate within their assigned GPU memory slice, preventing one tenant's containers from exhausting GPU memory shared by others and eliminating CUDA OOM errors caused by sharing violations.

HAI-9053

Fixed incorrect storage quota accounting where writes to NFS-mounted shared volumes were incorrectly charged against the per-instance ephemeral storage limit, causing premature storage exhaustion errors.

Critical

Users working with shared NFS volumes no longer hit unexpected storage limits mid-workload. Shared storage writes are now correctly accounted separately from instance-local storage, preventing spurious exhaustion errors on large dataset operations.

HAI-9656

Fixed a kernel-level storage quota bug that incorrectly triggered "no space left on device" errors during file copy operations inside GPU instances, blocking Docker installation and AppArmor configuration workflows.

Critical

Users can now install Docker and configure AppArmor inside GPU instances without hitting false "no space left on device" errors, unblocking a common containerised workload setup path that was failing on valid configurations.

HAI-9820

Fixed a self-recursion loop in the HAIShare runtime when nvidia-smi was invoked inside nested containers.

Normal

Running nvidia-smi inside nested containers no longer triggers a runtime loop, ensuring reliable GPU diagnostics and monitoring tools work correctly in containerised environments.

HAI-9866

Fixed an issue where a GPU instance could hang indefinitely on startup when a co-tenant held a shared GPU card's memory lock.

Blocker

GPU instances now start reliably regardless of co-tenant activity on the same card, eliminating indefinite startup hangs and ensuring predictable provisioning times in shared GPU environments.

4) Infrastructure & Networking

Ticket

Description

Priority

Business Impact / Value to End Users

HAI-9295

Fixed a non-atomic step in the baremetal provisioning flow that could leave SSH credentials out of sync between the platform database and the provisioned node, making the node credential unrecoverable without manual intervention.

Critical

Baremetal nodes are now provisioned atomically, ensuring SSH credentials remain consistent between the platform and the node at all times — eliminating the risk of unrecoverable nodes that previously required manual remediation.

HAI-9475

Fixed a false driver compatibility error when adding a GPU to a pool via the Admin Panel — the reconcile step was connecting to the wrong node during validation.

Normal

Platform administrators can now add GPUs to pools via the Admin Panel without encountering false driver compatibility failures, making pool expansion faster and less error-prone.

HAI-9760

Fixed an issue where deleting a Floating IP via the haiDra CLI did not remove the corresponding record from the admin database, leaving stale entries that could cause confusion on subsequent operations.

Critical

Floating IP deletions via the CLI now fully clean up all associated records, preventing stale database entries from causing confusion or blocking future IP allocation and management operations.

5) Marketplace & Shared Storage

Ticket

Description

Priority

Business Impact / Value to End Users

HAI-10012

Fixed marketplace regions incorrectly advertising shared storage support regardless of actual availability, and ensured team-level shared-storage resource policies are now correctly enforced at region selection.

High

Users now only see shared storage options for regions that genuinely support it, preventing failed deployments caused by selecting an unsupported region. Team-level storage policies are also correctly enforced, reducing misconfiguration risk.

MHA-810

Fixed a routing bug in the Marketplace where creating a shared volume always routed to the first available storage pool, ignoring the region selected by the user.

High

Shared volumes are now created in the region the user selects, ensuring data locality is respected and preventing volumes from silently landing in an unintended region.

6) Known Issues

Ticket

Description

Status

HAI-10020

The marketplace shared-storage capability indicator may not accurately reflect volume-group-specific availability in all regions. A dedicated capability flag is being introduced to address this in a subsequent patch.

In Progress

HAI-10060

HAIShare GPU isolation may not engage when a workload specifies runtime: nvidia directly in a docker-compose file. A fix is currently in review and will be included in the next patch.

In Review

2.4.2 Patch Upgrade Notes

Steps to update API/UI components

Run these on admin/user panel servers

wget -q https://assets.hosted.ai/deploy && bash deploy --upgrade

Steps to update recipes

Run following on panels to update recipes

./deploy --recipes --recipe-source stable

Sysbox patch

As part of this patch following two bugs related to sysbox are patched. Patching these will require restarting worker pods there by Downtime is required

https://hostedai.atlassian.net/browse/HAI-9053

https://hostedai.atlassian.net/browse/HAI-9656

Follow these steps to patch sysbox:

  1. If there are pods with rootfs disable take backup of file system using https://bitbucket.org/hostedai/build-and-installer-scripts/src/main/UpgradeScripts/reboot-deployment-worker.sh

  2. Update the hostedai-sysbox apt package on all worker nodes

  3. Restart the worker pods

  4. Run sysbox-persist-ctl recalc 2d26725d-7459-493b-986b-2732597bb82d/i-1fa44b36-f06e-448a-828a-65f81e6f84db (<namespace/worker-id>) on each node where the worker pods are present

Scheduler and device plugin upgrade

Bugs fixed as part of this patch:

https://hostedai.atlassian.net/browse/HAI-9866

https://hostedai.atlassian.net/browse/HAI-9820

https://hostedai.atlassian.net/browse/HAI-9284

https://hostedai.atlassian.net/browse/HAI-10060

Patching these will require restarting of scheduler, device plugin and worker pods

Get the upgrade scripts from https://bitbucket.org/hostedai/build-and-installer-scripts/src/main/SchedulerUpdateScripts/ These are generic scripts that could be used to re-render the device plugin, scheduler pods and configure the namespaces in scheduler pods.

All these steps are to be executed on master node

Step 0) Take backup

Take backup of /var/lib/hai folder

Step 1) Device-plugin:

This will re-render the device plugin, pulls the latest image and applies it

python3 update_device_plugin.py

Step 2) Scheduler:

This will re-render the scheduler, pulls the latest image and applies it

python3 update_scheduler_image.py

Step 3) Register namespaces:

This will register namespaces to respective scheduler pods by reading info from haidra

python3 register_pool_config.py

Step 4) Insert/Update haishare docker runtime for worker pods

This updates the pods yaml to include haishare docker runtime. Run the following dry-run command to see the number of pods that are going to be updated.

python3 update_haishare_runtime.py --hai-dir /var/lib/hai --dry-run

To modify the YAMLs run

python3 update_haishare_runtime.py --hai-dir /var/lib/hai

Once YAML files are updated, apply the change using kubectl apply -f <path-of-yaml> for each worker pod

Step 5) Restart all the worker pods

Restart all the worker pods that were not updated in previous steps (this will make sure existing works pickup new haishare runtime binaries). To restart just delete the worker pod, as these pods are running as deployments they will get recreated.