2.4 Release Notes

Baremetal GPU Instance Provisioning — (HAI-4444)

Introduces baremetal servers as a new instance type alongside VMs and GPUaaS pods. Service providers can onboard bare-metal GPU nodes, manage the full node lifecycle (offline → online → in service → not in service), and assign servers to users via auto-assign or manual allocation from the admin panel. Includes unified billing, SSH console access, and Ansible-based service provisioning.

Enables service providers to offer all GPU resource types — oversubscribed pods, VM GPU passthrough, and bare-metal — through a single portal with consistent billing and access credentials across regions. Expands the addressable market to users requiring full physical server access.

High Availability for KVM Cluster Panel Nodes — (HAI-6347)

Implements automatic primary/secondary failover for HAI KVM cluster panel nodes using etcd for distributed leader election. A periodic HA agent aligns controller services and SQL Database replication to the elected leader without manual intervention. VMs can be individually set to auto-restart or not depending on whether the VM should restart automatically on failure. HA is enabled via a cluster UI toggle, enforces a 3-node minimum, and supports full disable/revert.

Eliminates single points of failure in KVM cluster management, reducing unplanned downtime for VM workloads and providing automated recovery without operator intervention on primary node failure — improving SLA reliability for hosted VM infrastructure.

Dynamic GPU Allocation & VIP Priority Scheduling — (HAI-4627)

Introduces two new HaiShare scheduling modes per GPUaaS pool: Dynamic (end users select a minimum GPU percentage guarantee via slider during subscription) and VIP Priority (designated users receive a higher TQ_slice, ensuring consistently higher GPU cycles and credit accrual). Admins configure mode per pool; pricing policies support tiered pricing per TQ_slice increment.

Gives service providers the flexibility to differentiate GPU service tiers and end users the ability to select a guaranteed share of GPU resources — improving workload predictability, enabling premium offerings, and supporting use cases requiring sustained GPU throughput.

RootFS Persistence for GPUaaS Instances — (HAI-4679)

Makes the root filesystem of GPUaaS pods persistent across reboots using a host-path storage plugin — no separate remote volume, no periodic data sync, and zero performance overhead. Enabled by default on new instances. Includes first time Ansible service execution, not across instance reboot, factory reset with data wipe, Sysbox-enforced storage quotas (ENOSPC at 96% usage), and df/reboot wrappers for fully VM-like behaviour inside pods.

Eliminates the "reboot problem" where users lose installed packages, drivers, and configurations on every pod restart — significantly reducing support ticket volumes and developer friction for GPU workload users, and aligning GPUaaS pod behaviour with user expectations from VM environments.

Prometheus Metrics Framework for KVM — (HAI-4934)

Replaces the legacy RRD file-based stats system with a Prometheus + libvirt exporter architecture. Provides per-VM metrics (vCPU, memory, disk I/O, network I/O) and cluster-wide node exporter metrics with real-time dashboards, AlertManager webhook integration, and batch 10-minute collection. Prometheus config is auto-regenerated with 10-second target sync as nodes are added or removed.

Delivers a scalable, standards-based observability foundation for KVM infrastructure — eliminating filesystem scan overhead, enabling reliable alerting (node down, disk thresholds), and providing historical metric storage across all cluster nodes to support operational visibility at scale.

Issue Key

Issue Summary

Business Impact/Value to End users

HAI-4399

[BE] Unable to Rescan Resources in Admin CP

Fixed an issue preventing administrators from rescanning resources in the Admin Control Panel. Administrators can now reliably trigger resource rescans to keep infrastructure inventory accurate and up to date.

HAI-6044

[k8s]RDMA ib_write_bw fails with ADDR_ERROR between pods over InfiniBand (RoCE)

Resolved an RDMA connectivity failure between pods communicating over InfiniBand (RoCE) that caused ADDR_ERROR during bandwidth tests. High-performance workloads using InfiniBand networking now communicate reliably between pods.

HAI-6328

[BE] Usage cost is absent after upgrade from Oct release to Titan

Fixed a billing data loss issue where usage costs were not displayed after upgrading from v2.1 to v2.2. Users can now view accurate historical billing and usage cost data following platform upgrades.

HAI-6608

[FE] In admin panel, For baremetal node, if test connection is passed, Rescan resources button is disabled

Fixed an issue where the Rescan Resources button remained disabled after a successful test connection on a baremetal node. Administrators can now immediately rescan resources after verifying node connectivity.

HAI-4675

[ansible] Failed to start a POD. NVIDIA GPU injection failed(NVIDIA Tesla V100-SXM2-32GB).

Resolved an Ansible playbook failure that prevented GPU injection for NVIDIA Tesla V100-SXM2-32GB cards, blocking pod startup. GPUaaS instances using V100 hardware now provision and start successfully.

HAI-5414

SPATIAL mode requires 2 attempts to enable

Fixed a race condition requiring users to attempt enabling SPATIAL mode twice before it activated. SPATIAL mode now enables reliably on the first attempt, improving the GPU partitioning workflow.

HAI-6598

Main/Europa - Add Pool - Storage service asks for VG UUID instead of VG Name

Fixed a usability issue in the Add Pool flow where the storage service incorrectly requested a VG UUID instead of the human-readable VG Name. Administrators can now add storage pools using the familiar VG Name, reducing configuration errors.

HAI-6599

Restarting adminpanel-api.service silently kills running Docker containers executing recipes

Fixed a critical issue where restarting the admin panel API service silently terminated Docker containers running active recipes. Recipe executions are now preserved across admin service restarts, preventing unexpected job failures.

HAI-6693

[UI] Service Exposure records are duplicated on each service edit/save

Fixed a data duplication bug where saving or editing a service caused its port exposure records to be duplicated on each save. Service configurations now save correctly without creating duplicate exposure entries.

HAI-4784

GPUaaS Initialisation doesn't take care of all dependancies needed on minimal Ubuntu 24.04 install

Fixed GPUaaS initialisation to correctly install all required dependencies on minimal Ubuntu 24.04 installations. Fresh Ubuntu 24.04 minimal deployments can now run GPUaaS without requiring manual dependency resolution.

HAI-4836

Removal of the node from admin panel does'nt succeed.

Fixed a failure that prevented administrators from removing nodes via the Admin Control Panel. Node removal now completes successfully, enabling clean infrastructure decommissioning without manual intervention.

HAI-4862

Infiniband region, accelerator card and GPUaaS pool selections need to be logically linked during service creation

Fixed the service creation wizard to correctly link InfiniBand region, accelerator card, and GPUaaS pool selections, preventing invalid configurations. Users can now create InfiniBand-enabled services with correctly correlated resource selections.

HAI-6516

Ariel - Team level VRAM/TFLOP consumption metrics is 0 now

Fixed team-level VRAM and TFLOP consumption metrics showing as zero. Accurate GPU resource utilisation metrics are now displayed per team, enabling better capacity planning and chargeback reporting.

HAI-6649

[API] Webconsole connect fails because of missing resource clean up

Fixed a webconsole connection failure caused by stale resources not being cleaned up between sessions. Users can now reliably connect to the web console without encountering residual connection failures.

HAI-6685

Fix: Historical baremetal billing shows $0 after node deletion and re-subscription

Fixed a billing calculation bug where historical baremetal costs showed $0 after a node was deleted and re-subscribed. Billing history is now accurately maintained across node lifecycle changes.

HAI-4803

while a pod is being provisioned and the user-api service is restarted, the IP does not get recorded in the DB

Fixed a race condition where a pod's IP address was not recorded in the database if the user API service restarted during provisioning. Pod network configuration is now reliably persisted regardless of service interruptions.

HAI-5439

[BE] User Panel API: 7 discrepancies between swagger.yaml and actual routes in hostedai-user

Fixed 7 API route discrepancies between the Swagger documentation and actual User Panel API endpoints. API documentation now accurately reflects all available routes, improving the developer integration experience.

HAI-6281

Shared storage fee displays a negative value when inclusive amount exceeds actual usage

Fixed a billing calculation error where shared storage fees displayed as negative values when usage fell within the inclusive allowance. Storage billing now correctly displays zero or accurate overage costs when inclusive limits are not exceeded.

HAI-6342

During resizing the instance type, note message is not proper, it is saying "Note: Resizing requires a REBOOT and will erase existing ephemeral storage"Here

Fixed an inaccurate warning message shown during instance type resize that incorrectly stated ephemeral storage would be erased. Users now receive correct guidance about the impact of resizing operations.

HAI-6777

[BE] Recipe upload fails when archive filename doesn't exactly match internal folder/json names

Fixed a recipe upload validation that unnecessarily required the archive filename to exactly match internal folder and JSON file names. Administrators can now upload recipe archives with any filename without encountering upload failures.

HAI-4830

Upgrade backup/restore script to support Kubernetes Deployments

Updated the backup and restore script to include Kubernetes Deployment resources. Platform backups are now more complete, ensuring Kubernetes-managed services are included in disaster recovery procedures.

HAI-4838

Resource overview behavior on access of regions with no KVM cluster

Fixed the resource overview page to gracefully handle regions with no KVM cluster configured, preventing errors or blank views. Administrators can now view resource overviews for all regions, including those without KVM infrastructure.

HAI-4858

[UI] Status for 'Attach Volume' event on the instance log page does not reflect the actual status of recipe execution

Fixed the instance log page to accurately display the status of Attach Volume recipe execution events. Users can now correctly monitor volume attachment progress directly from the instance log view.

HAI-5421

[Disk Redundancy]Dropdown option is not dynamically adjusting according to webpage scaling percentage

Fixed a UI layout issue where the disk redundancy dropdown did not scale correctly when the browser zoom level was adjusted. The dropdown now renders correctly at all zoom levels, improving accessibility and usability.

HAI-5438

[BE]Admin API: 9 discrepancies between swagger.yaml and actual routes in main.go

Fixed 9 API route discrepancies between the Swagger documentation and actual Admin API endpoints. API documentation now accurately reflects all available admin routes, improving the developer and integration experience.

HAI-6343

For pod resize instances, there is no Logs data under the logs tab

Fixed missing log data on the Logs tab for instances that have undergone a pod resize operation. Users can now view complete event logs for resized instances, aiding in troubleshooting and audit.

HAI-6395

API allowed the same storage to be added to pods in different regions

Fixed an API validation gap that allowed the same storage volume to be attached to pods in different regions, which could lead to data inconsistency. Storage volumes are now correctly validated to only be attached to pods within the same region.

HAI-6508

[Health Status] UI Improvement: CPU Health Status displays "Error" when CPU utilisation reaches 100%

Fixed the CPU health status indicator incorrectly displaying Error at 100% CPU utilisation, which is a valid operational state. Health status now correctly reflects CPU load without raising false error alerts at full utilisation.

HAI-6518

[User_Panel] Bug in Baremetal Resource Policy, Pricing Policy, and Image Policy

Fixed multiple policy management issues in the User Panel affecting Baremetal Resource, Pricing, and Image Policies. Users can now reliably create, view, and manage all three policy types without errors.

HAI-7641

[UI] Fix role dropdown/UI breakage after new team roles addition + audit for hardcoded role handling

Fixed a UI breakage in the role selection dropdown that occurred after new team roles were added to the platform. Role assignment now works correctly for all team roles, including newly introduced ones.