2.5.0 Release Notes

Summary

HostedAI Platform | Major Feature Releases & Enhancements

Release Callisto turns Token Factory into a commercial, per-token product and lays the infrastructure groundwork to run and observe it at scale — a new multi-GPU sharing architecture, dedicated networking and per-GPU telemetry — while hardening the User Panel against destructive mistakes and unbounded disk growth.

New Features

Key Features This Release

Feature	Highlights	Business Impact
Token Factory — Sellable MVP (HAI-9928) & TF-Ansible-Integration - (HAI-10346)	Turns the internal Token Factory demo into a GA-ready, pay-per-token LLM product with an OpenAI-compatible API, fine-tuning, and RAG. Ansible automation that turns a plain GPU node into one capable of serving Token Factory models, and a region into one with a working control plane.	Opens a new revenue stream — customers consume models on demand without deploying or owning GPU capacity. The deployment backbone that makes Token Factory (HAI-9928) installable at scale.
HAISHARE_V2 : New HAISHARE Architecture based on Multi-GPU and Multi-Client Capability -(HAI-4597)	Redesigns the platform's GPU-sharing engine (HAISHARE V2) to support multiple GPUs per pod with NVLink-aware scheduling across a pool of physical GPUs, replacing the old single-GPU, static model.	Better GPU utilisation across tenants, live workload migration without restarts, and stronger isolation between users sharing hardware. Emulated GPU pools - Create virtual pools of any smaller emulated GPU type on top of existing physical GPU type pools without reserving any resources up front. This enables Service Providers to create and advertise many types of GPU pools using the same underlying physical GPU pool.
Advanced Networking for GPUaaS ( Phase 1 - Networking Foundations) - (HAI-3564)	Adds dedicated per-pod public IPs and private VLAN/overlay networking for GPUaaS (Phase 1), alongside the existing shared-IP model, without disrupting current workloads.	Stronger network isolation and predictable connectivity for teams, laying the foundation for more secure multi-tenant GPU workloads.
GPU Metrics Dashboard - (HAI-7720)	Adds live and historical per-GPU telemetry (temperature, power, clocks, running processes) across admin, user, and marketplace views.	Operators and end users can pinpoint real performance issues instead of guessing; marketplace suppliers gain visibility into pools they've sold.

Improvements to existing features

Ticket	Highlights	Area	Business Impact
Destructive Action Safety Observability - (HAI-8604)	Adds pre-flight previews and hard-blocks to risky delete actions (workspaces, teams, volumes, policies) so users see the impact before confirming.	User Panel	Prevents accidental data loss and cascade failures, reducing support tickets.
Node Logs Viewer - (HAI-5059)	Gives admins a consistent operational log across GPUaaS, KVM, and Baremetal nodes, clearly showing whether actions succeeded or failed with error detail.	Admin Panel	Admins can self-diagnose node issues instead of escalating to backend support, cutting resolution time.
Disk Hygiene and Retention - (HAI-8628)	Closes three known sources of unbounded disk/database growth, including a runaway log file that already caused a real outage.	Platform / Infrastructure	Prevents repeat disk-full incidents and keeps platform operations predictable.

Platform Initiatives

Ticket	Highlights	Platform/Product Benefit
API Code Refactoring Phase 1 - (HAI-10045)	Consolidates completed backend refactoring — dependency-injection cleanup and GORM v1→v2 framework migration — across the admin and user panel APIs (18 closed items across two source epics, HAI-9308 and HAI-3455).	A more maintainable codebase, reducing bug risk in future feature work — an indirect stability win for end users.
Implementation of UT Coverage Across all Dev components Phase 1 - (HAI-4893)	Establishes systematic unit test coverage (targeting 80%) across six core repos — User Panel API/UI, Admin Panel API/UI, Ansible recipes, and Cluster Panel UI — as a standing part of the Definition of Done.	Fewer regressions reaching end users, and faster, safer development going forward.

Bug fixes

Issue Key	Issue Summary	Business Impact / Value to End Users
HAI-6443	gpuaas_worker_init fails: "Could not find the requested service sshd" on Ubuntu nodes	Operators can now onboard Ubuntu-based GPU nodes without manual intervention or workarounds, and the initialization process completes as expected.
HAI-6513	[Security Issue] UI remains accessible in second tab after logout	Logging out now immediately ends the session across all open tabs and redirects them to the login page.
HAI-7607	haiDra: GetAvailableNodesHandlerHTTP and GetPodResourceUsageHandlerHTTP return HTTP 200 on error	These checks now correctly report failures so issues are caught and addressed more quickly.
HAI-7609	[ansible] Bug in conditional rendering of modified nvidia-smi and nvtop mounts into worker pods	These tools are now reliably included regardless of which node a pod runs on.
HAI-7658	Dynamic fallback from Temporal mode to Spatial mode does not trigger when workload is stopped	GPUs now automatically return to Spatial mode as soon as their workload stops, freeing capacity for other jobs sooner.
HAI-7892	Resource Policy: "Add Region" dropdown shows regions already selected by sibling region forms	Admins can now build Resource Policy region configurations with confidence, knowing duplicate region assignments are prevented at the point of selection.
HAI-8408	After resetting the node, it is not properly cleaned up; docker and calico interfaces remain.	Node resets now perform a complete cleanup, ensuring nodes return to a truly clean state and reducing the chance of networking issues afterward.
HAI-8531	[VM] 500 Internal Server Error on VM resize — degraded root disk blocks resize with no sync prompt	Operators can immediately take the correct remediation step without guessing or raising a support ticket.
HAI-8581	[UI] User email overflows the account dropdown container	The email address is now properly truncated in the account dropdown, keeping the menu tidy and readable for all users.
HAI-8600	[BE] User API: Swagger docs out of sync — 2 undocumented endpoints, missing card_count field, wrong response type	The API documentation has been updated to accurately reflect all available endpoints, fields, and response formats, making it easier for customers and integration partners to build against the API correctly.
HAI-8642	[VM] Auto Restart toggle shows disabled in UI while API returns auto_restart = true	Operators can now rely on the Auto Restart toggle to accurately reflect the true VM configuration, eliminating the risk of inadvertently disabling a setting that was already correctly enabled.
HAI-8655	Critical GPU pod failures due to haishare scheduler disruption	This issue has been resolved, improving the reliability and stability of GPU workloads running on shared GPU instances.
HAI-8772	[cluster panel] Instance Resize page: overlapping/misaligned text in Custom VM Specifications section	Spacing has been corrected so all labels and fields display clearly when customizing VM specifications during a resize.
HAI-8814	Investigate persistent CUDA hangs and GPU performance degradation after memswap workaround	Affected instances now run inference workloads reliably and at expected performance, without unexpected freezes or slowdowns.
HAI-8830	Intermittent issue(observed once):IP not set after pod provision in packet env.	This issue has been fixed, ensuring instance IP addresses are reliably assigned and displayed right after provisioning.
HAI-8843	Shared Storage utilization % (used/total) can mislead users when volume is actually full (0 avail per df)	This has been corrected so the indicator now accurately reflects when a volume is at capacity, helping you avoid failed writes from unexpected out-of-space errors.
HAI-8854	[BE] User Panel API: x-api-key auth is O(n) over all system-wide active API keys, causing 12-22s latency on every request	API key authentication has been optimized so these requests now return in normal time regardless of how many API keys exist across the platform.
HAI-8859	[UI] User Panel — "Create shared volume" button on Shared Storage page is disabled until a region is selected in the list filter	The button is now always enabled, removing this unnecessary extra step when creating a shared volume.
HAI-8892	[AdminPanel] 401 interceptor triggers infinite /logout request loop on login page when credentials are invalid	This has been fixed so an expired or invalid session now results in a single clear error message instead of repeated background activity.
HAI-8927	[UI] Admin CP: Editing a GPUaaS node falsely rejects it with "A node with this IP and port already exists"	This validation issue has been fixed so node edits that don't change the IP or port now save successfully.
HAI-8963	Bug Report: "Too many requests. Please try again later." blocks pod deployment	This has been addressed so routine page reloads and re-logins no longer prevent you from deploying pods.
HAI-9050	Instance type potential bug report	This has been addressed so eligible instance types are more accurately evaluated and made available during deployment.
HAI-9052	[BE] Pod deprovision and LV-delete recipes deadlock when run in parallel on the same node — GPU never released, node undeletable	This has been fixed so volume detachment is only confirmed once fully complete, preventing stuck deletions and permanently locked resources.
HAI-9053	Bug: sysbox-mgr BPF storage-quota counts writes to NFS shared volumes against the per-instance ephemeral-storage budget	This has been fixed so that writes to shared storage are no longer limited by the instance's separate local storage allowance, preventing training data, checkpoints, and other outputs from being interrupted.
HAI-9095	Regression in VRAM allocation on Dynamic RTX 6000 Pro instances causing CUDA OOM and VRAM availability issues	This has been fixed so that GPU memory is reliably available and correctly isolated, improving startup success and performance for these instances.
HAI-9233	[BE] Baremetal auto-assignment silently stalls in "Requested" forever — no error surfaced when instance type doesn't match node specs	This has been fixed so matching bare-metal instance requests to available hardware is more reliable, reducing instances that hang without ever being provisioned.
HAI-9284	Nested Docker containers bypass HAIShare entirely — LD_PRELOAD interposer never loaded, co-tenant workloads hit CUDA OOM on shared slices	This has been fixed so nested container workloads are now properly included in GPU memory isolation and scheduling, preventing this type of cross-tenant impact.
HAI-9295	Baremetal provisioning is non-atomic - password step can rotate/apply on the node without persisting ssh_password, leaving DB and node out of sync and the credential unrecoverable	Provisioning and credential handling have been made more robust so this scenario is reduced and produces a clear error rather than an unrecoverable state.
HAI-9296	CLONE - Intermittent "No GPU available" at Launch GPU → B200 → product selection step (customer-facing)	This has been fixed so B200 availability displays consistently and loads faster during the launch flow.
HAI-9430	As an administrator, I'm trying to invite users but they aren't getting emails	The platform now provides clearer visibility into email delivery configuration so administrators can identify and resolve invitation delivery issues during setup.
HAI-9436	Concurrent pool operations (add/remove_gpu_from_pool) are not isolated → recipe failures and cross-node device-plugin corruption	Concurrent pool changes are now properly coordinated, improving the reliability of multi-GPU pool operations and preventing them from affecting unrelated nodes.
HAI-9463	User Panel UI: cross-tab session isolation not enforced (JWT can leak between tabs on same origin)	Each browser tab now keeps its own independent session, preventing this kind of cross-tab account confusion.
HAI-9473	libhaishare scheduler deadlock with memswap enabled causes cuMemGetInfo_v2 to block indefinitely	GPU workloads now continue running reliably when memory swap is active, improving stability for workloads that share GPU memory under load.
HAI-9548	Marketplace team-metrics sync loop saturates admin-console.packet.ai: unbounded seller SELECT + frozen buyer watermark + silent 10MB truncation	Marketplace usage data now syncs more efficiently, so consumption charts display reliably and overall system responsiveness is improved.
HAI-9625	Team/Volume based utilization shows incorrect active instances and stale instance IDs	This has been fixed so the utilization view now displays accurate, current instance counts and IDs.
HAI-9626	Baremetal resource scan is not detecting the GPU Card	This has been fixed so GPU details are now correctly reported for these nodes.
HAI-9656	Bug: sysbox-mgr BPF storage-quota forges ENOSPC on copy_file_range() to the local rootfs, breaking apparmor configure / Docker install inside GPU instances	This issue has been resolved, so these installations and updates now complete successfully without the false error.
HAI-9724	If the baremetal service images are locked and it is visible, during instance creation, it should be visible, but it should not be editable, and also image is not displaying expected one.	This has been fixed so a locked image is now shown correctly and can no longer be changed during instance creation.
HAI-9763	Legacy GET /api/instance cannot list pod instances — returns an error instead of the list	This has been fixed so the list is now returned correctly in all cases.
HAI-9779	GET /api/policy/image/team/{team_id}/objects returns 500 for every caller, including full administrator	This restores the ability for all users, including administrators, to see which images are covered by their image policies.
HAI-9781	GET /api/gpuaas/compatible-instances/{team_id} returns 500 when the team has pod instances	Eligible instances are now correctly returned while pod instances, which do not qualify, are properly excluded.
HAI-9820	haishare-runtime: bind-mount real /usr/bin/nvidia-smi into nested containers to break shim self-recursion	GPU details are now reported correctly in these nested container scenarios.
HAI-9836	Team Members API ignores page query parameter; pagination may be broken platform-wide	Teams with more than 10 members can now retrieve their full member list as expected.
HAI-9899	apache2 fails to start (port 80 EADDRINUSE) after nightly unattended-upgrades — certbot standalone renewal races Apache restart on admin-400550	This issue is now resolved: the service automatically detects and recovers from this type of conflict within seconds, restoring availability without requiring manual intervention.
HAI-10016	GPUaaS service - instance type is locked and it is visible, this service combination is not displaying during instance creation	This has been fixed, so all correctly configured instance type options now display as expected when creating instances.
HAI-10055	Custom nvidia-smi binary inside pod incorrectly reports 100% GPU utilization when idle	This has been fixed so GPU utilization reported inside the instance now accurately reflects actual usage, giving customers reliable metrics for monitoring and capacity planning.
HAI-10060	Haishare bypass when docker-compose specifies runtime: nvidia	This gap has been closed so GPU access is now consistently enforced for all containers, ensuring fair and secure GPU sharing across tenants.
HAI-10256	[Bug][hostedai-user] Align WHMCS module to 0-based team-members paging + release-note the 2.4.2 contract change	Team member pagination, including in billing and account-management integrations, now works correctly across all pages.
HAI-10257	[Bug][hostedai-user] Two backward-incompatible API changes in 2.4.2 — login/profile drops user fields + used_size_in_gb integer→decimal	These fields now return consistently, improving reliability for anyone integrating with account and storage data.
HAI-10452	Node status shows "init_failed" after a Reset/Initialise Ansible run	Node status now accurately reflects whether initialization actually succeeded, avoiding unnecessary confusion and troubleshooting.
HAI-10580	GPUaaS: v_ram_gb is a derived per-slice figure, inconsistent across endpoints and billing paths	Reported vRAM figures are now consistent across the API and user interface, making it easier to understand what your subscription actually provides.
HAI-10627	GPU OOM at startup on Qwen3.6-27B-FP8 with HAISHARE_PASSTHROUGH=1	Model deployments using this configuration now start up reliably without requiring manual workarounds.
HAI-10863	Baremetal node stuck "In service" when SSH provisioning fails - sub marked Failed but node never rolled back	Failed assignments now roll back automatically so the server becomes available again right away, instead of requiring manual support intervention.
HAI-10984	Default Network Policy does not show assigned teams and contains no region	The policy list now accurately shows which teams and regions are using the Default Network Policy, making policy reviews more reliable.
HAI-11044	Memory-residency limits causing severe paging and performance degradation above 48 GB VRAM [RTX 6000 Pro - Dynamic]	Large-memory workloads, such as language-model serving with big context windows, now maintain consistent throughput closer to the card's full advertised memory capacity.
HAI-11083	GPU Usage Visibility Incorrect in nvidia-smi and Platform Metrics	GPU utilization and memory metrics now accurately reflect what is actually running on your instance.

Release V2.5.0 Upgrade Document - Upgrade Instructions Doc

What this does

This migration brings a pre-Release V2.5.0 GPUaaS cluster (Release V2.4.1 ) to the same networking runtime state as a fresh Release V2.5.0 installation.

This patch contains sysbox upgrade on host which is option. Unless sysbox is upgraded we do not need pod/workload restart.

Note: This guide is only for upgrading Release v2.4.1 → Release V2.5.0, any versions prior are to be updated manually.

Download the upgrade files available at https://fs.files.hostedai.tech/9/r/Callisto-upgrade.zip

Target state

Component	Before	After
Calico	Old version, default apply mode	v3.31.3, server-side apply with force conflicts, plus IPReservation
Multus	Absent	Installed
Kube-OVN	Absent	Installed and bound to a per-node provider interface
Sysbox (Optional, requires worker restart + backup if rootfs not enabled)	Old package version	Latest version from apt.hosted.ai
Haidra	Old package version	Latest version from apt.hosted.ai
Calico Cross namespace Ingress and Egress rules	Worker pods could connect/ping kube api server	All Network access to system resources is filtered/blocked
NFT firewall rules	–	Firewall rules enforced to protect sensitive endpoints from external traffic

1. Update Panels and Recipes

apt update
apt install --only-upgrade -y hostedai-adminpanel-api
apt install --only-upgrade -y hostedai-adminpanel-ui
apt install --only-upgrade -y hostedai-userpanel-api
apt install --only-upgrade -y hostedai-userpanel-ui

Update recipes using deploy utility

./deploy --recipes

2. Master node

Run the following steps in order on the master node.

Step 1: Create a backup

cd Migrate-gpuaas-network-to-callisto/
sudo bash migrateMaster.sh --backup

Step 2: Fix node IP configuration (Optional):

Prior to callisto kubelet node advertisement address was auto selected by a regex and sometimes it used to bind to localhost or tunnel interfaces. Using following command it could be fixed.

If there are multiple nodes in region with incorrect management ip, update both master and worker nodes at once (these values will be used to enforce firewall rules in later steps)

# for fixing master node use following
sudo bash migrateMaster.sh --fix-node-ip --management-ip <your-ipv4>
# for fixing the worker node use following
sudo bash migrateNode.sh   --fix-node-ip --management-ip <your-ipv4>

Step 3: Configure networking

Kube-OVN requires a provider interface for each node.

First, check the available physical interfaces on each node:

for i in /sys/class/net/*; do [ -e "$i/device" ] && basename "$i"; done

Identify the interface to be used for the Kube-OVN provider network.

For example:

master     → eth0
worker-1   → eth0
worker-2   → ens5

Use --provider-interface for the default interface, and --node-interface for nodes that use a different interface.

For the example above:

sudo bash migrateMaster.sh --calico --multus --kubeovn --provider-interface eth0 --node-interface worker-2=ens5

This means:

Default:
  All nodes → eth0
Override:
  worker-2 → ens5

If more nodes use different interfaces, add additional --node-interface options:

sudo bash migrateMaster.sh --calico --multus --kubeovn --provider-interface eth0 --node-interface worker-2=ens5 --node-interface worker-3=ens6

Format:

--node-interface <node-name>=<interface-name>

Important: --provider-interface is the default interface. Use --node-interface to override the interface for individual nodes.

Step 4: Verify cluster health

sudo bash migrateMaster.sh --health

Step 5: Upgrade Haidra

sudo bash migrateMaster.sh --haidra

Step 6: Upgrade HAI-K8-Controller

sudo bash migrateMaster.sh --hai-k8-controller

Step 7: Upgrade Sysbox (This is optional: sysbox update requires worker pod restart)

sudo bash migrateMaster.sh --sysbox

Step 8: Calico Ingress, Egress rules and tenant-isolation rules

cd Migrate-tenant-calico-firewall/
sudo bash migrateCalicoFirewall.sh --all

Step 9: Disable NodePort on prometheus, grafana and update the prometheus ssh tunnel

bash migrateCalicoFirewall.sh --prometheus

Step 10: NFT firewall on master node

Prior to callisto all of our GPUaaS nodes lacked firewall rules, this command applies NFT rules and blocks access to sensitive ports from external network.

Following command deploys a daemonset and creates a config map that syncs list of trusted ip’s across cluster

cd Migrate-host-nft-firewall/
sudo bash migrateHostFirewall.sh --master --cluster

Now render the firewall rules using

sudo bash migrateHostFirewall.sh --render --ssh-port <current-node-ssh-port>

once the rules are rendered apply them using --arm flag and test from another node if any connectivity breaks, if so these rules could be disarmed using --disarm flag

sudo bash migrateHostFirewall.sh --arm

Step 11: Update db columns in admin panel

The k8 management network interface and ovn interfaces are tracked in admin panel, these values are to be updated in admin panel database. Generate the query using

cd Migrate-gpuaas-network-to-callisto/
./generateNodeInterfaceSql.sh

Copy the generated query and update the placeholder <<FILL: mgmt NIC on upgrade-test>> management interface for each node on the cluster

Execute the query on admin panel DB

ssh into admin panel server → mysql -u root → use hai_admin → paste the above query and execute

cd Migrate-host-nft-firewall/
sudo bash /tmp/Migrate-gpuaas-network-to-callisto/migrateNode.sh --sysbox

Step 3: Apply firewall rules

render the firewall rules using

sudo bash migrateHostFirewall.sh --render --ssh-port <current-node-ssh-port>