API Changelog: Admin CP Release 2.2 | hosted·ai Documentation

API Changelog: Admin Control Panel — 2.1-release to 2.2

Summary

Metric	Count
Old version total endpoints (from code)	383 (+4 TUS/WebSocket)
Old version documented in Swagger	31 (8% coverage)
New version total endpoints (from code)	402 (+4 TUS/WebSocket)
New version documented in Swagger	408 (100% coverage)
Genuinely new endpoints	20
Removed endpoints	1
Modified endpoints (behavioral)	4
Previously undocumented, now documented	~371
New schemas/DTOs	145
Modified schemas	7
Breaking changes	3
New middleware	1 (SecureToken)

Critical context: The oct-release Swagger spec documented only 31 out of 383 endpoints (8% coverage). The titan release added comprehensive Swagger annotations across the entire API, bringing coverage to 100%. The overwhelming majority of "new" endpoints in Swagger (371+) are not functionally new -- they existed in code before but lacked documentation. This changelog distinguishes between genuinely new endpoints and newly-documented ones.

Breaking Changes

1. [BREAKING] GET /gpuaas/sub/{sub_id}/pool/{pool_id}/target-url -- Removed

Before	After
`GET /gpuaas/sub/{sub_id}/pool/{pool_id}/target-url`	Deleted

What it did: Called UpdateRayHeadForPoolSubscription(subID, poolID) to refresh/update the Ray head node port for a pool subscription. Returned the updated target URL.

Impact: Any client calling this endpoint will get 404.

Migration: No direct replacement. Ray head port management may need to be handled through other GPUaaS pool endpoints or direct cluster access.

2. [BREAKING] GET /gpuaas/cards-specs/all -- timestamp query parameter removed

Before	After
Requires `timestamp` query parameter (Unix timestamp format)	No `timestamp` parameter needed

What changed: The timestamp validation and time conversion logic was completely removed from the ListAllCardSpecs handler. The endpoint now returns all card specs without any time-based filtering.

Impact: Clients sending timestamp will not get an error (parameter is simply ignored), but clients that expected time-filtered results will now always get the full list.

Migration: Remove timestamp parameter from requests. If time-based filtering was used, implement it client-side.

3. [BREAKING] POST /pods/action -- Now enforces concurrency locks and state validation

Before	After
No concurrency protection	Acquires ordered locks at pool+team level
No state validation	Validates pods are not in transitional states
Silent on conflicting actions	Returns 409 Conflict for invalid state transitions

New validations:

Pods in states STARTING, STOPPING, RESTARTING, PROVISIONING, DEPROVISIONING, RESIZING cannot have actions performed on them - returns 409 Conflict
Stopped pods cannot be stopped again - returns 409 Conflict
Concurrent actions on the same pool+team are serialized via ordered locks

Impact: Clients that previously sent concurrent or duplicate pod actions without error will now receive 409 responses.

Migration: Add retry logic with backoff for 409 responses. Check pod status before sending actions.

New Endpoints

Pod Management (9 endpoints) -- Major new feature

Method	Path	Handler	Description
`GET`	`/pods`	GetRunningPods	List all running pods. Optional ?search= query filter
`POST`	`/pods/provision`	ProvisionPod	Provision a new pod with GPU, CPU, memory, storage, and recipe configuration
`POST`	`/pods/deprovision`	DeProvisionPod	Deprovision a pod and release its resources
`POST`	`/pods/resize`	PodResize	Resize a pod's CPU, memory, or ephemeral storage. Returns execution UUID
`POST`	`/pods/volume`	VolumeAction	Attach/detach volumes on a pod
`GET`	`/pods/volume-mount`	GetPodVolumeMount	Get volume mount operation status for a pod
`GET`	`/pods/extract-info`	ExtractPodInfo	Extract detailed pod info (worker info, connections, etc.)
`GET`	`/pods/status`	ExtractPodStatus	Get pod status (status, IP, is_deleted)
`POST`	`/pods/{pool_subscription_id}/migrate_pods`	MigratePoolSubscriptionPods	Migrate legacy pods for a specific pool subscription

Key request schemas:

POST /pods/provision - ProvisionPodOpts:

{ "pool_id": "integer", "team_id": "string", "team_name": "string", "vgpus": "integer", "vcpus": "integer", "memory_mb": "integer", "ephemeral_storage_gb": "integer", "lv_list": [{"name": "string", "size_gb": "integer", "lv_type": "string"}], "image_uuid": "string", "worker_id": "string", "recipe_id": "integer", "recipe_scope": "string", "infiniband_enabled": "boolean", "ssh_public_keys": ["string"] }

POST /pods/deprovision - DeprovisionPodOpts:

{ "pool_id": "integer", "team_id": "string", "worker_id": "string", "image_uuid": "string", "gpu_uuids": ["string"], "pool_subscription_id": "integer" }

POST /pods/resize - PodResizeOpts:

{ "pool_id": "integer", "team_id": "string", "worker_id": "string", "cpu_millicores": "integer", "memory_mib": "integer", "ephemeral_storage": "string" }

Instance IP Endpoints (2 endpoints)

Method	Path	Handler	Description
`POST`	`/instances/ips`	GetInstanceIPs	Get IPs for instances updated after a given timestamp
`GET`	`/ips/instance/{instance_id}`	GetInstancePublicIPsByInstanceID	Get public IPs for a specific instance

Request schema for POST /instances/ips:

{ "last_updated_at": "2024-01-01T00:00:00Z" }

Response: Array of {instance_id: uint, instance_ips: string[]}

GPUaaS Endpoints (3 endpoints)

Method	Path	Handler	Description
`POST`	`/gpuaas/{gpuaas_id}/max_deployable_vgpus/batch`	GetMaxDeployableVGPUsBatchForPools	Batch query max deployable vGPUs across multiple pools
`GET`	`/gpuaas/pool/{pool_id}/available_resources`	GetAvailableResourcesForPool	Available resources (CPU, memory, storage) for all nodes in a pool
`POST`	`/gpuaas/node/reboot`	GpuaasNodeReboot	Reboot a GPUaaS node. Query: gpuaas_node_id. Requires admin role

Request schema for batch vGPUs:

{ "pool_names": ["string"], "min_ram_mb": "integer", "min_storage_mb": "integer", "min_cpu_cores": "integer", "team_id": "string" }

Recipe/Inventory Unified Execution (6 endpoints)

Method	Path	Handler	Auth	Description
`POST`	`/recipes/inventories_unified`	CreateUnifiedInventory	JWT	Create a unified inventory supporting internal VMs, pods, and external VMs
`POST`	`/recipes/unified_execution`	CreateRecipeUnifiedTest	JWT	Execute a recipe through the new unified execution engine
`DELETE`	`/recipes/playbook_runs_unified/{id}`	DeleteUnifiedPlaybookRun	JWT	Delete a unified playbook run
`GET`	`/recipes/exec/status/{exec_uuid}`	GetRecipeStatusByExecutionUUID	JWT	Get recipe execution status by UUID
`GET`	`/recipes/internal/{id}/export`	ExportRecipeForK8s	SecureToken	Export recipe template as .tar.gz for K8s deployment
`GET`	`/recipes/internal/{id}/metadata`	GetRecipeMetadata	SecureToken	Get recipe template metadata (updated_at, version)

POST /recipes/inventories_unified - CreateUnifiedInventory:

{ "name": "string", "description": "string", "type": "string", "internal_vms": [{"instance_id": "integer", "alias": "string"}], "pods": [{"worker_id": "string", "alias": "string"}], "external_vms": [{"name": "string", "ip": "string", "port": "integer", "user": "string", "password": "string", "ssh_key_id": "integer"}] }

Note: The two /recipes/internal/* endpoints use SecureToken authentication (static X-Internal-Secret header) instead of JWT. They are exempted from JWT auth middleware.

Removed Endpoints

#	Method	Path	Handler	Replacement
1	`GET`	`/gpuaas/sub/{sub_id}/pool/{pool_id}/target-url`	RefreshRayHeadPort	None - functionality removed entirely

Modified Endpoints (Behavioral Changes)

POST /pods/action [BREAKING]

Now acquires ordered locks at pool+team level to prevent concurrent actions
Validates pods are not in transitional states before acting (returns 409 Conflict)
Now uses the unified recipe execution engine instead of legacy execution
Now uses CreateUnifiedInventory instead of CreateInventory

GET /gpuaas/cards-specs/all [BREAKING]

timestamp query parameter removed -- no longer does time-based filtering

GET /gpuaas/node/test_conn [NON-BREAKING]

Internal method call changed from TestNodeConn to TestNodeConnV2 -- same interface, improved implementation

POST /resource_groups/{rg}/instances [NON-BREAKING]

Commented-out GPU pool subscription code cleaned up
json.Marshal error handling improved

Previously Undocumented - Now Documented

The titan release added comprehensive Swagger annotations to ~371 endpoints that existed in oct-release code but had no Swagger documentation. These are not functionally new -- they worked in oct-release but consumers relying on Swagger docs wouldn't have known about them.

Documentation Coverage by Domain

Domain	Endpoints	oct-release Swagger	titan Swagger
Clusters (nodes, datastores, networks, etc.)	93	0	93
GPUaaS (pools, nodes, GPUs, subscriptions)	71	29	71
Recipes (templates, inventories, execution)	69	0	69
Resource Groups (instances, networks, etc.)	53	0	53
Instances (CRUD, VNC, resize, migrate)	31	0	31
Auth/Users/Roles	27	0	27
Pods	14	0	14
Regions	7	7	7
Providers	6	0	6
System (config, health, maintenance, etc.)	14	0	14
Others (tags, snapshots, branding, etc.)	17	0	17
Total	402	31 (8%)	402 (100%)

Key Categories Now Documented

The following major API areas were completely undocumented in oct-release and are now fully documented:

Cluster management -- Full lifecycle (CRUD, licensing, nodes, disks, networks, datastores, vdisks, images, flavors, PCI devices, support sessions, SAUS messages, DHCP, date/time, factory reset, migration, initialization)
Resource group management -- Full CRUD plus nested resources (networks, datastores, instances, templates)
Instance management -- Full lifecycle (CRUD, resize, migrate, VNC, CDROM, PCI/PLT devices, backup, credentials)
User/Role management -- Full RBAC (users CRUD, roles CRUD, assignment, permissions)
Recipe engine -- Templates, inventories, playbook execution, keystores, repositories
Provider management -- Cloud providers CRUD with instance type queries
System administration -- Configuration, domain access, white labeling, DB backups, maintenance mode

Schema/DTO Changes

New Schemas (145 total)

Key new schema groups:

Pod Schemas

Schema	Fields	Description
`models.ProvisionPodOpts`	14	Pod provisioning request
`models.DeprovisionPodOpts`	6	Pod deprovisioning request
`models.PodResizeOpts`	6	Pod resize request
`models.PodActionsOpts`	4	Pod action request (start/stop/restart)
`models.VolumeActionOpts`	5	Volume attach/detach request

Instance Schemas

Schema	Fields	Description
`models.Instance`	46	Full instance model
`models.CreateInstanceOpts`	25	Instance creation options
`models.ResizeInstanceOpts`	8	Instance resize options
`models.MoveInstanceOpts`	4	Instance move options
`models.InstanceIPOpts`	1	Instance IP query options

Cluster Schemas

Schema	Fields	Description
`models.Cluster`	21	Full cluster model
`models.NewCluster`	8	Cluster creation request
`models.Node`	21	Cluster node model
`models.Network`	24	Network model
`models.VDisk`	15	Virtual disk model
`models.Image`	24	OS image model

Recipe Schemas

Schema	Fields	Description
`models.CreateRecipeTemplateOpts`	9	Recipe template creation
`models.PlaybookOptions`	10	Playbook execution options
`models.CreateUnifiedInventory`	6	Unified inventory creation
`models.RecipeInstances`	16	Recipe execution instance

GPUaaS Schemas

Schema	Fields	Description
`models.GPUaaS`	8	GPUaaS instance model
`models.GPUSpecs`	10	GPU specifications
`models.NPUSpecs`	17	NPU specifications
`models.BatchMaxDeployableVGPUsInput`	5	Batch vGPU query

Removed Schemas (1)

Schema	Replacement
`db.GpuaasGPUScanRequestStatus`	Replaced by `db.ResourceScanRequestStatus` (generalized for both GPU and NPU scans)

Modified Schemas (7)

models.AddNodeGpuaasOpts [NON-BREAKING]

Change	Field
Added	`assign_open_sm_secondary_role` (bool)

models.CreateRegionReq [NON-BREAKING]

Change	Field
Added	`support_rdma` (enum: full, none, partial)

models.CreateRegionResp [NON-BREAKING]

Change	Field
Added	`data` (Region object in response)

models.GPUaaSNodeDetails [NON-BREAKING]

Change	Field
Added	`hcas` (HCA device info array)
Added	`node_open_sm_role` (OpenSM role assignment)
Added	`status` (node status)

models.GpuaasGPUScanRequest [NON-BREAKING]

Change	Field
Changed	`status` type: GpuaasGPUScanRequestStatus changed to ResourceScanRequestStatus

models.NodeGpuaasResp [NON-BREAKING]

Change	Field
Added	`node_open_sm_role` (OpenSM role)
Added	`status` (node status)

models.Region [NON-BREAKING]

Change	Field
Added	`support_rdma` (RDMA support level)

Auth and Middleware Changes

New: SecureToken Middleware [NON-BREAKING]

A new authentication mechanism for internal service-to-service communication:

Header: X-Internal-Secret: <token> Config: confs.Conf.RecipeAccessToken

Aspect	Detail
Purpose	Authenticate internal recipe API calls (K8s workers to admin API)
Mechanism	Static token in X-Internal-Secret header validated against config
Response on failure	401 Unauthorized
Endpoints using it	GET /recipes/internal/{id}/export, GET /recipes/internal/{id}/metadata

These two endpoints are excluded from JWT authentication via regex patterns in notAuthEndpoints, and instead use SecureToken at the handler level.

Improved: Error Handling in Auth Middleware [NON-BREAKING]

Regex matching errors in notAuthEndpoints and virtupianAllowedEndpoints loops now return 401 instead of being silently discarded

Improved: json.Marshal Error Handling [NON-BREAKING]

Across 30+ handlers, json.Marshal error handling was improved from b, _ := json.Marshal(...) to proper error checking with 500 responses on failure.

Route Structure Changes

No Breaking Route Structure Changes

All routes remain under the /api/ prefix
No route prefix changes
No API versioning changes
The router remains gorilla/mux

Internal Changes

Change	Before	After
Scheduler name	sunlight-scheduler	hai-scheduler
Username prefix	sunlight_	hosted_ai_
Graceful shutdown	Simple DB close + 15s HTTP timeout	Context cancellation -> 10s recipe worker drain -> DB close -> 5s HTTP timeout
App context	None	context.WithCancel propagated to scheduler and recipe workers

Configuration Changes

New Configuration Fields

Field	Type	Default	Description
`MaxConcurrentRecipeExecutions`	int	10	Controls unified recipe worker pool size
`DeleteContainerAfterExecution`	bool	true	Whether to remove Docker containers after recipe execution
`RecipeAccessToken`	string	(hardcoded default)	Static token for internal recipe API auth (SecureToken)

Changed Configuration Values

Field	Old Default	New Default	Impact
`DockerContainerImage`	alpine/ansible	alpine/ansible:2.18.6	Pinned version
`DefaultHAIAdminPassword`	hosted_ai	hostedai	Default admin password changed

Migration Guide

Step 1: Remove target-url Endpoint Usage

REMOVED: GET /api/gpuaas/sub/{sub_id}/pool/{pool_id}/target-url

This endpoint has been removed. If your integration used it to refresh Ray head ports, you'll need an alternative approach.

Step 2: Update cards-specs/all Calls

OLD: GET /api/gpuaas/cards-specs/all?timestamp=1700000000 NEW: GET /api/gpuaas/cards-specs/all

Remove the timestamp parameter. The endpoint now returns all card specs unconditionally.

Step 3: Handle 409 Conflict from Pod Actions

Add error handling for HTTP 409 responses from POST /api/pods/action:

HTTP/1.1 409 Conflict {"message": "pod is in transitional state"}

Implement retry logic with backoff. Check pod status before sending actions to avoid conflicts.

Step 4: Use New Pod Lifecycle Endpoints (Optional)

For new integrations, the full pod lifecycle is now available:

POST /api/pods/provision -> Create a new pod GET /api/pods/status -> Check pod status POST /api/pods/resize -> Resize pod resources POST /api/pods/volume -> Attach/detach volumes GET /api/pods/volume-mount -> Check volume mount status POST /api/pods/deprovision -> Remove a pod

Only 31 of 383 endpoints (8%) had Swagger documentation
Documentation was limited to: GPUaaS nodes (15 endpoints), GPUaaS pools (2 endpoints), Regions (7 endpoints), NPUs (2 endpoints), and a few others
Major API areas completely undocumented: Clusters, Resource Groups, Instances, Users, Roles, Permissions, Recipes, Providers, System Admin

After (titan)

402 of 402 endpoints (100%) have full Swagger documentation with:
- Summary and Description annotations
- Param annotations for all parameters (path, query, body)
- Success and Failure response types
- Router path and method
- Tags for API grouping
- Security annotations where applicable
Auto-generated swagger.json and swagger.yaml available at /api/docs/swagger/doc.json
RapiDoc UI available at /api/docs

Recommendations

Maintain 100% Swagger coverage -- require annotations for all new endpoints
Use Swagger codegen for client SDKs to prevent drift
Add CI checks to verify Swagger coverage doesn't regress

Verification Checklist

All 383 oct-release routes from main.go accounted for (382 common + 1 removed)
All 402 titan routes from main.go documented (382 common + 20 new)
20 genuinely new routes confirmed via main.go diff
1 removed route confirmed (target-url)
3 breaking changes identified and documented
All breaking changes covered in migration guide
Schema changes enumerated (145 new, 1 removed, 7 modified)
Auth/middleware changes documented (SecureToken, error handling)
Configuration changes documented (3 new, 2 changed)
~371 previously undocumented endpoints identified
Endpoint math verified: 383 old - 1 removed + 20 new = 402 new

Generated on 2026-02-25. Source: git diff oct-release..titan in sim_api repository.