Execute Vast.ai incident response for GPU instance failures and outages.
--- name: vastai-incident-runbook description: >- Analyze, diagnose, and recover a Vast.ai renter workload from outbid, exited, offline, scheduling, low-credit, or data-risk incidents while controlling billing and destructive actions. Use when a Vast.ai workload or account enters an incident state. Trigger with: "Vast.ai incident", "recover an offline Vast.ai job", "stop emergency Vast.ai spend". allowed-tools: Read, Grep, Write, Edit version: 2.0.0 argument-hint: '[severity-resource-ids-and-last-good-checkpoint]' model: inherit effort: high license: MIT author: Jeremy Longshore <jeremy@intentsolutions.io> tags: - saas - vastai - incident-response - recovery - billing compatibility: 'Requires affected resource IDs, read access, external checkpoints, incident authority, and a named billing owner.' --- # Vast.ai Workload Recovery Runbook ## Overview Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry. ## Prerequisites - Severity, start time, affected instances/endpoints, release, and user impact - Last verified external checkpoint or Serverless template - Incident commander, data owner, billing owner, and authority for stop/destroy/rollback ## Instructions ### Step 1: Stabilize identity and scope Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved. ### Step 2: Classify the provider state Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data. ### Step 3: Preserve recovery evidence Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy. ### Step 4: Contain cost or exposure Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision. ### Step 5: Restore on a clean target Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template. ### Step 6: Reconcile and close Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action. ## Authentication Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated. ## Tool Discipline Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval. ## Output - State-based incident timeline and containment decision - Checkpoint/template recovery and service-validation evidence - Resource cleanup, charge reconciliation, access rotation, and postmortem receipt Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources. ## Examples After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation. ## Error Handling | Failure | Response | | --- | --- | | No external checkpoint exists | State the recovery gap explicitly and attempt data salvage only if the host becomes reachable. | | Balance is zero or negative | Escalate immediately; instances stop and resources may be at risk of deletion. | | Destruction would erase the only data | Require the incident commander and data owner to make the containment decision. | | Replacement also fails | Stop churn, compare common image/data/config factors, and escalate with redacted evidence. | ## Resources - [First-party source notes](references/official-docs.md) - [Troubleshooting](https://docs.vast.ai/guides/reference/troubleshooting) - [Billing and negative balances](https://docs.vast.ai/guides/reference/billing#negative-balances) - [Manage instances](https://docs.vast.ai/guides/instances/manage-instances)--- name: vastai-incident-runbook description: >- Analyze, diagnose, and recover a Vast.ai renter workload from outbid, exited, offline, scheduling, low-credit, or data-risk incidents while controlling billing and destructive actions. Use when a Vast.ai workload or account enters an incident state. Trigger with: "Vast.ai incident", "recover an offline Vast.ai job", "stop emergency Vast.ai spend". allowed-tools: Read, Grep, Write, Edit version: 2.0.0 argument-hint: '[severity-resource-ids-and-last-good-checkpoint]' model: inherit effort: high license: MIT author: Jeremy Longshore <jeremy@intentsolutions.io> tags: - saas - vastai - incident-response - recovery - billing compatibility: 'Requires affected resource IDs, read access, external checkpoints, incident authority, and a named billing owner.' --- # Vast.ai Workload Recovery Runbook ## Overview Protect people and credentials first, then preserve recoverable data, stop uncontrolled spend, and restore service from an external checkpoint or last-known-good template. Provider states imply different actions and must not be collapsed into retry. ## Prerequisites - Severity, start time, affected instances/endpoints, release, and user impact - Last verified external checkpoint or Serverless template - Incident commander, data owner, billing owner, and authority for stop/destroy/rollback ## Instructions ### Step 1: Stabilize identity and scope Confirm account/team context, affected IDs, current state, balance, and whether a credential or data exposure is involved. ### Step 2: Classify the provider state Outbid/stopped may retain disk and storage charges; exited is workload failure; scheduling may await a reclaimed GPU; unknown/offline is host heartbeat loss; low credit threatens availability and data. ### Step 3: Preserve recovery evidence Copy reachable artifacts and logs, record template/image identity, and verify the newest external checkpoint before any destroy. ### Step 4: Contain cost or exposure Stop new creates, revoke compromised keys, and use stop or destroy only under the incident's data-versus-cost decision. ### Step 5: Restore on a clean target Resume from the verified checkpoint on a compliant different offer, or roll Serverless back to the last-known-good template. ### Step 6: Reconcile and close Confirm service and output, destroy superseded resources, audit charges, rotate temporary access, and document the gap and prevention action. ## Authentication Use an incident key with the minimum temporary permissions and a short revocation deadline. Keep billing-write, team administration, workload storage, and control-plane authority separated. ## Tool Discipline Use Read and Grep to inspect manifests, configuration, provider output, and existing tests before proposing a mutation. Use Write or Edit only for the approved plan, implementation, test, or redacted receipt; do not create, update, destroy, or fund Vast.ai resources without explicit operator approval. ## Output - State-based incident timeline and containment decision - Checkpoint/template recovery and service-validation evidence - Resource cleanup, charge reconciliation, access rotation, and postmortem receipt Return severity, IDs, state class, last good checkpoint/template, containment, restored target, data gap, spend impact, and closed resources. ## Examples After a host goes offline, the team avoids blind restarts, restores the last external checkpoint on a different verified offer, validates output, and retains the original instance ID for support and billing reconciliation. ## Error Handling | Failure | Response | | --- | --- | | No external checkpoint exists | State the recovery gap explicitly and attempt data salvage only if the host becomes reachable. | | Balance is zero or negative | Escalate immediately; instances stop and resources may be at risk of deletion. | | Destruction would erase the only data | Require the incident commander and data owner to make the containment decision. | | Replacement also fails | Stop churn, compare common image/data/config factors, and escalate with redacted evidence. | ## Resources - [First-party source notes](references/official-docs.md) - [Troubleshooting](https://docs.vast.ai/guides/reference/troubleshooting) - [Billing and negative balances](https://docs.vast.ai/guides/reference/billing#negative-balances) - [Manage instances](https://docs.vast.ai/guides/instances/manage-instances)
Imported from the listed AgentSkill.sh category. Complete supplied package files and notices are retained. Required external tools and credentials are not installed. No bundled code was executed during import. Individual agent task behavior has not been tested.