{ "markdown": "# Terradev-cli 6.2.15\n\n**Cross-cloud GPU orchestration CLI.**\n\n![Terradev Demo](https://raw.githubusercontent.com/theoddden/Terradev/main/demo/terradev-demo.gif)\n\n**License: Apache 2.0** - Free and open source for commercial and personal use.\n\nhttps://terradev.cloud/\n\n\nTerradev is a cross-cloud compute control plane for AI workloads, not just a provisioning wrapper. \n\nCombines quoting, provisioning, topology optimization, training orchestration, inference tuning, and cost analytics in one CLI, with an accelerated idempotent runtime underneath.\n\nContinued focus on lower cost, faster provisioning, and topology-aware execution with local credential storage.\n\nModel agnostic. Dataset agnostic. GPU agnostic. Provider agnostic. The only thing Terradev is not agnostic about is correctness: it enforces topology, idempotency, and sequencing.\n\n**NOTES ON v6.2.4**\n\n- Version bumped to **v6.2.4**.\n- Provider list consolidated to **17 active GPU cloud and inference providers**:\n - `aws` — Amazon Web Services\n - `azure` — Microsoft Azure\n - `baseten` — Baseten\n - `crusoe` — Crusoe Cloud\n - `digitalocean` — DigitalOcean\n - `e2enetworks` — E2E Networks\n - `gcore` — Gcore\n - `gcp` — Google Cloud Platform\n - `huggingface` — Hugging Face\n - `hyperstack` — Hyperstack\n - `inferx` — InferX\n - `latitude` — Latitude.sh\n - `runpod` — RunPod\n - `siliconflow` — SiliconFlow\n - `tensordock` — TensorDock\n - `vastai` — Vast.ai\n - `yottalabs` — YottaLabs\n- Added Gcore (`gcore`) as a new VM provider with full lifecycle support, API token auth, and drift monitoring.\n- Removed OVHcloud from the active provider list.\n- Removed Jio Cloud and Oracle Cloud from the active provider list; removed Alibaba Cloud references from the CLI UI.\n- Onboarding, `configure`, `setup`, `quote`, `credential_prompt`, and MCP tool schemas now all reflect the 17-provider set.\n\n**NOTES ON Multi-Stage Training Pipeline (v6.1.1)**\n\n- New `terradev train` subcommands for the full post-pretraining lifecycle:\n - `terradev train sft --model --data --nodes `\n - `terradev train dpo --base-checkpoint --data --algorithm `\n - `terradev train grpo --base-checkpoint --data --framework `\n - `terradev train pipeline --config examples/training_pipeline.yaml`\n- `terradev_cli/core/training_stages.py` and `terradev_cli/core/training_pipeline.py` provide declarative SFT / DPO / GRPO stage configs, provider-aware quote selection, auto-provisioning, checkpoint handoff, and DAG sequencing via the Python `DAGExecutor`.\n- `TrainingOrchestrator` now supports multi-node remote SSH launch and end-to-end completion tracking. Training scripts and embedded configs are staged to every node and the master process is polled until the job finishes.\n- Unsloth GRPO uses the native `unsloth.GRPOTrainer` with a default rule-based reward instead of a TRL fallback.\n- CLI-style frameworks (`axolotl`, `llama-factory`, `ms-swift`, `trl`, `openrlhf`) are wrapped in self-contained Python scripts that write their embedded config files at runtime, so they are safe to copy to remote nodes.\n- See `examples/training_pipeline.yaml` for a SFT → DPO → GRPO pipeline sample.\n\n**NOTES ON 6.0.8**\n\n- **New agent subcommands** — `terradev agent sandbox`, `terradev agent mesh`, and `terradev agent mcp` are now real, dependency-resolving commands instead of placeholders:\n - `terradev agent sandbox` runs untrusted payloads with hardware-isolated runtimes (`bwrap`, `runsc`, `firecracker`, and Linux Landlock LSM), all discoverable/downloadable via `DependencyManager`.\n - `terradev agent mesh` creates a decentralized peer-to-peer agent mesh using real `libp2p` (`go-libp2p-daemon` + `p2pclient`), A2A HTTP, and `WireGuard` encrypted transports.\n - `terradev agent mcp` is a dynamic Model Context Protocol bridge with stdio and HTTP transports and a multi-server bridge.\n\n**NOTES ON 6.0.0**\n\n- **Unsloth** (`terradev train unsloth`): optimized local LLM training, serving, and coding agents with 70% lower VRAM usage and faster training via Triton kernels. Subcommands: `run`, `start`, `stop`.\n - `terradev train unsloth run --model unsloth/Llama-3.1-8B`\n - `terradev train unsloth run --model unsloth/Qwen3.6-7B-GGUF:Q4_K_M --port 8080`\n - `terradev train unsloth start claude --model unsloth/Llama-3.1-8B`\n - `terradev train unsloth stop`\n\n- **Weaviate** (`terradev database weaviate`): vector database operations with local, embedded, cloud, and custom environments. Subcommands: `up`, `list-collections`, `create-collection`, `delete-collection`, `insert`, `query`, `hybrid-search`.\n - `terradev database weaviate up --environment local`\n - `terradev database weaviate create-collection --name Article --vector-size 384`\n - `terradev database weaviate insert --collection Article --objects '[{\"properties\": {\"title\": \"Hello\"}, \"vector\": [0.1, ...]}]'`\n - `terradev database weaviate query --collection Article --vector '[0.1, ...]' --top-k 5`\n - `terradev database weaviate hybrid-search --collection Article --query \"generative AI\" --top-k 5`\n\n- **Letta** (`terradev agent letta`): stateful agents with long-horizon memory across sessions. Subcommands: `create`, `list`, `chat`, `status`, `delete`, `remember`.\n - `terradev agent letta create --name my-agent --model openai/gpt-4.1`\n - `terradev agent letta list`\n - `terradev agent letta chat --agent-id --message \"hello\"`\n - `terradev agent letta remember --agent-id --text \"Our staging cluster is on us-east-1\" --label fact`\n - `terradev agent letta status --agent-id `\n - `terradev agent letta delete --agent-id `\n\n- **Removed integrations**: `terradev ml databricks` and `terradev ml langsmith` command groups, all related MCP tools, and Jaeger tracing endpoints in the Helm values have been removed.\n- Version bumped to **6.0.0** to reflect these breaking changes.\n\n**NOTES ON 5.7.10**\n\n- **Local Ollama integration** (`terradev ml ollama`): list, pull, generate, chat, inspect, and check running models on a local Ollama server (`list`, `pull`, `generate`, `chat`, `info`, `ps`).\n- **DeepEval integration** (`terradev ml deepeval`): install, run test suites, list metrics, and evaluate single LLM outputs with metrics like `AnswerRelevancyMetric`, `FaithfulnessMetric`, and `HallucinationMetric` (`install`, `init`, `run`, `metrics`, `evaluate`).\n- **MCP tool surface** now at 237 tools, including `ollama_*` and `deepeval_*` tool definitions and handlers.\n\n**NOTES ON 5.7.7**\n\n- **Vault command** (`terradev vault`): store, sync, and use cloud API secrets from environment variables or the encrypted local vault. Designed for GitHub Actions / CI/CD pipelines where secrets are provided as `TERRADEV__` env vars.\n - `terradev vault sync` imports supported `TERRADEV__` secrets into the encrypted `~/.terradev/credentials.json` (use `--all` to also import custom keys).\n - `terradev vault run -- ` injects vault secrets into a sub-process and zeroizes them afterwards.\n - `terradev` now automatically falls back to `TERRADEV_*` environment variables when the local vault file is missing, so `terradev up` works directly from GitHub Secrets without a separate `configure` step.\n\n- **Database command** (`terradev database`): universal database and vector store operations with SQLite, PostgreSQL, Qdrant, and Redis adapters. Subcommands: `terradev database up`, `database down`, `database crud`, `database search`, `database sql`, and `database qdrant` with its own `search`, `scroll`, `upsert`, `create-collection`, and `delete-collection` operations.\n\n**NOTES ON 5.6.0**\n\nAdded **API Gateway for inference serving** with OpenAI/Anthropic/custom API entry and exit points:\n\n- **Gateway Service** (`core/gateway_service.py`): FastAPI-based gateway that provides OpenAI-compatible, Anthropic-compatible, and custom workflow API endpoints for inference serving. Integrates with Terradev's inference router and KV cache management for intelligent routing.\n\n- **Gateway CLI command** (`terradev gateway`):\n ```bash\n terradev gateway\n terradev gateway --host 0.0.0.0 --port 8080\n terradev gateway --no-anthropic --max-concurrent 50\n terradev gateway --model meta-llama/Llama-3.1-8B-Instruct\n ```\n\n- **OpenAI-compatible endpoints**:\n - POST /v1/chat/completions\n - POST /v1/completions\n\n- **Anthropic-compatible endpoints**:\n - POST /v1/messages\n - POST /v1/messages/batches\n\n- **Custom workflow endpoints**:\n - POST /v1/custom/entry/{workflow_id}\n - POST /v1/custom/exit/{workflow_id}\n\n- **Management endpoints**:\n - GET /health\n - GET /v1/gateway/status\n\n- **Features**:\n - Streaming response support\n - Configurable CORS, concurrent requests, timeouts\n - Integration with inference router for intelligent routing\n - Request/response transformation and validation\n\n**NOTES ON 5.3.9**\n\nAdded **LoRAX (LoRA eXchange) integration** and **HuggingFace PEFT import** for production-grade multi-LoRA inference serving:\n\n- **LoRAX Service** (`ml_services/lorax_service.py`): Async HTTP client for Predibase LoRAX multi-LoRA inference server that serves thousands of fine-tuned models on a single GPU with dynamic adapter loading, heterogeneous continuous batching, and adapter exchange scheduling.\n\n- **LoRAX CLI commands** (`terradev lora lorax`):\n ```bash\n terradev lora lorax deploy -m mistralai/Mistral-7B-Instruct-v0.1 --docker\n terradev lora lorax test --host localhost --port 8080\n terradev lora lorax list-adapters\n terradev lora lorax load-adapter -a vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k\n terradev lora lorax unload-adapter -a my-adapter\n terradev lora lorax generate -p \"What is 2+2?\" -a my-adapter\n terradev lora lorax sync-registry\n ```\n\n- **PEFT Import Service** (`ml_services/peft_import_service.py`): Download, validate, and prepare LoRA adapters from HuggingFace using the PEFT library with auto-detection of rank, alpha, and target modules.\n\n- **PEFT CLI commands** (`terradev lora peft`):\n ```bash\n terradev lora peft import -a vineetsharma/qlora-adapter-Mistral-7B-Instruct-v0.1-gsm8k\n terradev lora peft import -a username/adapter --local-name my-adapter --register --base-model mistralai/Mistral-7B-Instruct-v0.1\n terradev lora peft list\n terradev lora peft validate -p ~/.terradev/peft_adapters/username--adapter\n terradev lora peft delete -a username/adapter\n ```\n\n- **LoRAX Helm Template** (`clusters/lorax-template/helm/`): Production-ready Kubernetes manifests with GPU resource limits, storage configuration.\n\n- **Registry Integration**: One-step import from HuggingFace and automatic registration in Terradev LoRA registry with version tracking, cross-replica sync, and cost attribution.\n\n**NOTES ON 5.3.3**\n\nAdded **provider registration and profiling system** for intelligent quirk-aware routing across 17 cloud providers, and registration for custom providers from .yaml import:\n\n- **ProviderProfile schema** (`providers/types.py`): Encodes provider-specific behaviors including API style (REST/GraphQL/JSON:API), authentication type (Bearer/Basic/HMAC/X-Api-Key), rate limits, spot instance support, egress costs, fallback routing, capacity checks, container image pinning, and spot interruption handling.\n\n- **Built-in profiles** (`providers/provider_profiles.py`): Pre-configured profiles for all 23 providers (RunPod, Vast.ai, Lambda Labs, AWS, GCP, Azure, Oracle, Crusoe, CoreWeave, DigitalOcean, Yotta Labs, E2E Networks, FluidStack, Alibaba, Hetzner, SiliconFlow, TensorDock, Baseten, HuggingFace, Hyperstack, InferX, Latitude).\n\n- **Dynamic registration**: Users can register custom provider profiles programmatically or load from YAML/JSON files for internal clusters or proprietary cloud providers.\n\n- **Profile-aware routing** (`providers/registry.py`): `ProviderRegistry.ranked_providers()` now incorporates provider profiles into scoring, using egress costs, fallback routing preferences, and spot preemption rates for intelligent provider selection.\n\n- **CLI commands** (`terradev providers`): New command group for managing custom provider profiles:\n ```bash\n terradev providers load-profiles ~/.terradev/custom_providers.yaml\n terradev providers list-profiles\n terradev providers show-profile runpod\n terradev providers remove-profile my_custom_provider\n terradev providers export-example -o ~/.terradev/custom_providers.yaml\n ```\n\n**NOTES ON 5.2.1**\n\nAdded two new BYOAPI providers: **Yotta Labs (Shakti Cloud)** and **E2E Networks** — India's leading GPU clouds. Yotta Labs uses a pod-based compute model (similar to RunPod), and E2E Networks is a traditional VM-style hyperscaler that is NSE-listed and MeitY empanelled. Both are BYOAPI: your key, stored locally, never touches a Terradev server.\n\n```bash\nterradev configure --provider yottalabs\nterradev configure --provider e2enetworks\n```\n\n**NOTES ON 5.0.0**\n\nWe removed the paywall, open-sourced Terradev, and added accelerators for safe and snappy delivery...\n\nWith the DAG orchestrator, the execution graph enforces correct sequencing and idempotency at the runtime level. You or the agent can issue commands freely... the orchestrator ensures they're safe to execute.\n\n217 tools not including subcommand/flags require heavy context. The MCP orchestrator processes tool calls with minimal overhead: deserializing, routing, executing, and responding faster than pure-Python-based MCP servers by an order of magnitude. For an agent running a complex provisioning workflow across 17 cloud providers, that compounds across every tool call in the chain.\n\n## BYOAPI Configuration\n\nYour API keys are stored locally at ~/.terradev/credentials.json and never sent to Terradev servers.\n\n```bash\n# Configure multiple providers\nterradev configure --provider runpod\nterradev configure --provider vastai\nterradev configure --provider aws\nterradev configure --provider gcp\n```\n\n## Performance\n\n- **2-8x throughput improvements** with vLLM optimization\n- **30-50% bandwidth penalty eliminated** with NUMA topology\n- **2-5x CUDA Graph speedup** with optimal topology\n- **Up to 90% cost savings** with automatic provider switching\n- **<2 minute spot recovery** with KV cache checkpointing\n- **up to 3.6x faster cold starts** with weight streaming\n- **Up to 50% cost savings** with MLA-aware VRAM estimation\n \n## Complete Tutorial\n\n### Step 1: Install Terradev\n```bash\npip install terradev-cli\n```\n\nFor all cloud provider SDKs and ML integrations:\n```bash\npip install terradev-cli[all]\n```\n\nVerify and list commands:\n```bash\nterradev --help\n```\n\n### Step 2: Configure Your First Cloud Provider\nTerradev supports 17 GPU cloud providers. Start with one, RunPod is the fastest to set up:\n\n```bash\nterradev setup runpod --quick\n```\n\nThis shows you where to get your API key. Then configure it:\n\n```bash\nterradev configure --provider runpod\n```\n\nPaste your API key when prompted. It's stored locally at ~/.terradev/credentials.json, never sent to a Terradev server. Add more providers later:\n\n```bash\nterradev configure --provider vastai\nterradev configure --provider lambda_labs\nterradev configure --provider aws\n```\n\nThe more providers you configure, the better your price coverage.\n\n### Step 3: Get Real-Time GPU Prices\nCheck pricing across every provider you've configured:\n\n```bash\nterradev quote -g A100\n```\n\nOutput is a table sorted cheapest-first: price/hour, provider, region, spot vs. on-demand. Try different GPUs:\n\n```bash\nterradev quote -g H100\nterradev quote -g L40S\nterradev quote -g RTX4090\n```\n\n### Step 4: Provision\nMost clouds hand you GPUs with suboptimal topology by default. Your GPU and NIC end up on different NUMA nodes, RDMA is disabled, and the kubelet Topology Manager is set to none. That's a 30-50% bandwidth penalty on every distributed operation and you'll never see it in nvidia-smi.\n\nWhen you provision through Terradev, topology optimization is automatic:\n\n```bash\nterradev provision -g H100 -n 4 --parallel 6\n```\n\nWhat happens behind the scenes:\n- **NUMA alignment** — GPU and NIC forced to the same NUMA node\n- **GPUDirect RDMA** — nvidia_peermem loaded, zero-copy GPU-to-GPU transfers\n- **CPU pinning** — static CPU manager policy, no core migration\n- **SR-IOV** — virtual functions created per GPU for isolated RDMA paths\n- **NCCL tuning** — InfiniBand enabled, GDR_LEVEL=PIX, GDR_READ=1\n\nYou don't configure any of this. It's applied automatically.\n\nTo preview the plan without launching:\n```bash\nterradev provision -g A100 -n 2 --dry-run\n```\n\nTo set a price ceiling:\n```bash\nterradev provision -g A100 --max-price 2.50\n```\n\n### Step 5: Run a Workload\n\n**Option A** — Run a command on your provisioned instance:\n```bash\nterradev execute -i -c \"nvidia-smi\"\nterradev execute -i -c \"python train.py\"\n```\n\n**Option B** — One command that provisions, deploys a container, and runs:\n```bash\nterradev run --gpu A100 --image pytorch/pytorch:latest -c \"python train.py\"\n```\n\n**Option C** — Keep an inference server alive:\n```bash\nterradev run --gpu H100 --image vllm/vllm-openai:latest --keep-alive --port 8000\n```\n\n### Step 6: Manage Your Instances\n```bash\n# See all running instances and current cost\nterradev status --live\n\n# Stop (keeps allocation)\nterradev manage -i -a stop\n\n# Restart\nterradev manage -i -a start\n\n# Terminate and release\nterradev manage -i -a terminate\n```\n\n### Step 7: Track Costs and Find Savings\n```bash\n# View spend over the last 30 days\nterradev analytics --days 30\n\n# Find cheaper alternatives for running instances\nterradev optimize\n```\n\n### Step 8: Distributed Training Pipeline\nNow that your nodes have correct topology, distributed training actually runs at full bandwidth:\n\n```bash\n# Validate GPUs, NCCL, RDMA, and drivers before launching\nterradev preflight\n\n# Launch training on the nodes you just provisioned\nterradev train --script train.py --from-provision latest\n\n# Watch GPU utilization and cost in real time\nterradev monitor --job my-job\n\n# Check status\nterradev train-status\n\n# 6. List checkpoints when done\nterradev checkpoint list --job my-job\n```\n\nThe `--from-provision latest` flag auto-resolves IPs from your last provision command. Supports torchrun, DeepSpeed, Accelerate, and Megatron.\n\n### Step 9: Optimize vLLM Inference (The 6 Knobs)\nIf you're serving a model with vLLM, there are 6 settings most teams leave at defaults — each one costs throughput:\n\n| Knob | Default | Optimized | Impact |\n|------|---------|-----------|--------|\n| max-num-batched-tokens | 2048 | 16384 | 8x throughput |\n| gpu-memory-utilization | 0.90 | 0.95 | 5% more VRAM |\n| max-num-seqs | 256/1024 | 512-2048 | Prevent queuing |\n| enable-prefix-caching | OFF | ON | Free throughput win |\n| enable-chunked-prefill | OFF | ON | Better prefill |\n| CPU Cores | 2 + #GPUs | Optimized | Prevent starvation |\n\nAuto-tune all six from your workload profile:\n```bash\nterradev ml vllm auto-optimize -s workload.json -m meta-llama/Llama-2-7b-hf -g 4\n```\n\nOr analyze a running server:\n```bash\nterradev ml vllm analyze -e http://localhost:8000\n```\n\nBenchmark:\n```bash\nterradev ml vllm benchmark -e http://localhost:8000 -c 10\n```\n\n### Step 10: Deploy a MoE Model with Auto-Applied Optimizations\nFor large Mixture-of-Experts models (GLM-5, Qwen 3.5, DeepSeek V4), Terradev's MoE templates include every optimization auto-applied — KV cache offloading, speculative decoding, sleep mode, expert load balancing:\n\n```bash\nterradev provision --task clusters/moe-template/task.yaml \\\n --set model_id=Qwen/Qwen3.5-397B-A17B\n```\n\nOr a smaller model:\n```bash\nterradev provision --task clusters/moe-template/task.yaml \\\n --set model_id=Qwen/Qwen3.5-122B-A10B --set tp_size=4 --set gpu_count=4\n```\n\nWhat's auto-applied (no flags needed):\n- **KV cache offloading** — spills to CPU DRAM, up to 9x throughput\n- **MTP speculative decoding** — up to 2.8x faster generation\n- **Sleep mode** — idle models hibernate to CPU RAM, 18-200x faster than cold restart\n- **Expert load balancing** — rebalances routing at runtime\n- **LMCache** — distributes KV cache across instances via Redis\n\n### Step 11: Disaggregated Prefill/Decode (Advanced)\nThis separates inference into two GPU pools optimized for each phase:\n\n- **Prefill (compute-bound)** — processes input prompt, wants high FLOPS\n- **Decode (memory-bound)** — generates tokens, wants high HBM bandwidth\n\nThe KV cache transfers between them via NIXL — zero-copy GPU-to-GPU over RDMA. This is why getting the NUMA topology right in Step 4 matters: NIXL only runs at full speed when the GPU and NIC share a PCIe switch.\n\n```bash\nterradev ml ray --deploy-pd \\\n --model zai-org/GLM-5-FP8 \\\n --prefill-tp 8 --decode-tp 1 --decode-dp 24\n```\n\nTerradev's inference router automatically uses sticky routing. Once a prefill GPU hands off a KV cache to a decode GPU, future requests with the same prefix go to that same decode GPU, avoiding redundant transfers.\n\n### Step 12: Create a Kubernetes GPU Cluster\nFor production, create a topology-optimized K8s cluster:\n\n```bash\nterradev k8s create my-cluster --gpu H100 --count 8 --prefer-spot\n```\n\nThis auto-configures Karpenter NodePools with NUMA-aligned kubelet Topology Manager, GPUDirect RDMA, and PCIe locality enforcement.\n\n```bash\n# List clusters\nterradev k8s list\n\n# Get cluster info\nterradev k8s info my-cluster\n\n# Tear down\nterradev k8s destroy my-cluster\n```\n\n## Why This Order Matters\nEach step builds on the one before it:\n\n- **Step 4**: NUMA / RDMA / SR-IOV topology ← foundation\n- **Step 8**: Distributed training at full BW ← depends on topology\n- **Step 9**: vLLM knob tuning ← depends on correct memory layout\n- **Step 10**: KV cache offloading + sleep mode ← depends on CPU bus not saturated\n- **Step 11**: Disaggregated P/D ← depends on RDMA for KV transfer\n\nIf the provisioning layer is wrong, every optimization above it underperforms. A disaggregated P/D setup with a cross-NUMA KV transfer is slower than a monolithic setup with correct topology.\n\nTerradev handles the foundation automatically so the rest of the stack works the way it's supposed to.\n\n## Quick Reference\n```bash\n# Set up cloud provider credentials\nterradev configure\n\n# Real-time GPU pricing across 18 clouds\nterradev quote -g H100 \n\n# Provision with auto topology optimization\nterradev provision -g H100 -n 4\n\n# Provision + deploy + run in one command\nterradev run --gpu A100 --image ...\n\n# View running instances and costs\nterradev status --live\n\n# Launch training on provisioned nodes\nterradev train --from-provision latest\n\n# Auto-tune 6 critical vLLM knobs\nterradev ml vllm auto-optimize\n\n# Topology-optimized Kubernetes cluster\nterradev k8s create\n\n# Cost analytics\nterradev analytics --days 30\n\n# Find cheaper alternatives\nterradev optimize\n```\n\n### Troubleshooting Training Workflows\n\n**NCCL Connectivity Problems**\n```bash\n# Symptoms: Training hangs, NCCL errors, slow communication\n\n# Diagnosis: Check inter-node connectivity\nterradev preflight --detailed\nterradev execute -i -c \"nccl_test -b 8G -e 8G -s 1073741824\"\n\n# Fix: Re-provision with proper NUMA alignment\nterradev provision -g H100 -n 4 --parallel 6 --ensure-numa-alignment\n```\n\n**GPU Memory Issues**\n```bash\n# Symptoms: OOM errors, CUDA out of memory\n\n# Diagnosis: Check memory usage across nodes\nterradev monitor --job --memory-usage\nterradev execute -i -c \"nvidia-smi --query-gpu=memory.used,memory.total --format=csv\"\n\n# Fix: Reduce batch size or enable gradient checkpointing\nterradev train --script train.py --from-provision latest --script-args \"--batch-size 16 --gradient-checkpointing\"\n```\n\n**Dataset Staging Failures**\n```bash\n# Symptoms: Slow data loading, transfer timeouts\n\n# Diagnosis: Check dataset cache status\nterradev stage --status --dataset-id \nterradev stage --list-cached --region us-east-1\n\n# Fix: Re-stage with higher parallelism or compression\nterradev stage -d ./my-dataset --target-regions us-east-1 --parallel-streams 64 --compression zstd\n```\n\n**FlashOptim Compatibility Issues**\n```bash\n# Symptoms: FlashOptim fails to apply, training crashes\n\n# Diagnosis: Check FlashOptim compatibility\nterradev train-status --job | grep flashoptim\nterradev preflight --flashoptim-check\n\n# Fix: Disable FlashOptim or adjust configuration\nterradev train --script train.py --flashoptim off --from-provision latest\n# or with manual configuration\nterradev train --script train.py --flashoptim on --flashoptim-optimizer adamw --flashoptim-master-weight-bits 8\n```\n\n**Checkpoint Recovery Issues**\n```bash\n# Symptoms: Can't resume from checkpoint, corrupted checkpoints\n\n# Diagnosis: Verify checkpoint integrity\nterradev checkpoint list --job --verify\nterradev checkpoint validate --checkpoint \n\n# Fix: Create new checkpoint or repair existing\nterradev checkpoint save --job --force\nterradev checkpoint repair --checkpoint \n```\n\n**Performance Optimization**\n\n**Slow Training Speed**\n```bash\n# Diagnose bottlenecks\nterradev monitor --job --bottleneck-analysis\nterradev execute -i -c \"nvtop --interval 1\"\n\n# Common fixes\n# 1. Enable mixed precision training\nterradev train --script train.py --script-args \"--mixed-precision --fp16\"\n\n# 2. Optimize data loading\nterradev stage --hf-dataset --target-regions us-east-1 --preprocess \"shuffle,cache\"\n\n# 3. Increase parallelism\nterradev provision -g H100 -n 8 --parallel 12\n```\n\n**Network Bottlenecks**\n```bash\n# Check network performance between nodes\nterradev preflight --network-test\nterradev execute -i -c \"ibstat -v\"\n\n# Fixes for RDMA/InfiniBand issues\nterradev provision -g H100 -n 4 --ensure-rdma --enable-gpudirect\n```\n\n## Contributing\n\nWe welcome contributions! Please see our [Contributing Guide](CONTRIBUTING.md) for details.\n\n## License\n\nApache 2.0.\n\n## Support\n\n- **Issues**: [GitHub Issues](https://github.com/theoddden/Terradev/issues)\n", "bytes": 25969, "sha": "0593d9c677e274dde57476a61a537d64091ea5e2578150ff686799cf505ee06a", "repo_slug": "theoddden/terradev", "fonte": "repo", "truncated": false, "api": "https://agentalog.com/api/listings/mcp_io_github_theoddden_terradev_0e16a547/readme" }