--- name: huggingface-local-models-814640db1d5f description: "Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving." --- # Hugging Face Local Models Search the Hugging Face Hub for llama.cpp-compatible GGUF repos, choose the right quant, and launch the model with `llama-cli` or `llama-server`. ## Default Workflow 1. Search the Hub with `apps=llama.cpp`. 2. Open `https://huggingface.co/?local-app=llama.cpp`. 3. Prefer the exact HF local-app snippet and quant recommendation when it is visible. 4. Confirm exact `.gguf` filenames with `https://huggingface.co/api/models//tree/main?recursive=true`. 5. Launch with `llama-cli -hf :` or `llama-server -hf :`. 6. Fall back to `--hf-repo` plus `--hf-file` when the repo uses custom file naming. 7. Convert from Transformers weights only if the repo does not already expose GGUF files. ## Quick Start ### Install llama.cpp ```bash brew install llama.cpp winget install llama.cpp ``` ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp make ``` ### Authenticate for gated repos ```bash hf auth login ``` ### Search the Hub ```text https://huggingface.co/models?apps=llama.cpp&sort=trending https://huggingface.co/models?search=Qwen3.6&apps=llama.cpp&sort=trending https://huggingface.co/models?search=&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending ``` ### Run directly from the Hub ```bash llama-cli -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ### Run an exact GGUF file ```bash llama-server \ --hf-repo unsloth/Qwen3.6-35B-A3B-GGUF \ --hf-file Qwen3.6-35B-A3B-UD-Q4_K_M.gguf \ -c 4096 ``` ### Convert only when no GGUF is available ```bash hf download --local-dir ./model-src python convert_hf_to_gguf.py ./model-src \ --outfile model-f16.gguf \ --outtype f16 llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M ``` ### Smoke test a local server ```bash llama-server -hf unsloth/Qwen3.6-35B-A3B-GGUF:UD-Q4_K_M ``` ```bash curl http://localhost:8080/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer no-key" \ -d '{ "messages": [ {"role": "user", "content": "Write a limerick about exception handling"} ] }' ``` ## Quant Choice - Prefer the exact quant that HF marks as compatible on the `?local-app=llama.cpp` page. - Keep repo-native labels such as `UD-Q4_K_M` instead of normalizing them. - Default to `Q4_K_M` unless the repo page or hardware profile suggests otherwise. - Prefer `Q5_K_M` or `Q6_K` for code or technical workloads when memory allows. - Consider `Q3_K_M`, `Q4_K_S`, or repo-specific `IQ` / `UD-*` variants for tighter RAM or VRAM budgets. - Treat `mmproj-*.gguf` files as projector weights, not the main checkpoint. ## Load References - Read [hub-discovery.md](references/hub-discovery.md) for URL-first workflows, model search, tree API extraction, and command reconstruction. - Read [quantization.md](references/quantization.md) for format tables, model scaling, quality tradeoffs, and `imatrix`. - Read [hardware.md](references/hardware.md) for Metal, CUDA, ROCm, or CPU build and acceleration details. ## Resources - llama.cpp: `https://github.com/ggml-org/llama.cpp` - Hugging Face GGUF + llama.cpp docs: `https://huggingface.co/docs/hub/gguf-llamacpp` - Hugging Face Local Apps docs: `https://huggingface.co/docs/hub/main/local-apps` - Hugging Face Local Agents docs: `https://huggingface.co/docs/hub/agents-local` - GGUF converter Space: `https://huggingface.co/spaces/ggml-org/gguf-my-repo` --- Source attribution: 420company Original project: https://github.com/420company/artemis/blob/336cd3d1f46b17163d76cacccd6de2db489bc93f/skill-packs/cco/analysts/cco-huggingface-skills-huggingface-local-models/SKILL.md License: MIT License information: https://opensource.org/license/mit Original upstream instructions; downloaded from Dave AI. ## Upstream license notice MIT License Copyright (c) 2026 420.COMPANY This software is open source and free for everyone. Fork it. Improve it. Make it yours. Pull requests, ideas, and contributions are always welcome. https://github.com/420company/artemis --- Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Dave AI resource metadata Creator / publisher: 420company Original project: https://github.com/420company/artemis/blob/336cd3d1f46b17163d76cacccd6de2db489bc93f/skill-packs/cco/analysts/cco-huggingface-skills-huggingface-local-models/SKILL.md Attribution basis: Source repository record Fields of work: Development & Engineering, Data & Analytics Roles: ML / AI Engineer, Data Scientist Specialties: API Tools: Llama.Cpp, Hugging Face Country applicability: Country not specified Classification basis: Source category, role and specialization labels Instruction SHA-256: db6ad1bb3d7f784a45c6751f4de1399fff98431e8cdc8cac5f024284b61648ad