feat(importer): expand importer flow to almost all backends by mudler · Pull Request #9466 · mudler/LocalAI

mudler · 2026-04-21T09:14:56Z

Description

This pull request adds all missing importers to the core/gallery/importers package, improves the documentation on backend importer integration, and introduces comprehensive table-driven tests for ambiguous model detection cases across several modalities. These changes enhance the /import-model experience by supporting more model types and making ambiguous detection scenarios explicit and testable.

Document the importer registry workflow so contributors know that adding a new backend also requires updating the /import-model dropdown source: either a new importer in core/gallery/importers/, extending an existing one for drop-in replacements, or the pref-only slice for backends with no reliable auto-detect signal. Always covered by a table-driven test. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Introduce failing tests that drive Batch 0 of the importer expansion: - pkg/huggingface-api: assert GetModelDetails populates PipelineTag and LibraryName from /api/models/{repo}, and that a failing metadata endpoint still returns file details (best-effort fetch). - core/gallery/importers/helpers_test.go: new table-driven coverage for HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile. - core/gallery/importers/importers_test.go: assert ErrAmbiguousImport sentinel exists and round-trips through errors.Is. - core/gallery/importers/local_test.go: extend with detection cases for ggml-*.bin (whisper), silero_vad.onnx (silero-vad), and the piper .onnx + .onnx.json pair. - core/http/endpoints/localai/import_model_test.go: assert ImportModelURIEndpoint returns HTTP 400 with a structured {error, detail, hint} body when ErrAmbiguousImport surfaces. All tests fail in the expected places (missing fields, missing helpers, missing sentinel, endpoint still wraps as 500). Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

…l detection Implements the Batch 0 primitives that subsequent importer batches build on: - pkg/huggingface-api: ModelDetails gains PipelineTag and LibraryName. GetModelDetails now layers a best-effort GET /api/models/{repo} fetch on top of ListFiles — a metadata outage leaves the fields empty but still returns full file details. Uses a dedicated response struct because the single-model endpoint uses snake_case keys while the list endpoint historically returned camelCase. - core/gallery/importers/helpers.go: generic HasFile, HasExtension, HasONNX, HasONNXConfigPair, HasGGMLFile helpers working on []hfapi.ModelFile so per-backend importers can detect artefact patterns without duplicating string wrangling. - core/gallery/importers/importers.go: adds the ErrAmbiguousImport sentinel. DiscoverModelConfig now returns it (wrapped with fmt.Errorf("%w: ...")) when no importer matched AND the HF pipeline_tag falls in a whitelist of narrow modalities (ASR, TTS, sentence-similarity, text-classification, object-detection). The whitelist is intentionally narrow — unknown tags keep the previous "no importer matched" behaviour to avoid blocking rare repos. - core/gallery/importers/local.go: three new local-path detections, inserted before the existing merged-transformers branch: * ggml-*.bin → whisper * silero*.onnx → silero-vad * *.onnx + *.onnx.json pair → piper - core/http/endpoints/localai/import_model.go: ImportModelURIEndpoint surfaces ErrAmbiguousImport as HTTP 400 with {error, detail, hint} JSON, preserving existing behaviour for unrelated errors. Green tests: go test ./core/gallery/importers/... ./pkg/huggingface-api/... \ ./core/http/endpoints/localai/... Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

…adata Add failing tests that drive Batch UI-Dropdown: - importers_test.go: assert importers expose Name/Modality/AutoDetects and that LlamaCPPImporter advertises drop-in replacements via a new AdditionalBackendsProvider interface. A Registry() accessor is also expected. - backend_test.go (new): assert GET /backends/known returns []schema.KnownBackend, covers every importer, exposes drop-in llama-cpp replacements, includes curated pref-only backends, has no duplicates, and is sorted by Modality+Name. These tests fail at compile time against master; they are intentionally red so the follow-up green commit is reviewable. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

…d list Extend the Importer interface with Name/Modality/AutoDetects so the import system can self-describe its registry, and introduce the AdditionalBackendsProvider interface so importers can advertise drop-in replacements (llama-cpp advertises ik-llama-cpp and turboquant). Expose the new GET /backends/known endpoint that merges: - the importer registry (auto-detect supported), - drop-in replacements hosted by importers (preference-only), - a curated knownPrefOnlyBackends slice for backends with no dedicated importer (sglang, tinygrad, trl, mlx-vlm, whisperx, kokoros, Qwen TTS variants, sam3-cpp) — kept at the top of backend.go so contributors adding a new pref-only backend have one obvious place to edit, - backends installed on disk but unknown to the importer (marked AutoDetect=false, empty Modality). The endpoint deliberately does NOT filter by gallery membership or host capability (unlike /backends/available): LocalAI may auto-install a backend that is not yet present, so the import form dropdown must show everything the importer knows about. Response is deduplicated (importer wins over pref-only) and sorted by Modality+Name for deterministic output. Registered in core/http/routes/localai.go next to /backends/available under the same admin middleware. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Replace the hard-coded BACKENDS constant in ImportModel.jsx with a live fetch of /backends/known on mount. Users now see every backend the importer layer knows about (including preference-only entries) grouped by modality, not a stale subset. Changes: - config.js: add backendsKnown endpoint constant next to backendsAvailable. - api.js: add backendsApi.listKnown() wrapper. - ImportModel.jsx: remove BACKENDS constant, fetch the list via useEffect, and derive grouped options via buildBackendOptions. Preference-only entries render with a " (preference-only)" suffix. Loading state disables the dropdown with a "Loading backends…" placeholder; on fetch failure the form falls back to auto-detect only and surfaces a non-blocking toast. - SearchableSelect.jsx: accept items flagged isHeader=true and render them as non-selectable section dividers. Keyboard navigation skips headers and search queries hide them so filtered output stays relevant. Vitest is not set up in this project (devDependencies ship Playwright only). Per the brief's guard-rail, no frontend test framework is introduced; coverage is provided by the Go handler tests that assert the /backends/known contract consumed by the React form. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Asserts detection on ggerganov/whisper.cpp (via ggml-*.bin filename), the preferences.backend=whisper override path for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Recognises whisper.cpp GGML models by the "ggml-*.bin" filename convention (direct URL or HF repo member) and by the explicit preferences.backend="whisper" override. Emits backend: whisper with the transcript use-case. Registered before llama-cpp so the narrow filename signal wins before any generic GGUF match is attempted. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Asserts detection on UsefulSensors/moonshine-tiny via owner + ONNX files, the preferences.backend=moonshine override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Matches UsefulSensors-owned HF repos whose artefacts or metadata identify them as ASR: on-disk .onnx files (the canonical Moonshine packaging) OR pipeline_tag=automatic-speech-recognition (covers transformers/safetensors-only sibling repos). preferences.backend= moonshine overrides detection. Test uses the live moonshine-tiny repo because the canonical UsefulSensors/moonshine repo currently hits a recursive-subfolder bug in pkg/huggingface-api ListFiles. Registered after WhisperImporter but before LlamaCPPImporter and TransformersImporter so the narrower owner+ASR signal wins before the generic tokenizer.json check routes the repo to transformers. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Asserts detection on nvidia/parakeet-tdt-0.6b-v3 via owner + .nemo file, the preferences.backend=nemo override for arbitrary URIs, and the Importer interface metadata (name/modality/autodetect). Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Matches nvidia-owned HF repos that ship a .nemo checkpoint archive, the canonical NeMo ASR packaging. preferences.backend=nemo forces detection. Registered between moonshine and llama-cpp so the narrow owner + extension signal wins before any downstream generic matcher. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Asserts detection on Systran/faster-whisper-large-v3 (owner + model.bin + config.json + ASR pipeline), the preferences.backend= faster-whisper override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Recognises CTranslate2-packaged whisper checkpoints distributed for the faster-whisper runtime: model.bin + config.json + ASR pipeline_tag, narrowed to Systran-owned repos or repo names containing "faster-whisper" to avoid falsely claiming vanilla OpenAI whisper HF repos. preferences.backend=faster-whisper overrides detection. Registered before llama-cpp and transformers so the narrow signal wins before tokenizer.json routes the repo to the generic transformers importer. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Asserts detection on Qwen/Qwen3-ASR-1.7B via owner + ASR substring in the repo name, the preferences.backend=qwen-asr override for arbitrary URIs, and the Importer interface metadata. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Matches Qwen-owned HF repos whose name contains "ASR" (case-insensitive), routing them to the qwen-asr backend rather than the generic transformers/vllm path. The substring check scans the repo portion only so the owner field cannot leak a false match. preferences.backend=qwen-asr forces detection. Registered before llama-cpp and transformers so the narrow owner+name signal wins. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]

Locks in the behaviour added in Batch 0: an HF repo whose pipeline_tag marks it as automatic-speech-recognition but whose artefacts match no ASR importer (and no generic importer) must fail with ErrAmbiguousImport so callers know to pass preferences.backend rather than silently guess. pyannote/voice-activity-detection is the fixture — its file list is only config.yaml + README, leaving every importer's artefact check negative. Assisted-by: Claude:claude-opus-4-7[1m] [Agent]