ref:7ffcb3e717da08d5bf688cf8aaeed3bf78d3d73f

feat(runner): advertise local models and their capabilities on heartbeat

The routed model picker in Anvil is populated by the runner, not the server: `Anvil.Agents.Inference.list_inference_models/1` unions `metadata["inference_models"]` across the online inference runners in scope. The heartbeat only ever sent `parallel` and `slots_busy`, so that union was always empty and the picker rendered the configured model as unchangeable text — indistinguishable from a provider that cannot enumerate. The server side has been documented since the feature landed (`docs/agents.md`, "Routed discovery has a runner-side half"); this is that half. Each entry carries the model name and the capabilities Ollama reports for it: {"name": "llama3.2:latest", "capabilities": ["completion", "tools"]} Anvil filters on these. An agent chat always sends a tool set, so a model without `tools` fails on its first call — offering it is offering a guaranteed error. A model whose capabilities Ollama does not report is sent with the key omitted rather than an empty list: the server treats that as *unknown* and keeps the model, which is the right behaviour when nothing is known. A bare string remains a valid entry server-side, so a fleet running an older runner keeps working. Best-effort by design. Ollama being down or slow must never cost us a heartbeat, so a failed lookup omits the key and the heartbeat goes out regardless; the previously reported list stays in place server-side, where reporting an empty list would retract it. The lookup has its own 3s timeout, well inside the heartbeat interval. Verified end to end against a local Anvil dev instance: the picker lists the local Ollama's models grouped by provider, selecting one routes the run through this runner, and a model reporting only `["completion"]` is correctly hidden when the chat requires `tools`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SHA: 7ffcb3e717da08d5bf688cf8aaeed3bf78d3d73f
Author: Cole Christensen <cole.christensen@gmail.com>
Date: 2026-08-05 03:58
Parents: 2f5d136
1 files changed +73 -1
Type
src/runner/heartbeat.rs +73 −1
@@ -12,5 +12,6 @@
let auth = config.auth_header();
let interval_ms = config.heartbeat_interval_ms;
let parallel = config.parallel;
let ollama_url = config.ollama_url.clone();
let client = reqwest::Client::new();
@@ -21,11 +22,20 @@
loop {
interval.tick().await;
// Advertise what this host can actually serve. The server unions
// this across the online runners in scope to populate the chat's
let body = serde_json::json!({
// model picker (`Anvil.Agents.Inference.list_inference_models/1`);
// a runner that reports nothing contributes nothing, which is why
// the picker was permanently empty on routed transport.
let mut body = serde_json::json!({
"parallel": parallel,
"slots_busy": slots.busy(),
});
if let Some(models) = local_models(&client, &ollama_url).await {
body["metadata"] = serde_json::json!({ "inference_models": models });
}
match client
.post(&url)
.header(AUTHORIZATION, HeaderValue::from_str(&auth).unwrap())
@@ -65,5 +75,67 @@
Ok(())
} else {
Err(format!("heartbeat failed: {}", resp.status()).into())
}
}
/// What the local Ollama can serve, for the heartbeat's `metadata`.
///
/// Each entry carries the model name and the capabilities Ollama reports for
/// it (`"completion"`, `"tools"`, `"thinking"`, …). The server filters on
/// these: an agent chat always sends a tool set, so a model without `tools`
/// fails on its first call, and offering it in the picker is offering a
/// guaranteed error.
///
/// A model whose capabilities Ollama does not report is sent with the key
/// omitted rather than an empty list — the server treats that as *unknown*
/// and keeps the model, which is what a caller filtering on a capability
/// should do when nothing is known.
///
/// Best-effort: Ollama being down or slow must never cost us a heartbeat, so a
/// failure returns `None` and the heartbeat goes out without the key rather
/// than not going out at all. Omitting it leaves the previously reported list
/// in place server-side; reporting an empty list would retract it.
async fn local_models(
client: &reqwest::Client,
ollama_url: &str,
) -> Option<Vec<serde_json::Value>> {
let url = format!("{}/api/tags", ollama_url.trim_end_matches('/'));
let resp = client
.get(&url)
.timeout(std::time::Duration::from_secs(3))
.send()
.await
.ok()?;
if !resp.status().is_success() {
return None;
}
let body: serde_json::Value = resp.json().await.ok()?;
let models: Vec<serde_json::Value> = body
.get("models")?
.as_array()?
.iter()
.filter_map(|m| {
let name = m.get("name")?.as_str()?;
let mut entry = serde_json::json!({ "name": name });
if let Some(caps) = m.get("capabilities").and_then(|c| c.as_array()) {
let caps: Vec<&str> = caps.iter().filter_map(|c| c.as_str()).collect();
if !caps.is_empty() {
entry["capabilities"] = serde_json::json!(caps);
}
}
Some(entry)
})
.collect();
if models.is_empty() {
None
} else {
Some(models)
}
}