Delegated inference, or: your phone is not the boundary

The objection to running AI on your phone is always the same, and it is a good objection.

A phone has eight to sixteen gigabytes of memory, shared with the operating system and everything else you have open. A model that fits into what remains has been compressed and quantized down to something a laptop would consider a rounding error. Meanwhile there is a machine in Virginia with eighty gigabytes of dedicated GPU memory that will answer you in four hundred milliseconds, and it is free at the point of use.

Local-first advocates usually respond by arguing that small models are better than you think, which is true and does not settle the argument, or by appealing to privacy, which loses to convenience as reliably as it always has.

QVAC’s response is different, and I think it is the most interesting idea in the platform. It is to observe that the constraint is not “the phone.” It is “the phone, considered alone.” And then to stop considering it alone.

The mechanism

Your phone and your laptop are both yours. They are frequently in the same room, on the same network. One of them has a GPU and thirty-two gigabytes of memory sitting idle. The other is the one in your hand.

Delegated inference lets the phone run a model on the laptop. Not through a server you deploy, or an account you create, or an API gateway you configure. Through a direct encrypted connection between two devices that find each other over a distributed hash table.

In the SDK it is a configuration option:

loadModel({
  modelSrc: "...",
  modelType: "llm",
  delegate: {
    topic,
    providerPublicKey,
    fallbackToLocal: true,
  },
})

The rest of your code does not change. You call completion exactly as before. Somewhere underneath, the tokens are being generated on a different machine, and the result is streaming back over a peer-to-peer connection.

If the laptop is asleep, or you left the house, fallbackToLocal runs the model on the phone instead. The application does not need to know which happened.

Why nobody else has this

Because it requires having already built a peer-to-peer network, and building one is a decade of work that has nothing to do with machine learning.

QVAC runs on Holepunch’s stack — Hyperswarm for peer discovery, Hypercore for append-only logs, Hyperdrive for file distribution. Tether owns Holepunch. The same infrastructure carries Keet’s video calls and distributes the Pear runtime’s applications. It is battle-tested by a messaging product with real users, and it long predates the AI work.

So when Tether’s AI team needed two devices to find each other and stream tokens between them without a broker, the answer was a library the company already owned. Discovery happens over a DHT. Peers connect directly, with hole punching through NATs where required, and everything is encrypted end to end. There is no rendezvous server, which means there is no rendezvous server to subpoena, to bill you for, or to go down.

For anyone else to ship this feature, they would need to build that network first, or depend on someone else’s — at which point they have reintroduced the thing local-first was meant to eliminate.

The model registry works the same way. It is a Hyperdrive over Hyperswarm rather than a bucket behind a CDN. You can host your own registry and serve your own models, and the weights arrive from other people’s devices. Nobody is paying egress.

What it actually changes

The obvious reading is that this makes on-device AI more capable, which it does. Your phone can now run a model sized for your laptop. If your laptop is a reasonably specified machine, that is a substantial jump — from models that fit in a few gigabytes to models that fit in thirty.

The more interesting reading is that it dissolves a distinction people had stopped questioning.

We talk about “the edge” and “the cloud” as though they name two places. But the actual property that matters is not proximity. It is control. A model running on a rented GPU three miles away is not yours in any sense that matters; a model running on your laptop from another room is. The relevant boundary was never the device. It was the set of machines you own.

Once you see it that way, other things follow. A household has several computers, and one of them is usually plugged in and doing nothing. A small office has a machine under a desk that is far more capable than the laptops using it. Delegated inference is the mechanism by which those become AI infrastructure, without any of them becoming a server, and without anyone administering anything.

There is a longer-range version of this that Tether talks about more openly than I would: a peer-to-peer market in idle compute, in which strangers’ machines run inference for each other and are paid for it, presumably in a currency Tether happens to issue. Blind relays exist in the SDK today, which is the routing primitive such a thing would need.

I would treat that as a roadmap rather than a product, and I would note that inference on a stranger’s hardware surrenders precisely the guarantee that made local inference worth having. Your prompt is now on somebody else’s machine, encrypted in transit and plaintext at the point of computation. The trust model there is not the trust model of the feature described above, and running them together under one name would be a mistake.

The version that works today is narrower and better: your devices, your network, nobody else in the loop.

The parts to be careful about

Latency over a local network is not latency within a device, and a peer connection that has to hole-punch through two NATs is not instant. For streaming token generation this is mostly hidden by the fact that you are reading the output as it arrives. For anything interactive and short, it will be felt.

The provider device has to be awake, unlocked in the relevant sense, and running the provider. This is a background process on your laptop that you will forget you installed and will one day wonder why your fans are loud.

And the security model deserves more scrutiny than it has received. The connection is encrypted, the peer is identified by public key, and you are expected to know which key belongs to your own laptop. Fine. The question is what happens when that assumption is wrong — when a providerPublicKey arrives from somewhere you did not verify, or a topic is joined by something that is not the machine you meant. Tether’s documentation covers the happy path. The failure modes are where peer-to-peer systems have historically been interesting, and I have not seen a published threat model.

None of these are reasons not to use it. They are the ordinary caveats of an alpha feature in a young platform, and I would rather they were stated than discovered.

Why it is the thing to watch

Most of QVAC is convenience. A unified API over llama.cpp and whisper.cpp and stable-diffusion.cpp is genuinely useful, and several other projects offer something similar. If that were all it was, the platform would be a good piece of engineering with no particular claim on anyone’s attention.

Delegated inference is the part with no equivalent. It is the answer to the strongest argument against the entire local-first premise, it falls directly out of infrastructure the company already owned, and it reframes what “on-device” is supposed to mean in a way that survives the reframing.

Whether Tether builds the compute market on top of it is a question about Tether. Whether the underlying idea is correct is a separate question, and I think it plainly is. The boundary was never the phone.

QV.AC, Tether’s own flagship app, is where this ships today — the peer-to-peer sync described above is the same mechanism.