Needle 2 is a 45-million-parameter agentic model that fits in a 14MB binary. That headline is fun, but it is not the useful part. The useful part is the boundary it draws: if your "agent" mostly maps messy sentences to a fixed set of typed actions, a cloud model may be absurd overkill.

Needle 2 architecture diagram for tiny-device tool calling

Cactus says a full session stays around 28MB of RAM and reaches about 500 tokens per second while decoding on a Raspberry Pi 5. That puts local tool routing in reach of devices that cannot host a normal 1B or 3B chatbot. Phones, watches, home controllers, small robots, and microcontrollers are the intended targets. The catch is just as important: Needle 2 emits structured calls and extracted fields. It is not a replacement for a model that needs to explain a tax rule, search a large knowledge base, or hold a broad conversation.

The right way to evaluate it is as a local control plane. Let the tiny model decide whether the user said "turn off the kitchen lights," select the matching function, and fill its arguments. Then keep a larger model behind an escalation path for everything that falls outside the device vocabulary.

Hardware fit

Start with the memory ceiling, not the parameter count. The official release describes a 14MB model binary and a peak session footprint around 28MB. The session uses a bounded 256-token sliding window, so RAM does not grow with an endlessly long conversation. That is a much better fit for an embedded product than a small general chatbot whose cache expands with every turn.

Cactus reports roughly 500 tokens/sec decode on a Raspberry Pi 5. It reports 300 to 700 tokens/sec on sub-$200 Samsung A-Series phones, and 400 to 1,500 tokens/sec on devices such as Meta Quest 3S and Apple Vision Pro. Treat those as vendor measurements, not a promise for your board. The input length, compiler, kernel path, tool list, and thermal state all matter. Still, the order of magnitude changes the product design. A device can parse a voice command locally instead of waiting for a network round trip.

The official repository is the practical starting point. Clone cactus-compute/needle, install its environment, and launch the needle playground command. The playground is designed to test a tool schema and fine-tune it on local examples. For a first experiment, define only the actions your device can safely perform:

[
  {
    "name": "set_light",
    "description": "Set a room light to on or off",
    "parameters": {
      "room": {"type": "string", "required": true},
      "state": {"type": "string", "required": true}
    }
  }
]

Then test ordinary language, missing arguments, irrelevant requests, and deliberately confusing phrasing. The model should return a valid call when the request is in scope and an empty or rejected call when it is not. A device agent that invents a function is worse than one that admits uncertainty.

Fine-tuning is where the tiny footprint becomes useful instead of merely impressive. The repository recommends at least 120 examples per tool, split across training, validation, and test data. That is a manageable dataset for a product team. Capture the language your users actually use, including abbreviations, accents, room nicknames, and incomplete commands. Do not train only on clean examples that sound like API documentation.

The public page reports 63.7% on 961 rows in the Mobile Actions evaluation, using ordered strict exact match. That score requires the function name, call order, and every argument to match. LFM2.5 230M scores 69.1 and FunctionGemma 270M scores 64.0 on the same table. Needle 2 is not winning that general comparison. Its advantage is the deployment envelope: the larger baselines are measured as f16 checkpoints under vLLM, while Needle 2 is measured through its CQ2-bit production engine. The choice is not "which number is highest?" It is whether the extra accuracy is worth hundreds of megabytes of memory and a different runtime.

For a fixed device vocabulary, I would run a local first pass and attach a confidence threshold. Above the threshold, execute only allowlisted actions. Below it, ask a clarifying question or send the request to a larger private or cloud model. Keep that escalation decision outside the model. The device should not be able to grant itself broader permissions because it feels confident.

Failure boundaries

Needle 2 gets less convincing as the task becomes open ended. Cactus says it was trained around consumer device actions, structured extraction, and tool use. That focus helps it beat larger models on some narrow rows, but it also limits what you should ask it to do. A watch command is a good fit. "Read these 80 pages, compare the vendor contracts, and draft a recommendation" is not.

The BFCL results show the same shape. On the 3,641-row single-turn evaluation, Needle 2 reports 42.6 overall and a 93.4 well-formed rate. The output format is usually valid, but correctness falls on categories outside its training focus, especially JavaScript, Java, and parallel multi-call cases. A valid JSON call can still be the wrong call. Schema validity is not task accuracy.

There is also a naming trap. The GitHub repository still contains documentation and metadata that call the earlier project a 26M-parameter Needle model, while the current Cactus page describes Needle 2 as a 45M-parameter model with CQ2-bit compression and a 14MB binary. Do not copy an old benchmark or memory figure into a Needle 2 deployment plan without checking which release it describes. The repository, model page, and benchmark page should be pinned to a revision when you build a product.

The benchmark comparison has another limitation that deserves more attention than the launch post gives it. Needle 2 is specialized and evaluated through its shipped engine, while the baselines carry general language capability and run under a different serving stack. That is a fair answer to a narrow engineering question, namely which system can route device actions under a tight memory budget. It is not evidence that a 45M model replaces a general assistant.

Use a test matrix with four buckets before shipping: valid in-scope commands, missing arguments, irrelevant requests, and multi-step requests. Record exact function accuracy, refusal or escalation accuracy, malformed-call rate, latency, and energy under a sustained workload. The 14MB binary is only half the system. Your tool descriptions, permission checks, retry policy, and escalation path decide whether the product behaves safely.

That is why Needle 2 matters. It does not make a tiny model generally intelligent. It makes a narrow slice of agency cheap enough to put inside the object being controlled. If your product has typed actions, fixed schemas, and a credible fallback, test it. If the product needs open-ended reasoning, keep the larger model and do not pretend a small parser is an agent.

For the security side of that design, NestFrontier's sandbox and approval boundaries are still relevant. Local execution removes a network dependency, not the need to constrain what a tool call can change.

Sources