Benchmarks lie about agents. This 19-minute run didn’t.

x# Bonsai 2 27B Under Real Agentic Load — MacBook Pro M4 Max

The core finding

This is a different kind of stress test than a standard benchmark question. Reasoning-focused benchmarks (AIME, GPQA-style physics/math problems) test depth on a single, short exchange. I needed something else: whether a model can stay on-task across a long, compounding, real workflow. Several tool calls deep, tens of thousands of tokens of accumulated context, no single "final answer" to look up. I needed it to be my daily driver. A tool I would actually reach for. That is the test. If it isn't a go-to, it just takes up space.

The result on this run: task coherence held for the full 19-minute (which seemed really long), 8,808-token response - suprisingly, no topic drift, a correct mid-reasoning self-correction, honest flagging of its own uncertainty rather than guessing, a final answer matching the source document's own section codes which I checked. Unfortunately, raw throughput did not hold. It fell to roughly a quarter of the model's clean, shallow-context speed by the end of that same response. The size is great, and it runs on my machine without catching fire, but, and this is a big but, it can't take too long.

These are two separate claims. Coherence and raw speed do not necessarily move together. A shorter test that only measures one of them would have missed half the picture.

1. Hardware and system state

Machine: MacBook Pro, Apple M4 Max, 48GB unified memory
Power: on AC throughout the run reported below (confirmed via pmset -g batt)

Server config:

llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 \
  --host 0.0.0.0 --port 8080 --cache-ram 16384 \
  --cache-type-k q8_0 --cache-type-v q8_0 -np 1

Memory: before vs. after two config fixes

Two real, separate issues showed up earlier in the same session. Both were measurable:

Configuration	Memory behavior observed
Before fix — full FP16 KV cache, default 4-slot pooling (no `-np 1`)	Free memory fell to ~1.2GB during a moderate vision workload; throughput degraded continuously and did not recover; one earlier attempt under this same config crashed outright
After fix — Q8 KV cache + `-np 1`	Memory Used held at 33–38GB (Activity Monitor) across every subsequent test, including the 76,580-token task detailed below; swap 5–7GB; memory pressure indicator stayed green throughout; no fan spin-up reported

I keep seeing posts that say a 27B "runs in 12GB VRAM." Technically true if you only count the weights. This file is 7.2GB. At the model's full 262,144-token window, FP16 KV cache alone is ~16GB by the vendor's own per-token figure (64 KiB/token). That is why free memory fell to 1.2GB and one of my run crashed.

Q8 KV cuts that roughly in half. Separately, launching without -np 1 silently defaults to a 4-slot pooled mode. There is a confirmed upstream bug: each slot advertises the full context window, but the pool is not sized for all four at once. Background work — in this case UI-side title generation — then fights the real conversation for the same under-provisioned space.

2. The task

The actual prompt (short, plain language, no special formatting):

"ok which images would fit the project that I am currently doing?"

What that one sentence triggered, autonomously, with no further user input, across roughly 20 minutes of real wall-clock time:

Enumerate all projects in the document vault
Locate the specific file for the named project
Multiple attempts to read the document's full text — several early attempts failed on a real, previously-undiscovered bug: the source filename contained accented characters, and the model reproduced them slightly wrong each attempt (a missing accent, or an accent substituted with an underscore), compounded by a genuine macOS Unicode normalization mismatch between how the filename is stored on disk (NFD) and how it was being requested (NFC). Both issues were root-caused and fixed in the vault server's code during this session; a later, separate run confirmed the fix resolved it.
A full read of the actual document — 45,715 tokens of real prompt content, not a summary
Multiple searches cross-referencing the new document's requirements against 8 other already-indexed past projects in the vault
5 separate image searches, apparently one per major document section it had identified (dashboards, system screenshots, drawings, equipment photos)
A final, synthesized answer: candidate images grouped into three honest tiers — genuinely usable, superficially similar but wrong (e.g. correctly-formatted dashboards carrying a different project's real figures, explicitly identified and excluded), and confirmed gaps with no matching image anywhere in the vault — each tied to the document's own internal section codes rather than generic categories

3. The numbers

These came straight off llama-server's per-task summary lines, not a stopwatch:

Context @ release	Start (t/s)	End (t/s)	Real avg (t/s)	Tokens generated	Wall time (s)
545	34.95	34.93	34.93	215	7.8
10,330	32.83	32.57	32.55	520	60.0
10,373	32.72	32.72	32.68	143	6.7
10,560	32.62	32.62	32.60	148	5.6
56,622	18.65	17.89	17.82	499	410.9
68,938	17.49	13.31	13.27	3,734	389.8
76,580	7.26	8.26	8.26	8,808	1,147.0
3,542	27.66	25.51	25.52	731	44.0
5,744	28.01	25.49	25.50	2,987	132.5

The 76,580-context row is the task that produced the tiered image answer above. 8,808 tokens over 19 real minutes, whole-task average 8.26 tokens/second — roughly a quarter of the clean, shallow-context speed (34.93 t/s at 545 tokens). Nineteen minutes was almost all generation, not a hung tool loop.

The shape of the decay within the long task

The average hides what I found to be the more interesting part. Speed on that task did not just fall in a straight line, and when it recovered, it didn't recover to anywhere near where I thought it should:

dropped from 7.26 t/s down to a floor around 5.60–5.63 t/s roughly 40% of the way through
then recovered, climbing back up to 8.26 t/s by the end

llama.cpp's "graphs reused" counter climbed the whole time (to nearly 14,000 by the end). The answer kept repeating its own shape - numbered lists, similar per-item phrasing - so graph caching started to earn something back against the cost of a growing context. Two forces, not one clean downward line.

The pattern across the whole session

Ignore the last two rows for a second. Speed tracks how deep this conversation is, almost linearly:

545 tokens → 34.93 t/s
76,580 tokens → 8.26 t/s

Those last two rows looked wrong at first. They ran right after the long task, but at ~25.5 t/s instead of the ~32.6 t/s I had seen at similar depth earlier. Timestamps kill the thermal story: the long task released at 35:31:13, the next conversation started at 35:33:59. Almost no gap. They were new conversations (3,542 and 5,744 tokens of context), not a continuation of the long thread.

So: speed follows the current conversation's context depth. Not the clock, and not leftover heat from the job before it. That rule fits all nine rows.

4. Images as first-class content, not discarded

The job was image retrieval, so the pipeline that made those searches possible is in scope. I also need to know where the images come from - the context in the document that the images represent.

The goal here is not "which model wins a benchmark." It is which tool actually does the work: take a real document (PDF or Word) and get useful information back out of it. Most "chat with your documents" setups only extract text. That misses a lot in real business documents, medical publications, marketing decks. Technical reports carry a large share of their actual information in images: dashboards, system screenshots, drawings, equipment photos. A tool that silently drops all of that is missing the dense part, and quite frankly, serves me no purpose.

The architecture (model-independent)

Every embedded image in an indexed document is detected and extracted automatically at index time
Deduplicated globally by content hash, not per-document — the same logo or diagram appearing across dozens of files gets processed once, not once per occurrence. This was critical. Many documents have logos, headers or footers that are the same across the document. This can become noise for any model/chat.
Each unique image is captioned once, and that caption becomes ordinary, fully searchable content — a semantic search for "dashboard" or "org chart" surfaces real images sitting inside real documents, alongside text results, not a separate disconnected step
Real thumbnail and full-resolution serving, so a result can actually be looked at, not just described.

What actually varied by model — real, measured findings

A dedicated, small vision-specific model captioned reliably at a fixed, modest token budget. Fast. No tuning.
A general-purpose reasoning model (Qwopus) needed its captioning token budget raised repeatedly — 150 → 600 → 1200 — before captions stopped coming back empty on genuinely complex, ambiguous images. Simple screenshots were fine at the smaller budgets. Dense, multi-element marketing graphics were not. Which btw, was based on a subsequent test, but good context for this.
Even at 1200 tokens, 2 specific images never succeeded. Both genuinely ambiguous (a mixed logo/decorative/text graphic), not just a budget problem
- This also turned up a bug worth flagging if you build something similar: an empty caption was being cached as if it were a real success. A failed image would never be retried, and nothing told you it had failed. This failure also could cause the model to consistently try and give me what I wanted, which forced a restart of llama server as well as my tool servers. It was fixed by treating an empty response as a failure, storing the reason, and giving the model a tool to report which images failed and why.
I needed a manual-caption override for the cases a model cannot reliably caption. Supply the text once; it indexes the same as an automatic caption.

The practical takeaway: this is not "some models can do vision, some can't." Every model I tried here could produce real captions. It is a reliability and tuning finding. A reasoning-heavy general model needs more headroom for this job than a small dedicated vision model. Even then, a few genuinely ambiguous images may need a human.

5. MTP (multi-token prediction) — tested, does not work on this model

This build of llama-server supports speculative decoding, including MTP (--spec-type draft-mtp). The feature is supposed to be self-contained — extra prediction-head weights in the same GGUF, no separate draft model. Bonsai's file does carry nextn-named tensors (they show up as "unused tensor" warnings on every load without the flag).

Test result: it fails outright. It does not fall back.

llama-server -m Ternary-Bonsai-2-27B-PQ2_0.gguf -ngl 99 \
  --host 0.0.0.0 --port 8080 --cache-ram 16384 \
  --cache-type-k q8_0 --cache-type-v q8_0 -np 1 --spec-type draft-mtp

common_speculative_init_result: creating MTP draft context against the
target model 'Ternary-Bonsai-2-27B-PQ2_0.gguf'
llama_init_from_model: context type MTP requested but model doesn't
contain MTP layers
common_speculative_init_result: failed to create MTP context
llama_server: exiting due to model loading error

The raw nextn tensor bytes are in the file. llama.cpp still checks for valid MTP layers before it will use them, and that check is tied to metadata the ternary conversion did not produce. Load dies instead of quietly running without MTP. Some other architectures (GLM is the usual example) skip MTP and keep going. This one does not.

If you are building on a limited budget, do not count on MTP as a free speedup on Bonsai 2 27B - well not with this loader and this type of conversion. Every number in Section 3 is the real ceiling. There is no hidden flag I forgot to flip.

Would I use this as my daily driver? The jury is still out. It doesn't heat my machine up like Qwopus3.8:27b, but it also takes 19 minutes to give me a result. That's even too long to make a coffee and come back to results. Maybe going out for a coffee, but not sure it's practical. I also don't play with every new model that comes out. I like the idea of setting it and forgetting it - so that it just works.