The model can't see what it drew

July 30, 2026 · updated August 19, 2026 · growing · certain · 9 min

You ask a chat app for a picture, and a moment later a picture appears. Somewhere in the middle a model decided to call an image tool, the tool called an image API, and a URL came back. Here is the part that took me a while to really see. The model that asked for the picture never looks at it. It cannot. The thing it called produces pixels, and the model reads text. So what, exactly, should that tool hand back to the model when it finishes?

ChatHub’s answer is: almost nothing. The image tool returns an empty string to the model, and it sends the actual image somewhere else entirely, to you. One tool run, finishing once, produces two different outputs aimed at two readers who do not want the same thing. That fork is the mechanism I want to walk through, because once you see it, the ordinary design, one tool returns one value, starts to look like it is quietly assuming there is a single audience. There are two, and they do not read the same language.

The executor is a sealed box

ChatHub runs its tools through LangChain’s AgentExecutor. You give it a model and a list of tools, you call .invoke(), and it runs the whole loop: ask the model, notice it wants a tool, run the tool, feed the result back, ask again, until the model produces a final answer. What comes back to you at the end is a single object with an output string on it. That is all.

Not the image the image tool made. Not the rows a web search returned. Not even which tools ran, or how many times. The executor is a black box that swallows every intermediate artifact and returns only the model’s closing words. This is the right shape for the model’s purposes, because the model only ever needed text in its scratchpad. It is the wrong shape for a user interface, which needs the artifacts the box just ate.

So there is a real problem to solve, and it is not a small one. If you want to show a person the image their request produced, you cannot get it from the executor’s return value. It never comes back out that door. You have to get it out some other way, while the tool is still running, from inside the box.

One run, two readers

The move ChatHub makes is to stop pretending a tool has one output. Look at what the image tool actually does when its API call succeeds:

sendToolResponse({
  toolName: "image_generation",
  toolRenderArgs: { image: result },
});
const searchPrompt = "";
return searchPrompt;

Two statements, two destinations. The return value is what LangChain writes into the agent’s scratchpad, the text the model reads before deciding its next step. It is an empty string, on purpose, because there is nothing useful to say to a reader who cannot see the picture. The sendToolResponse call carries the image URL to the other reader, the interface, where it becomes an actual rendered image. The tool did not compute one result and format it two ways. It chose different content for each reader based on what that reader can perceive.

Web search makes the same fork with different halves. The model gets a block of text: the search rows wrapped in an instruction that tells it to answer the question with proper citations. That instruction is useless to a human and essential to a model that is about to write the reply. The human gets the rows themselves, meant to be drawn as a list of results. And the calculator shows the other end of the spectrum: it returns its number and calls no side channel at all, because a number is legible to both readers and there is no richer artifact to route anywhere. One reader, when one reader is enough.

The thing I keep coming back to is that these are not two views of one payload. They are two payloads, and the difference between them is deliberate. The model is handed what will help it think. You are handed what you came for. Deciding those are different things, per tool, is the entire design.

The side door, and the handshake

That leaves the mechanical question. If the image cannot come back through the executor’s exit, how does sendToolResponse reach the interface from inside the box?

Through a closure. Before the executor ever runs, each tool is built, and at that moment it is handed a function that is closed over React’s state setter. The tool body, which will later run deep inside LangChain’s loop, holds a reference that points back out into the component. It does not return the rich data up through the executor. It reaches sideways, past the box entirely, into state:

sendToolResponse: (arg) => {
  setCurrentTools((tools) =>
    tools.map((t) =>
      t.toolName === arg.toolName ? { ...arg, toolLoading: false } : t
    )
  );
};

But there is a timing problem a single reach cannot solve. The interface should show a spinner the instant a tool starts, long before it has any result to send. So the two sides perform a small handshake. LangChain fires a handleToolStart callback the moment a tool begins, and ChatHub uses it to push a placeholder into the same state, { toolName, toolLoading: true }. The interface sees that entry and renders a spinner with the tool’s loading message. Later, when the tool body finally calls sendToolResponse, the code above finds that placeholder by name and swaps the real render arguments in, flipping toolLoading to false. A spinner is planted on the way in; the artifact fills it on the way out.

The words wrapped around that handshake come from the tool registry, not the tool run. Each registered tool carries a loadingMessage and a resultMessage, so the spinner reads “Searching on Google…” and then, once the result lands, “Results from Google Search,” without the search tool itself having to say either. The registry describes how a tool should look while it waits and after it finishes; the closure only has to supply the payload in between.

All of this happens in ephemeral state that only exists while the message is generating. When generation stops, ChatHub flattens it: the live tool entries are copied onto the message, their loading flags forced off, and the whole message is written to IndexedDB. On reload, the persisted tool entries render through exactly the same path as the live ones, so a conversation you reopen tomorrow replays its images and result lists from the durable transcript rather than rerunning any tool. The side door’s output is not a runtime flourish. It becomes part of the saved record.

Scar tissue

An honest look at this design has to sit with what the shortcuts cost, because the thing that makes it pleasant to write is the same thing that makes it fragile.

Start with how the handshake matches. The placeholder and the result are paired by tool name, the string web_search, and nothing more. LangChain actually hands handleToolStart a unique runId for each invocation, which is exactly the identifier you would use to tell two runs apart. ChatHub receives it and discards it. So when the model fires two web searches in one turn, both placeholders carry the same name, and the matcher, t.toolName === arg.toolName, cannot distinguish them. The results land on the same card. Two distinct searches collapse into one, and the first answer is overwritten by the second.

Then there is what happens when a tool does not succeed. The only two things that ever turn a spinner off are a successful sendToolResponse and the end-of-turn commit that forces every loading flag false. The executor’s handleToolError and handleToolEnd callbacks are both empty. So a tool that fails midway clears nothing on its own: its spinner keeps spinning for the rest of the turn, and if the turn never reaches a clean stop, it spins with no end. The interface has no state that means “this tool tried and failed,” only “loading” and “here is the result,” because the failure path was left as a pair of empty braces.

The interface even carries an alert for a failure that cannot happen. There is a rendered “Recursion detected” message wired to a recursion stop reason, but no code anywhere sets that reason. It is a warning for a case the executor was never configured to detect, which means a genuine tool loop would run without the one message built to explain it, and the message that does exist can never appear.

Underneath all of it is a contract that checks nothing. The tool response type declares its render arguments as any, and each tool’s renderer reads whatever fields it likes off that untyped bag. That looseness is why adding a new tool with its own bespoke card is almost free: you invent a shape and a renderer for it and nobody has to agree on a schema. It is also why the web search tool can populate render arguments that the registry never actually draws, its entry has no renderer wired at all, and why the memory tool’s renderer is a copy of the image tool’s, reaching for an image field it is never sent. Nothing complains, because there is nothing there to complain. The freedom and the gaps are one decision seen from two sides.

What the fork was really about

The lesson I take from this is about the quiet assumption inside the phrase “return value.” A return value is built for one caller. The moment a tool has two consumers with different senses, a model that reads and a person who sees, one returned value cannot honestly serve both, and any attempt to make it do so means shipping the model something it cannot use or shipping the person something they did not want. ChatHub’s answer is to stop trying: let the tool say one thing to the model and hand a different thing to the person, and route the second one around the executor that would otherwise eat it.

The cost of doing that the easy way is written plainly in the same code, in a discarded runId and two empty callbacks and an any that guards nothing. That is the trade, and it is a defensible one for a chat app moving fast. What it is not is invisible.

Give each reader what they can actually read. The bugs live in the places you decided one message would do.