How to build a bot like our Grok image agent — the anatomy of an agent tool

People ask how we built the “Grok bot” — the little agent you ask for an illustration and it hands one back inline. The honest answer deflates the magic: a bot in an agent platform is two small things — a tool (a function the model can call) and a persona (an agent that knows when to call it). Once you see that split, adding a new capability stops being a project and becomes an afternoon.

Here’s the anatomy, using the image generator as the worked example.

First decision: tool, MCP, or engine?

Not every capability should be a hand-written tool. Pick the cheapest shape that fits.

Shape	Use when	Example
Built-in tool	A capability you implement in code and reuse everywhere	generate_image, web_search
Custom HTTP tool	Calling an external API, no code — just a URL + params in the DB	an internal reporting endpoint
MCP server	The vendor already ships a rich tool server (many tools, auth)	GitHub, Linear via MCP OAuth
Engine	The thing is the executor that runs a whole task	a cloud coding agent

The image generator is a built-in tool: one function, called from anywhere, returning an artifact. That’s the sweet spot for “give the model a new verb.”

The tool is a function with a schema — and the schema is a prompt

A tool is defined twice: a schema the model reads to decide when and how to call it, and an implementation that does the work. The schema matters more than people expect — its description is the only instruction the model gets about your tool.

{
  "type": "function",
  "function": {
    "name": "generate_image",
    "description": "Generate an image from a text prompt. Returns the image inline. Use for illustrations, mockups, concept art, social visuals.",
    "parameters": {
      "type": "object",
      "properties": { "prompt": { "type": "string", "description": "Detailed description of the image to create." } },
      "required": ["prompt"]
    }
  }
}

Rules that pay off: name it as a verb, write the description as when to use it (not what it is), keep parameters few and typed, and say what it returns. A vague description is a tool the model never calls or calls wrong.

The implementation: do the work, return something the model can use

The function itself is ordinary code. The one trick worth internalizing: return a result the model and the UI can both use. Our image tool doesn’t return raw bytes — it saves the image as an artifact and returns a markdown image tag, so the answer renders inline in chat with zero extra plumbing.

def tool_generate_image(prompt):
    # 1. call the provider (Grok Imagine via our proxy)
    img = provider.generate(prompt)
    # 2. persist as an artifact
    aid = save_artifact(img, mime="image/jpeg")
    # 3. return a model- and UI-friendly string
    return "![%s](/files/%s/image.jpg)" % (prompt[:60], aid)

Good tool returns are: short, self-describing, and reference artifacts by URL rather than dumping blobs into the context window.

The tool loop: how a call actually happens

You don’t orchestrate the call — the model does. Your runtime just executes what it asks for and feeds the result back until it stops.

flowchart LR
  U["User: 'draw a fox mascot'"] --> M["Model"]
  M -->|"tool_call: generate_image"| RT["Runtime"]
  RT -->|run tool_generate_image| T["Tool"]
  T -->|"![](/files/…)"| RT
  RT -->|tool result| M
  M -->|final answer + image| U

Register the implementation next to its schema so the runtime can find it by name:

TOOL_FN = { "generate_image": lambda a: tool_generate_image(a.get("prompt", "")), ... }

That’s the entire mechanism. Every “bot” you’ve admired is this loop with different tools bolted on.

From tool to bot: add a persona

A tool anyone can call becomes a bot when you wrap it in an agent — a system prompt that gives it a job, a voice, and the one or two tools it should reach for.

Field	Grok image bot
name / icon	”Grok Artist” 🎨
tools	["generate_image"]
system prompt	”You are a visual artist. When asked for an image, write a vivid, specific prompt and call generate_image. Offer one variation.”
engine / model	the tool-loop driver

Now “Grok Artist” is a thing on the roster you can chat with, hand to a workflow, or trigger on a schedule — all because you added one verb and one persona.

The recipe, generalized

1. Choose the shape — tool, HTTP tool, MCP, or engine.
2. Write the schema — verb name, a when-to-use description, few typed params, a stated return.
3. Implement it — return a short, artifact-referencing string.
4. Register it — schema + function, by name.
5. Wrap it in a persona — an agent that knows when to reach for it.

We shipped an image generator, a document builder, a code sandbox, and a cloud-coding agent through these exact five steps. The platform doesn’t grow by getting bigger; it grows one verb at a time.