Build an AI design editor

A textbox and an attachment box in, one design ID out — and a chatbox to keep editing.

This guide builds the core loop of an AI design editor: the user types what they want, optionally attaches files, hits submit — and gets a design they can keep editing conversationally. The entire UX surface is a textbox plus an attachment box, and the entire state you track is a single design ID.

Each further message continues from the latest design ID (the dashed edge) — a session is just a chain of design IDs.

Submit: user words + attachments

When the user submits, upload each attachment, then create the design. Relay the user's words in user_prompt and keep the task focused on scope:

submit.ts

async function submit(userText: string, attachments: File[], designId?: string) {
  // 1. Upload attachments — these become references
  const references = [];
  for (const file of attachments) {
    const form = new FormData();
    form.append("file", file);
    const uploaded = await fetch("/api/upload", { method: "POST", body: form }).then((r) => r.json());
    references.push({ type: "file", id: uploaded.id });
  }

  // 2. Create (or continue) the design
  const res = await fetch("/api/design?stream=true", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({
      // First turn: create. Later turns: apply the user's request.
      task: designId
        ? "Apply the user's requested changes to the design. Do not change anything else."
        : "Create a design for the user's request. Use the provided files as the user intends.",
      user_prompt: userText, // the user's words, interpreted freely
      references,
      ...(designId ? { continue_from: designId } : {}),
    }),
  });

  return res; // an SSE stream — see "Streaming the conversation" below
}

The browser code above talks to two tiny routes on your own server — /api/upload and /api/design. Their only job is to hold your API key server-side and forward requests (piping SSE streams straight through), so rmk_live_… never ships to the browser:

app/api/upload/route.ts

export async function POST(req: Request) {
  const upstream = await fetch("https://api.rivermark.ai/upload", {
    method: "POST",
    headers: { Authorization: `Bearer ${process.env.RIVERMARK_API_KEY}` },
    body: await req.formData(), // forwards the multipart body
  });

  return new Response(upstream.body, {
    status: upstream.status,
    headers: { "Content-Type": "application/json" },
  });
}

app/api/design/route.ts

export async function POST(req: Request) {
  // forwards the query string too, so ?stream=true comes through
  const upstream = await fetch("https://api.rivermark.ai/design" + new URL(req.url).search, {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.RIVERMARK_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: await req.text(),
  });

  // pass JSON — or the SSE stream — straight through
  return new Response(upstream.body, {
    status: upstream.status,
    headers: {
      "Content-Type": upstream.headers.get("content-type") ?? "application/json",
      "Cache-Control": "no-cache",
    },
  });
}

In production, add whatever auth you already use to these routes — anyone who can reach them can spend your credits.

The completed event carries the new design_id — that's your session state. Store it next to the conversation; on the next submit, pass it as continue_from. Every turn forks the design, so "undo" is just remembering an earlier ID.

Why `task` + `user_prompt`

Users say things like "can the headline be smaller? also more blue" — vague, compound, casual. If that text went into task, it would be followed literally and exhaustively. In user_prompt, the agent weighs it: it will find the headline, make it smaller, pull the palette towards blue — and use judgment on the rest, without wrecking the design chasing ambiguity.

The fixed task is your guardrail: "apply the user's requested changes; do not change anything else" keeps scope tight no matter how the user phrases things.

Streaming the conversation

Design tasks take seconds. Stream them: progress messages work as the agent's "typing" indicator, and preview events give you working drafts — keyed by id, so a new preview for the same frame replaces the old one. Users perceive the editor as thinking and working — because it is.

TypeScript — wiring a turn into UI state

const previews = new Map<string, string>(); // frame id -> preview URL

for await (const { type, data } of readSSE(await submit(text, files, designId))) {
  if (type === "progress") setTypingIndicator(data.message);
  else if (type === "preview") previews.set(data.id, data.url);
  else if (type === "completed") {
    setDesignId(data.design_id); // advance the session
    setOutputs(data.output.map((o: { url: string }) => o.url));
  } else if (type === "error") showError(data.message);
}

See Streaming for the SSE parsing itself.

A complete React component

Everything above, as a minimal but production-shaped editor:

app/editor/DesignEditor.tsx

"use client";

import { useRef, useState } from "react";

export function DesignEditor() {
  const [designId, setDesignId] = useState<string | null>(null); // your only session state
  const [status, setStatus] = useState<string | null>(null);
  const [images, setImages] = useState<string[]>([]);
  const [text, setText] = useState("");
  const [files, setFiles] = useState<File[]>([]);
  const [busy, setBusy] = useState(false);
  const fileInput = useRef<HTMLInputElement>(null);

  async function onSubmit() {
    setBusy(true);
    setStatus(null);
    try {
      const references = [];
      for (const file of files) {
        const form = new FormData();
        form.append("file", file);
        const uploaded = await fetch("/api/upload", { method: "POST", body: form }).then((r) => r.json());
        references.push({ type: "file", id: uploaded.id });
      }

      const res = await fetch("/api/design?stream=true", {
        method: "POST",
        headers: { "Content-Type": "application/json" },
        body: JSON.stringify({
          task: designId
            ? "Apply the user's requested changes to the design. Do not change anything else."
            : "Create a design for the user's request. Use the provided files as the user intends.",
          user_prompt: text,
          references,
          ...(designId ? { continue_from: designId } : {}),
        }),
      });

      for await (const { type, data } of readSSE(res)) {
        if (type === "progress") setStatus(data.message);
        else if (type === "preview") setImages([data.url]);
        else if (type === "completed") {
          setDesignId(data.design_id);
          setImages(data.output.map((o: { url: string }) => o.url));
          setText("");
          setFiles([]);
        } else if (type === "error") setStatus(`Error: ${data.message}`);
      }
    } finally {
      setBusy(false);
    }
  }

  return (
    <div className="editor">
      {images.map((url) => (
        <img key={url} src={url} alt="Design preview" />
      ))}
      {status && <p role="status">{status}</p>}

      <textarea
        value={text}
        onChange={(e) => setText(e.target.value)}
        placeholder={designId ? "Ask for changes…" : "Describe what you want…"}
        disabled={busy}
      />
      <input ref={fileInput} type="file" multiple hidden onChange={(e) => setFiles([...e.target.files ?? []])} />
      <button onClick={() => fileInput.current?.click()}>
        📎 {files.length ? `${files.length} file(s) attached` : "Attach files"}
      </button>
      <button onClick={onSubmit} disabled={busy || !text.trim()}>
        {busy ? "Working…" : designId ? "Send" : "Create"}
      </button>
    </div>
  );
}

// minimal SSE reader for a fetch Response
async function* readSSE(res: Response) {
  const reader = res.body!.getReader();
  const decoder = new TextDecoder();
  let buffer = "";
  let event = "";
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    // Keep an incomplete line buffered until the next network chunk.
    let newline;
    while ((newline = buffer.indexOf("\n")) !== -1) {
      const line = buffer.slice(0, newline).replace(/\r$/, "");
      buffer = buffer.slice(newline + 1);
      if (line.startsWith("event: ")) event = line.slice(7);
      else if (line.startsWith("data: ")) yield { type: event, data: JSON.parse(line.slice(6)) };
    }
  }
}

What users can do with it

Because the agent sees the full session history, not just the latest frame, edits compose naturally over many turns:

Precise changes — "make the title 20% smaller", "swap the background photo for the second reference"
Creative direction — "make it feel more like nighttime", "less corporate, more playful"
Structural edits — "move the ticket info to the bottom", "turn this into a 3-slide carousel"
New inputs — attach a file mid-conversation (upload it, add to references for that turn)

Design decisions worth stealing

One ID, not a document model. You never handle layers, pages, or assets — just a design ID per session. Rivermark is the document model.

task guards scope, user_prompt carries intent. The fixed task keeps a chatty user from derailing a design, while their words still steer the edits.

Attachments are just references. The same box that accepts a logo on turn one can accept a new product photo on turn five — the agent figures out what it's for from the conversation.

Let users branch. Since every turn forks, "try a different direction" is continue_from an earlier ID — offer it as a freeform undo/history.