Multimodal Inputs

Multimodal Inputs

Every decision endpoint accepts multiple input modalities. The decision is made over whatever you attach.

Text (state)

The primary text context. Can be empty when other modalities carry the content.

{ "state": "Customer says: I was double-charged.", "question": "Is this urgent?" }

Images (images)

Provide images by URL or base64. Up to 16 images, each ≤ 25 MB and ≤ 50 megapixels.

{
  "question": "Does this show damage?",
  "images": [{ "url": "https://example.com/photo.jpg" }]
}

Or inline:

{
  "images": [{ "b64": "<base64 bytes>", "mime": "image/png" }]
}

Documents (documents)

Attach PDF, DOCX, PPTX, XLSX, JSON, CSV, TXT, Markdown, HTML, or source code. The server extracts text and folds it into context.

Limits: 25 MB per file, 16 documents per request, 48,000 extracted characters total.

{
  "question": "Does this contract have an auto-renewal clause?",
  "documents": [{ "url": "https://example.com/contract.pdf" }]
}

Each document’s extraction status is returned in document_notes:

{ "document_notes": [{ "document": "contract.pdf", "status": "included", "chars": 12400 }] }

Audio (audio)

Attach audio clips by URL or base64. Supported formats: WAV, MP3, OGG, FLAC. Auto-resampled to 16 kHz.

{
  "question": "Which word was spoken?",
  "choices": ["go", "stop", "left", "right"],
  "audio": [{ "url": "https://example.com/clip.wav" }]
}

Signals (signals)

Raw numeric signals for data-science decisions via /ds. The server profiles the signal before deciding.

{
  "signals": [{ "name": "temperature", "data": [[72.1, 73.5, 89.2, 91.0]], "fs": 1 }],
  "question": "Is there an anomaly?",
  "kind": "boolean"
}

Combining modalities

You can mix modalities freely — for example, text context with images and documents in the same request. The model sees all of them when making its decision.