Text-only agents are just the beginning. The most valuable enterprise workflows are document-heavy, image-rich, or audio-based. Multimodal agents handle all of these input types natively, enabling automation that wasn't possible when agents could only read text.
The four modalities your agents can use
📝
Text
Emails, chat messages, code, structured data — the foundation all agents start with.
🖼️
Images / Screenshots
UI analysis, chart interpretation, product photos, handwritten forms, visual inspection.
📄
Documents (PDF, DOCX)
Contracts, invoices, reports — native parsing preserves layout and table structure.
🎤
Audio / Video
Meeting transcription, call analysis, video frame extraction, spoken-language input.
2026 multimodal model capabilities
| Model | Images | PDFs | Audio | Video |
|---|---|---|---|---|
| Claude Sonnet 4 | ✓ | ✓ | Via transcription | ✗ |
| GPT-4o | ✓ | ✓ | ✓ Native | Frame extraction |
| Gemini 2.5 Pro | ✓ | ✓ | ✓ Native | ✓ Native |
| Llama 3.2 Vision | ✓ | Via parsing | ✗ | ✗ |
Building a multimodal document agent
from moltbot import Agent
from moltbot.inputs import Document, Image
agent = Agent(model="gemini-2.5-pro")
# Process a PDF invoice + screenshot of the product
response = agent.run(
inputs=[
Document("invoice_march_2026.pdf"),
Image("product_photo.jpg"),
],
prompt="""
Extract all line items from the invoice.
Verify the product photo matches item 3 (SKU: X-441).
Flag any discrepancies.
"""
)
High-value multimodal use cases
- Contract review: Upload a PDF contract → agent extracts key terms, flags non-standard clauses, and compares against your standard template.
- Invoice processing: Process PDF + image invoices with native layout understanding — no brittle regex parsing needed.
- Meeting summarization: Record → transcribe → summarize → extract action items. One agent, four steps.
- UI testing: Screenshot-based agents can navigate web UIs, fill forms, and validate that rendered output matches design specs.
- Visual QC: Manufacturing inspection agents process product images to detect defects faster and more consistently than human inspectors.
Multimodal agents on MoltBot
Text, images, PDFs, audio — unified input API across Claude, GPT-4o, and Gemini. 14-day free trial.
Start Free Trial →