Multimodal Payload Estimator

Understanding Multimodal Tokenisation

When you send an image, video clip, or audio file to a vision or multimodal model via API, the provider converts that media into tokens before processing. Unlike text — where 1 token ≈ 4 characters — media tokenisation depends on resolution, aspect ratio, duration, and the provider's internal encoding strategy. The same 1920×1080 image can cost anywhere from 765 to 1,548 tokens depending on which model you use.

Understanding this is critical for budgeting AI applications. A product catalogue analysis pipeline processing 500 product images per day generates very different monthly bills on GPT-4o versus Gemini — often a 3–5× difference — before you even consider the output token costs.

GPT-4o: Tile-Based Image Tokenisation

OpenAI's GPT-4o uses a two-mode system. In low detail mode, every image costs a flat 85 tokens regardless of resolution — useful for applications where fine visual detail is not needed. In high detail mode, the image goes through a three-step process: first scaled to fit within a 2048×2048 bounding box, then rescaled so the shortest side is exactly 768 pixels, then divided into 512×512 tiles. Each tile costs 170 tokens, plus a fixed 85-token base. A typical 1280×720 HD image in high-detail mode produces a 1280×720 → scales to 1365×768 → 3×2 tiles = (6 × 170) + 85 = 1,105 tokens.

For applications sending many moderate-resolution images where exact visual detail matters (e.g. document extraction, UI screenshot analysis), high detail is appropriate. For thumbnail-level classification tasks (e.g. "is there a person in this photo?"), low detail at 85 tokens flat is dramatically cheaper.

Claude 3.5 Sonnet: Pixel-Division Formula

Anthropic uses a simpler but equally effective approach. The image is resized so its longest dimension is at most 1568 pixels, preserving the aspect ratio. The token count is then the ceiling of (width × height) ÷ 750. This linear formula means token costs scale predictably with image area. A 1568×1568 square at maximum size costs ⌈2,457,856 ÷ 750⌉ = 3,278 tokens — but most real-world images at HD or lower are well under 2,000 tokens each.

Claude's formula is transparent and easy to reason about. Unlike tile-based approaches, there's no non-linear step change when crossing tile boundaries. The cap at 1568 pixels also means that sending a 4K image costs the same as sending a 1568px equivalent — Anthropic's preprocessing erases the resolution difference before billing.

Gemini 1.5 Pro: 258-Token Tiles for Images, Video, and Audio

Google's Gemini 1.5 Pro applies a unified 258-token-per-tile model across all media types. For images, the image is divided into 768×768 tiles without prior resizing: tile count = ⌈width ÷ 768⌉ × ⌈height ÷ 768⌉, with a minimum of one tile. This means a 3840×2160 4K image uses ⌈3840÷768⌉ × ⌈2160÷768⌉ = 5 × 3 = 15 tiles = 3,870 tokens — the only model where 4K images incur a meaningfully higher token count than 1080p.

For video, Gemini samples at 1 frame per second (regardless of the original frame rate) and encodes each frame as a single tile at 258 tokens, then adds audio at 32 tokens per second. This makes Gemini the only model of the three with native video file support. A 3-minute product demo video costs (180 × 258) + (180 × 32) = 46,440 + 5,760 = 52,200 tokens, or about $0.065 at $1.25/M — inexpensive enough for high-volume video analysis pipelines.

Audio is encoded at a flat 32 tokens/second. This covers speech, music, ambient sound, and any audio format supported by the Gemini API (MP3, WAV, AAC, FLAC, and others).

Practical Cost Comparison

For a typical HD image (1280×720) sent 10,000 times per month: GPT-4o high detail costs about $17/month, Claude 3.5 Sonnet costs about $18/month, and Gemini 1.5 Pro costs about $6.45/month. For video or audio workloads, Gemini is the clear choice — it's the only model with native support, and its pricing is competitive even for high-volume use.

However, cost alone shouldn't drive model selection. GPT-4o and Claude 3.5 Sonnet often produce higher-quality vision outputs for tasks requiring fine-grained reasoning about image content. For high-stakes document extraction, medical imaging analysis, or UI parsing, the marginal cost difference may be well worth the quality uplift.