DeepSeek Harness Multimodal

DeepSeek has launched an experimental multimodal vision understanding model, DeepSeek-V4-Flash-Vision-Exp, and simultaneously opened multimodal API services and a free Files API.

Install the latest version before use:

npm install -g @deepseek-ai/dsh@latest

After updating to the latest version, you can see that the model list already includes DeepSeek-V4-Flash-Vision-Exp:

Switch to the vision model, and then we can directly drag images or PPT files into the document to let it look at the content in the images ( Test image download）：

What is multimodal

To understand multimodality, we first need to understand the word "modality." Modality refers to different forms of information representation: text is one modality, and images, audio, and video are each different modalities as well.

In the past, large language models only processed a single modality—text. Everything you sent them had to be converted into text first.

Multimodal models can simultaneously receive images and text, uniformly converting them into token sequences before processing them together.

The key point: images are not "directly seen" by the model. Instead, a vision encoder first cuts them into patches and converts them into vectors, which are then concatenated with text tokens into the same sequence.

Images also consume tokens. One image takes up to 384 tokens—this is exactly the source of the multimodal billing rules mentioned later.

For Agents, multimodality brings a qualitative change, not just a nice-to-have enhancement.

V4-Flash-Vision-Exp: Multimodal vision model now available

DeepSeek-V4-Flash-Vision-Exp is an experimental multimodal vision understanding model, now available on the DeepSeek API platform.

Users can set viamodel='deepseek-v4-flash-vision-exp'Then you can access the model.

Its capability positioning can be summarized in one sentence: text capabilities without compromise, visual capabilities with a major leap.

In pure text capabilities (Agent, reasoning, world knowledge, etc.), it is on par with the official DeepSeek-V4-Flash release.

On Agent Benchmarks requiring visual understanding, it has achieved a significant leap over DeepSeek-V4-Flash, with multimodal Agent capabilities now approaching Opus-4.8.

Evaluation methodology note: For Code Agent text tasks in public benchmark sets, DeepSeek series models are tested using DeepSeek Harness minimal mode as the framework, with max setting, temperature=1.0, topp=0.95.

In the ApexBench and Agents' Last Exam evaluations, the text model DeepSeek-V4-Flash ignores the multimodal elements in them.

Multimodal API Quick Start

The multimodal API supports three calling formats: Chat Completions, Messages, and Responses, making it easy to integrate with various Agent tools.

The three formats have identical capabilities—just choose based on the interface style you're familiar with.

All three formats support mixed image-text input. Images can be passed in three ways: base64 inline, external URL, or Files API.

Method 1: base64 inline

After encoding a local image as a base64 string, write it directly into the request body as a data URL.

Run output:

图片中的文字是 "EXAMPLE"。

Method 2: External URL

Images are deployed on publicly accessible servers, and the request only carries a URL string, keeping the request body very small.

The URL must be a public address accessible by the platform. Internal network addresses and local loopback addresses cannot be fetched.

Both base64 and URL methods support adding an optional detail field in image_url to control image resolution precision.

Setting detail to low when running screenshot-type tasks in batches is the most direct way to control multimodal costs.

Messages and Responses format

Those who already have Anthropic or new-style OpenAI code can switch to DeepSeek multimodal almost seamlessly.

Messages format uses image content blocks, and images are passed through the source field.

The Responses format uses an input array, with input_text for text and input_image for images.

Files API: upload first, then reference

The Files API is now open, and this API is free of charge.

Users can upload images to the platform first, then reference them via file_id in requests.

This has two direct benefits: requests no longer carry large chunks of base64, saving request bandwidth; the same image does not need to be uploaded repeatedly across multiple requests.

Step 1: Upload the image and get the file_id.

After successful upload, the platform returns the file's metadata.

During upload, you can also use the expires_after parameter to specify a validity period, ranging from 1 hour to 30 days. If not passed, the file remains valid permanently.

Step 2: Reference the image using file_id in the multimodal request.

{
  "model": "deepseek-v4-flash-vision-exp",
  "messages": [
    {
      "role": "user",
      "content": [
        {"type": "text", "text": "这个图表的整体趋势是什么？"},
        {"type": "file", "file_id": "file-api-qw3x8v2m6n"}
      ]
    }
  ]
}

Billing instructions and notes

Multimodal requests are billed by token. Images are converted into tokens and then participate in billing.

Image tokens occupy context window space. For multi-image tasks, watch the total consumption: 10 images can occupy up to 3840 tokens.

Images are automatically scaled before entering the model: images smaller than about 384×384 are enlarged, while larger images are shrunk to roughly the equivalent total pixels of 800×800.

Therefore, token consumption only depends on the resized dimensions; a 2000×2000 image and a 5000×5000 image consume the same amount; in multi-image requests, each image is calculated independently under the same rule.

base64 noticeably enlarges the request body. For large images or high-frequency calls, consider using external URLs or the Files API instead.

Multimodal requests also have some hard limits. Exceeding them will directly return a 400 error.

V4-Flash-Vision-Exp is an experimental model. Capabilities and service specifications may change with version iterations. For critical production pipelines, it is recommended to also evaluate official models.

For more complete usage instructions, refer to the official API documentation:API Guide - Image Understanding、API Guide - Files API。

Summary

V4-Flash-Vision-Exp brings multimodal capabilities into DeepSeek's Agent ecosystem with the positioning of "text without compromise, vision with a major leap."

For developers, the cost of getting started is very low: just change one model name, and Agents in DeepSeek Harness can understand screenshots, design mockups, and charts.

It is recommended to first try the three official examples to get a feel, then bring in your own business images and get your first "see the image and do the work" Agent task running.