Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens

Yiming Qin¹, Bomin Wei², Jiaxin Ge¹, Konstantinos Kallidromitis³,
Stephanie Fu¹, Trevor Darrell¹, XuDong Wang^1*

UC Berkeley¹ UCLA² Panasonic AI Research³

*Corresponding author

Paper arXiv Code HF Model HF Data

Vision–Language Models (VLMs) excel at reasoning in linguistic space but struggle with perceptual understanding that requires dense visual perception, e.g., spatial reasoning and geometric awareness. This limitation stems from the fact that current VLMs have limited mechanisms to capture dense visual information across spatial dimensions. We introduce Chain-of-Visual-Thought (CoVT), a framework that enables VLMs to reason not only in words but also through continuous visual tokens — compact latent representations that encode rich perceptual cues. Within a small budget of roughly 20 tokens, CoVT distills knowledge from lightweight vision experts capturing complementary properties such as 2D appearance, 3D geometry, spatial layout, and edge structure. During training, a VLM equipped with CoVT autoregressively predicts these visual tokens to reconstruct dense supervision signals (e.g., depth, segmentation, edges, and DINO features). At inference, the model reasons directly in the continuous visual-token space, preserving efficiency while optionally decoding dense predictions for interpretability. Evaluated across more than ten diverse perception benchmarks, including CV-Bench, MMVP, RealWorldQA, MMStar, WorldMedQA, and HRBench, integrating CoVT into strong VLMs such as Qwen2.5-VL and LLaVA consistently improves performance by 3% to 16%, demonstrating that compact continuous visual thinking enables more precise, grounded, and interpretable multimodal intelligence.

Method

CoVT generates interleaved discrete (textual) and continuous (visual) thinking tokens first, and then leverages these thinking tokens to condition next-token prediction and reason the final answer.

The training pipeline of CoVT. CoVT first generates the thinking process, containing visual thinking tokens, and then leverages these visual thoughts to condition next-token prediction and reason the final answer. To endow these tokens with perceptual meaning, we align them with lightweight vision experts (SAM, DepthAnything, PIDINet, DINO) on their respective tasks during training. Specifically: SAM uses 8 visual tokens as mask prompts; DepthAnything uses 4 tokens to reconstruct depth; PIDINet uses 4 tokens to reconstruct edges; and DINO uses 4 tokens to match patch-level features. The VLM is finetuned with LoRA and all projection layers are trainable. Note: During inference, dense predictions are decoded only when interpretability is desired; otherwise, reasoning occurs entirely in the latent visual space.

👀 Explore how each type of CoVT token contributes to reasoning

01 · 2D Perception

Segmentation Tokens · SAM

Segmentation tokens capture instance-level masks, equipping VLMs with the 2D perception capability.

8 segmenation tokens are aligned to the SAM decoder-space at the prompt-level.
During training, the decoded masks are supervised by using Hungarian Mathing algorithm, along with the cost function of dice loss and focal loss.

Multiple CoVT Tokens Visualization

Different CoVT tokens contribute complementary cues that enable the model to solve complex perceptual reasoning tasks.

Click different <CoVT Tokens> to view the decoded visualization results.

Question

Describe the scene in the picture in detail, and find out how many clouds are there in the sky.

Answer

<Think> Because the segmentation of the image is <SEGMENT Token>, the depth map of the image is <DEPTH Token>, and the edge map of the image is <EDGE Token>. </Think>

<Answer> The image depicts a charming, handcrafted scene featuring a small doll-like figure standing on a grassy patch. The figure is dressed in an orange outfit with matching boots, and it has a cheerful expression. The doll is holding a stick with a smiling sun at the top, which appears to be made from felt or similar material.
In the background, there are two green felt trees, each with a simple triangular shape and a brown trunk. Above the trees, there are four fluffy white clouds, each adorned with a brown button-like detail. The sky is painted in a soft blue color, providing a serene backdrop for the scene.
The overall composition is whimsical and playful, likely intended to evoke a sense of joy and innocence. The use of bright colors and simple shapes contributes to the overall aesthetic appeal of the image.
To answer the question about the number of clouds: There are four clouds visible in the sky above the trees. </Answer>

Question

Two points are circled on the image, labeled by A and B. Which point is closer to the camera?

Answer

<Think> Because the segmentation of the image is <SEGMENT Token>, the depth map of the image is <DEPTH Token>. <SEGMENT Token> </Think>

<Answer> The answer is the point B. </Answer>

Question

Which object is closer to the camera: the books (red box) or the chair (blue box)?

Answer

<Think> The depth map of the image is <DEPTH Token>, and the edge map of the image is <EDGE Token>. </Think>

<Answer> The books are closer to the camera than the chair. </Answer>

Question

How many white vertical lines are drawn on the ground?

Answer

<Think> Because the segmentation of the image is <SEGMENT Token>, the edge map of the image is <EDGE Token>. </Think>

<Answer> There are five white vertical lines. </Answer>

Token Playground

Drag the slider to view the decoded visualization results.

Original

Visual Token

Original

Visual Token

Original

Visual Token

Single CoVT Token Visualization

Inspect how a single type of CoVT token guides reasoning.

Original

Segment Token

Question

—

Answer

—

Gallery

Click arrows to navigate through reasoning examples.

Question

—

Answer

Experiment Results

The same three result figures are now rendered directly in HTML with semantic tables and tiny benchmark charts.

Main Result

Major Benchmarks

Comparison of CoVT with the baseline and closed-source models across CV-Bench and other vision-centric benchmarks.

3 tokens: +5.5 on CV-Bench 4 tokens: 89.2 on CV depth Best HRBench-8K: 69.9

Setting	Visual tokens				CV-Bench				Other vision-centric benchmarks
Setting	Seg	Depth	DINO	Edge	CVBench	Count	Depth	Dist.	BLINK	RW-QA	MMT	MMStar-P	MMVP	MME-RW	V*	HR_4K	HR_8K
Closed-source Models
Claude-4-Sonnet	-	-	-	-	76.3	62.2	77.7	80.5	39.6	63.7	-	58.8	48.7	-	15.2	32.3	22.7
GPT-4o	-	-	-	-	79.2	65.6	86.7	81.0	63.0	69.7	-	65.2	72.0	-	42.9	50.6	46.7
Qwen2.5-VL-7B	-	-	-	-	74.5	65.0	72.8	75.5	55.7	68.6	61.7	67.1	56.0	60.0	76.4	68.6	64.9
CoVT (1 Visual Token)
Seg only	✓	-	-	-	77.9	66.0	80.8	80.5	57.4	71.1	62.1	68.5	58.7	62.1	79.1	71.9	69.0
Depth only	-	✓	-	-	78.7	65.4	83.2	78.2	56.4	71.5	62.7	69.9	58.7	62.0	79.1	71.9	69.4
DINO only	-	-	✓	-	71.3	64.7	72.3	66.7	55.8	71.5	62.5	67.9	57.3	61.1	77.5	71.0	68.6
CoVT (3 Visual Tokens)
Seg + Depth + DINO	✓	✓	✓	-	80.0	66.2	86.8	82.5	56.0	71.6	62.1	69.2	58.7	63.7	78.0	72.9	69.4
Δ (vs Baseline)	-	-	-	-	+5.5	+1.2	+14.0	+7.0	+0.3	+3.0	+0.4	+2.1	+2.7	+3.7	+1.6	+4.3	+4.5
CoVT (4 Visual Tokens)
Seg + Depth + DINO + Edge	✓	✓	✓	✓	79.8	66.1	89.2	80.5	56.2	71.8	61.9	68.4	56.7	63.3	78.5	72.5	69.9
Δ (vs Baseline)	-	-	-	-	+5.3	+1.1	+16.4	+5.0	+0.5	+3.2	+0.2	+1.3	+0.7	+3.3	+2.1	+3.9	+5.0

Comparison of CoVT with the baseline and closed-source models. CoVT delivers consistent improvements across all vision-centric benchmarks and further reveals that each visual token type contributes most effectively to the tasks related to its rich information.

CoVT vs Aurora

Comparison between CoVT and Aurora based on LLaVA-v1.5-13B.

Depth tokens: 7 wins + 1 tie Seg tokens: BLINK rel. depth +45.2

Model	CV-Bench			BLINK
Model	Count	Depth	Dist.	Count	Obj. Loc.	Rel. Depth	Vis. Corr.	Vis. Sim.
LLaVA	59.3	61.8	50.2	56.7	54.9	52.4	29.7	51.1
Aurora† (depth)	54.9	67.7	52.3	53.3	55.7	62.9	26.2	47.4
CoVT (w/ Depth)	60.7	71.0	52.3	56.7	59.8	75.8	31.4	53.3
Δ (vs Aurora)	+5.8	+3.3	+0.0	+3.4	+4.1	+12.9	+5.2	+5.9
Aurora† (count)	56.0	62.2	47.8	31.7	26.2	24.2	26.7	21.5
CoVT (w/ Seg)	61.9	60.7	51.3	58.3	56.6	69.4	29.7	52.6
Δ (vs Aurora)	+5.9	-1.5	+3.5	+26.6	+30.4	+45.2	+3.0	+31.1

Comparison between CoVT and Aurora based on LLaVA-v1.5-13B. † indicates our reproduced results based on the provided checkpoints.

Non-vision-centric Tasks

Qwen2.5-VL-7B Ours

Average: 66.2 to 67.4 MME-translate: +7.5 WorldMedQA: +5.6

Average

69.5 65.0 61.0

66.2

Base

67.4

Ours

OCRBench

90.5 83.0

88.5

87.1

MME-translate

101.0 87.0

92.5

100.0

MUIRBench

60.5 53.0

58.3

58.2

HallBench

54.0 47.5

52.9

52.7

A-OKVQA

91.5 83.0

88.3

89.2

TaskMeAnything

71.5 65.0

70.2

69.9

WeMATH loose

55.0 46.0

53.1

51.1

WorldMedQA

33.0 21.0

25.7

31.3

Beyond the gains on vision-centric benchmarks, CoVT also achieves slight improvements on non-vision-centric tasks.

BibTeX

@article{qin2025chain,
  title={Chain-of-Visual-Thought: Teaching VLMs to See and Think Better with Continuous Visual Tokens},
  author={Qin, Yiming and Wei, Bomin and Ge, Jiaxin and Kallidromitis, Konstantinos and Fu, Stephanie and Darrell, Trevor and Wang, Xudong},
  journal={arXiv preprint arXiv:2511.19418},
  year={2025}
}