LensVLM-9B by Apple

以下正文同步自 Hacker News，版权归原站所有，已转换为易读排版。

Papers arxiv:2605.07019

Copy markdown

LensVLM: Selective Context Expansion for Compressed Visual Representation of Text

Published on May 7

Upvote 3

Authors: Roy Xie ,

Dan Friedman ,

Donghan Yu ,

Bowen Pan ,

Christopher Fifty ,

Jang-Hyun Kim ,

Xianzhi Du ,

Zhe Gan ,

Vivek Rathod ,

Bhuwan Dhingra

Abstract

Vision-Language Models can process text as rendered images, but accuracy degrades with compression; LensVLM addresses this by scanning compressed images and selectively expanding relevant parts through learned tools, maintaining high accuracy even at high compression ratios.

Generated by Qwen/Qwen2.5-Coder-32B-Instruct

Vision Language Models (VLMs) offer the exciting possibility of processing text as rendered images, bypassing the need for tokenizing the text into long token sequences. Since VLM image encoders map fixed-size images to a fixed number of visual tokens, varying rendering resolution provides a fine-grained compression knob. However, accuracy deteriorates quickly as compression increases: characters shrink below the vision encoder's effective resolution, making them indistinguishable. To address this, we propose LensVLM, an inference framework and post-training recipe that enables VLMs to scan compressed images, then selectively expand only the relevant images to their uncompressed form via learned tools. Building on Qwen3.5-9B-Base, LensVLM maintains accuracy comparable to the full-text upper bound at 4.3x effective compression and outperforms retrieval-based, text- and visual-compression baselines up to 10.1x effective compression across seven text QA benchmarks. LensVLM also generalizes to multimodal document and code understanding tasks, with the accuracy gain over baselines growing as compression increases. Our analysis validates this approach: training makes visual compression robust to rendering choices, and as compression grows the model increasingly relies on expanded content rather than unreliable visual reading. The analysis also yields practical tool-choice guidance: text expansion is preferable for rendered text, while high-resolution image expansion suits native documents whose layout cues carry task-relevant information.

Upvote 3

Get this paper in your agent:

Don't have the latest CLI? curl -LsSf https://hf.co/cli/install.sh | bash

Models citing this paper 2

apple/LensVLM-9B

Image-Text-to-Text • 9B • Updated 1 day ago • 233 • 70

suryatmodulus/LensVLM-9B

Image-Text-to-Text • 9B • Updated about 2 hours ago • 1

Datasets citing this paper 0

No dataset linking this paper

Spaces citing this paper 1

🔍

hugging-apps/lensvlm-9b-demo

Collections including this paper 0

No Collection including this paper

正文由 FLUX 从来源站点 RSS 同步，内容未经改写；遇到排版缺失或需要图片、视频时请以原文为准。