Papers

Kaifeng Zhao, Mathis Petrovich, Haotian Zhang et al.

For interactive applications, ARDY trades off some offline generation quality to achieve real-time streaming motion synthesis with flexible text and kinematic control, making it practical for animation tools and robotics where responsiveness matters as much as precision.

ARDY is a real-time motion generation system that creates realistic 3D human animations from text prompts and pose constraints.

Ayda Eghbalian, Kevin Desai

You can now add biomechanical analysis (forces, activations, loads) to any existing 3D pose estimator without retraining it, making markerless motion capture useful for rehabilitation, sports, and clinical applications.

This paper introduces BioModule, a lightweight transformer that converts 3D skeletal poses from any pose estimator into biomechanical quantities like joint forces and muscle activation. By aligning Human3.6M video data with biomechanical labels, the authors show how pose estimation errors propagate to biomechanical predictions across seven different pose estimators.

Chen Tang, Yizhou Wang, Jianyu Wu et al.

By making molecular and crystal structures inspectable during reasoning, SciReasoner achieves state-of-the-art predictions while generating human-readable explanations—showing how AI can solve chemistry and materials problems while remaining scientifically interpretable.

SciReasoner is a multimodal AI model that reasons about structure-property relationships in proteins, molecules, and crystals by treating structural information as interpretable evidence. It discretizes 3D coordinates and chemical topologies into a unified vocabulary, enabling transparent predictions grounded in scientific principles like bonding and symmetry.

Hyunjae Kim, Dain Kim, Pan Xiao et al.

High-fidelity data curation from scientific literature can create large-scale medical multimodal datasets that rival or exceed models trained on much larger datasets, enabling better medical AI without requiring new data collection.

MedPMC is a framework that automatically extracts and curates 11 million high-quality medical image-text pairs from 6.1 million PubMed Central articles. The resulting dataset trains multimodal models that significantly outperform existing biomedical baselines on medical imaging tasks, with 95.3% of extracted images validated as medically relevant by human reviewers.

Tianjiao Yu, Xinzhuo Li, Yifan Shen et al.

By routing text-to-geometry interactions across matched abstraction scales rather than flattening everything into one representation, you can build faster, more accurate 3D models that understand both coarse structure and fine details.

ELSA3D is a unified 3D foundation model that improves how language and 3D geometry interact by using 'anchor tokens' to match text concepts with the right level of geometric detail. Instead of treating all information equally, it routes language features to specific scales of 3D structure, making the model both more efficient and more accurate for generating and understanding 3D objects.

Zhenyu Liu, Yunxin Li, Xuanyu Zhang et al.

Modality interference—caused by gradient conflicts between audio and semantic processing—is the root cause of poor full-duplex SLM performance; hierarchical parameter separation solves this while maintaining cross-modality coherence.

This paper identifies and solves a critical problem in full-duplex spoken language models: when audio and text processing share the same neural network layers, they create conflicting gradients that degrade performance.

Han-Jun Ko, Jr-Jen Chen, Haobo Yuan et al.

Grounding VLM reasoning directly to visual observations and action consequences—rather than letting models generate free-form explanations—significantly improves physical reasoning generalization and reduces hallucination.

This paper tackles a key problem in vision-language models: they hallucinate reasoning that contradicts physics and misalign their explanations with actual actions. VAORA introduces two reward signals that anchor model reasoning to visual evidence and action outcomes, helping VLMs learn grounded physical reasoning that generalizes to new tasks and environments.

Wenhao Li, Xueying Jiang, Quanhao Qian et al.

Robot policies can achieve view robustness without camera calibration by learning to predict both action in camera space and camera-to-robot geometry, making deployment more practical when camera positions vary.

This paper introduces CamVLA, a robot vision-language-action model that learns to figure out camera positioning automatically instead of requiring explicit calibration. By predicting both camera-relative actions and the geometric relationship between camera and robot, the model works with any camera setup without needing depth data or prior calibration.

Haozhe Wang, Weijia Feng, Jinpeng Yu et al.

Visual generators need to learn *when* to search for external knowledge, not just *how* to use it—and this knowledge boundary is discoverable through co-training, not fixed in advance.

This paper identifies a critical gap in visual generators: they confidently create incorrect images for requests about new entities, trending topics, and post-training events. The authors show that naive search-augmentation fails because generators have an evolving 'knowledge boundary'—a threshold between what they learned and what needs external context.

Xuehui Wang, Xuankun Yang, Wei Shen

When pruning visual tokens in VLMs, filtering textual noise with entropy and selecting tokens as a structured optimization problem (not just picking top-K) preserves fine-grained details better while reducing computation.

This paper tackles the problem of compressing image tokens in vision-language models (VLMs) while preserving important visual details. The authors identify that existing pruning methods fail because textual noise corrupts the scoring process and selected tokens become fragmented.

Junhao Shi, Siyin Wang, Xiaopeng Yu et al.

Separating motor skill learning from language grounding dramatically reduces the labeled data needed for robot learning—TAP matches models trained on 1M+ expert trajectories while using far less labeled data and shows better robustness to real-world perturbations.

This paper proposes Task-Agnostic Pretraining (TAP), a two-stage approach for training Vision-Language-Action robots that separates learning how to move (from unlabeled robot interactions) from learning what to do (from minimal labeled data).

Quoc Bao Phan, Tuy Tan Nguyen

Quantum circuits can replace classical fusion layers in federated learning with 72 parameters instead of 33K, making multi-agent activity recognition more practical for resource-constrained robotic systems.

This paper presents QFedAgent, a federated learning system for activity recognition across multiple robotic agents. It uses quantum circuits to fuse sensor data (accelerometer and gyroscope) more efficiently than classical neural networks, reducing parameters by 10x while maintaining accuracy on distributed, non-uniform data.

Boyang Sun, Jiajie Li, Yung-Hsu Yang et al.

Robots can learn intent-aware camera control from passive human video by mining supervision pairs of language descriptions, observation changes, and target poses—turning everyday egocentric footage into training data for active perception.

This paper tackles language-conditioned camera motion for robots by learning from egocentric video. Given an image and natural language intent, LIME predicts the next camera pose by combining observation-gain prediction with flow-matching, enabling robots to actively position cameras for inspection, occlusion handling, or user-intent-driven viewing.

Xianhui Meng, Zirui Song, Yuchen Zhang et al.

For 3D scene generation in irregular spaces, hierarchical placement strategies and statistical priors about object distributions significantly improve physical plausibility and reduce geometric violations compared to flat optimization approaches.

This paper tackles text-to-3D indoor scene generation in non-Manhattan (non-rectangular) spaces, where existing methods fail. SPG-Layout uses statistical priors about object placement and hierarchical layout strategies (placing large objects first) to generate physically realistic scenes that respect non-orthogonal spatial relationships.

Cristian-Gabriel Florea, Stelian Spînu

By running six specialized deep learning models locally on a smartphone with INT8 quantization, VisionAId achieves real-time visual assistance for blind and low-vision users without cloud dependency, while a few-shot learning pipeline lets users teach the system to find their personal objects.

VisionAId is an Android app that helps visually impaired people navigate and interact with their environment using on-device AI models. It combines depth estimation, object detection, and facial recognition to identify obstacles, locate personal items, and recognize faces—all running locally on a smartphone without requiring cloud connectivity, with optional AI for scene descriptions.

Nina Begus

Literary disciplines offer practical tools for making AI systems more culturally literate and pluralistic, moving beyond the monolingual, automated cultural encounters that current LLMs create.

This essay argues that literary analysis methods—comparative reading, narratology, critical theory, and world literature approaches—are essential for building culturally aware AI systems.

Chenyang Ma, Yue Yang, Radu Corcodel et al.

Progress signals and semantic subtask grounding are critical for long-horizon bimanual manipulation—the model predicts both actions and continuous progress to automatically transition between assembly steps and reduce compounding errors.

FurnitureVLA tackles real-scale bimanual robot furniture assembly using vision-language-action models. The system combines a VR teleoperation interface for data collection, a simulation pipeline for training, and a progress-aware model that predicts both actions and assembly progress to handle long-horizon tasks (up to 1550 steps).

Zekai Chen, Kairui Yang, Xuaner Chen et al.

Federated multimodal graph learning can achieve strong performance while maintaining privacy and interpretability by organizing knowledge into typed semantic codebooks that explicitly track how different modalities and graph structure contribute to predictions.

FedLAB enables federated learning on multimodal graphs (graphs with text, images, and attributes) while preserving privacy by organizing knowledge into traceable semantic codebooks.

Sanghyuk Chun, William Yang, Amaya Dharmasiri et al.

Breaking uncertainty into interpretable components—what's ambiguous about the task versus how many right answers exist—lets you estimate confidence efficiently in multimodal models without expensive sampling.

CoMet decomposes uncertainty in multimodal AI models into two components: context-specific ambiguity (from the task or prompt) and multiplicity (how many valid answers exist). A lightweight module estimates these without generating multiple answers, enabling efficient uncertainty quantification for open-ended tasks like visual question answering.

Shun Lei, Huaicheng Zhang, Dapeng Wu et al.

For music generation at scale, separating semantic planning (what to generate) from acoustic refinement (how to generate it) and training them sequentially rather than simultaneously improves both coherence and sound quality.

LeVo 2 generates full-length songs by combining language models and diffusion models in a hierarchical approach: first predicting mixed vocal-instrument tokens for overall coherence, then refining each track separately for acoustic detail.

Yuhong Deng, Yuyao Liu, David Hsu

By decomposing tool affordance grounding into semantic (which object/part) and geometric (where) levels, robots can generalize to novel objects and creative tool use without expensive end-to-end training.

GROW² enables robots to creatively use any available object as a tool by breaking down the problem into two steps: using vision-language models to identify which object and which part to use, then using vision models to locate the exact 3D position. This lets robots solve tasks like cutting cake with a plate when no knife is available, without needing large labeled datasets.

Liyao Wang, Ruipu Wu, Haojun Xu et al.

Combining explicit 3D geometry (camera poses, spatial relationships) with visual matching dramatically improves cross-view localization and enables zero-shot transfer between ground and drone views without paired training data.

This paper tackles cross-view object geo-localization—finding a target object in satellite imagery when given a ground or drone photo. The authors introduce a large dataset with 220K+ image pairs and geometric metadata, plus GAGeo, a unified framework that predicts object locations, masks, and camera poses simultaneously using 3D spatial understanding rather than just appearance matching.

Sihang Nie, Xiaofen Xing, Rui Xing et al.

Separating content and emotion into distinct latent spaces during training prevents reward conflicts and enables better emotional control in TTS systems without sacrificing intelligibility.

This paper addresses emotional expressiveness in LLM-based text-to-speech by proposing HPRO, a hierarchical reward optimization framework that separates emotional and semantic information to avoid conflicting gradients, then progressively aligns rewards across frame, word, and sentence levels to improve emotional control while maintaining speech clarity.

Wei Zhou, Xiongwei Zhu, Zelin Xu et al.

Multi-task image generation models can be trained more effectively by treating each capability (T2I, local edit, global edit) as a separate velocity field and having the student learn to compose them on its own generated trajectories.

DanceOPD is a training framework that helps image generation models master multiple tasks—text-to-image, local editing, and global editing—without them interfering with each other. It uses a distillation approach where a student model learns from specialized 'capability fields' (velocity fields in flow-matching models), routing each image to the right expert for its task.

Kirill Solovev, Jana Lasser

Open-weight multilingual NLP can scale political network analysis beyond manual coding, extracting signed relationships from news at scale while remaining reproducible and avoiding proprietary APIs.

This paper presents an open-source pipeline for automatically extracting political relationships from multilingual news articles. It combines named-entity recognition, entity linking to Wikidata, and a specialized model to build knowledge graphs of political networks—showing it can reconstruct party lifecycles and uncover patronage networks in Austria and Poland.

Mohammad Mehdi Hosseini, Mohammad H. Mahoor, Hiroko H. Dodge

Language-based digital twins can authentically mimic individual conversational behavior while simultaneously serving as cognitive health monitors—enabling continuous, personalized monitoring of cognitive decline without requiring frequent clinical visits.

This paper creates AI-powered digital twins of elderly people using language models to capture their unique conversational patterns and writing style. The system learns from real conversations to generate authentic responses and can predict cognitive health scores, offering a non-invasive way to monitor early signs of cognitive decline.

Junwei Luo, Shuai Yuan, Zhenya Yang et al.

For Earth observation forecasting, explicitly conditioning on weather anomalies and cumulative physical stress—rather than treating weather as generic conditioning—improves predictions of how vegetation responds to extreme weather events.

EO-WM is a video diffusion model for predicting future Earth surface conditions from satellite imagery while accounting for weather effects. Unlike existing methods, it explicitly models how weather forcing (heat, drought) drives changes in vegetation and other surface features, using separate conditioning pathways for baseline climate and weather anomalies.

Jiajun Wu, Haoyu Kang, Yining Sun et al.

Evaluating harmful content detection requires multi-layered reasoning beyond surface-level classification; models need to explain their decisions and understand implicit harms, not just flag obvious ones.

HarmVideoBench is a benchmark for evaluating how well AI models understand harmful content in videos. Unlike existing tests that just ask yes/no questions, this benchmark uses 1,379 videos with 4,137 multiple-choice questions across three difficulty levels—from spotting obvious harmful elements to reasoning about context beyond what's shown.

Henrik Müller, Daniel Kudenko

You can use smaller, cheaper VLMs to automatically design reward shaping functions that guide RL agents without the risk of reward hacking, eliminating the need for manual reward engineering.

This paper automates reward shaping for reinforcement learning by using vision language models to learn a potential function that guides exploration without causing reward hacking. The method queries lightweight VLMs to compare image pairs, trains a model of the potential function from these preferences, and preserves optimal policies while improving sample efficiency in robotic tasks.

Dong Jing, Tianqi Zhang, Jiaqi Liu et al.

Pretraining action modules on motion structure before vision-language alignment significantly improves robot learning efficiency and cross-embodiment generalization, particularly in data-scarce real-world settings.

This paper proposes a two-stage training approach for robot manipulation models that first learns motion patterns from action trajectories alone, then transfers this knowledge to vision-language-action models.

Martijn Bartelds, Federico Bianchi, James Zou

Real-time voice AI systems can hear emotional cues but don't use them in decision-making; they need explicit prompting to consider tone, and even then improve only partially—making them risky for emotionally sensitive interactions.

This paper evaluates four leading real-time voice AI systems (GPT-4 Realtime, Gemini Live, Qwen Omni) and finds they ignore emotional tone and vocal delivery when making decisions, even though they can perceive these cues when asked directly.

Akshay Paruchuri, Sanmi Koyejo, Ehsan Adeli

Multimodal AI models are unreliably sensitive to input order—a property that should be baseline for production systems. Simple prompt fixes don't solve this; the problem likely requires changes during model training or design.

This paper audits 18 multimodal AI models to check if they give consistent answers when information is presented in different orders. The researchers found that all models fail this basic reliability test, with 24-50% of answers changing based on order.

Yuxing Cheng, Yuan Wu, Yi Chang

High accuracy on clean images doesn't guarantee robustness to visual corruption—VLMs struggle significantly with degraded text-rich content, especially structured formats like charts and tables, which matters for real-world deployment.

This paper introduces OCR-Robust, a benchmark for testing how well vision-language models handle text recognition and reasoning when images are corrupted or degraded.

Yu-Yang Chen, Lan-Zhe Guo

Current multimodal AI models have a fundamental weakness in multi-view spatial reasoning—they can't reliably track objects across different camera angles, and this limitation can't be fixed by better prompting strategies alone.

TriViewBench is a controlled benchmark for testing how well multimodal AI models handle complex visual reasoning across multiple views of 3D scenes.

Maggie Wang, Lars Osterberg, Stephen Tian et al.

By making VLA models steerable at the primitive-action level, you can create a self-improving loop where robots identify skill gaps, practice autonomously, and expand their capabilities continuously.

InSight enables vision-language-action models to autonomously learn new manipulation skills by breaking down demonstrations into reusable primitive actions (like "move gripper to bowl").

Haorui Ji, Weizhe Liu, Hongdong Li et al.

To build better image-to-3D systems, focus on using features designed for reconstruction rather than classification, and explicitly align 2D image information with 3D geometry during generation.

FLUX3D generates high-quality 3D Gaussian Splatting models from images by improving how 2D image features are converted into 3D representations and how the generation process aligns 2D and 3D data. It uses specialized techniques to preserve fine visual details that previous methods lost, resulting in better-looking 3D assets.

Zixuan Li, Haokun Lin, Yicheng Xiao et al.

Separating structure planning from appearance rendering in image generation improves prompt following for complex spatial and compositional requirements without needing intermediate outputs.

This paper improves text-to-image generation by separating structural planning from appearance rendering. IV-CoT uses two types of queries—structural and semantic—that work together in a single pass: structural queries create a latent visual plan (like an invisible sketch), then semantic queries render the final image based on that plan.

Chenrui Fan, Paolo Favaro

Using 3D reconstruction as an anchor to guide video generation creates better 3D consistency than generating videos alone, and you can do this by reusing existing video models without task-specific training.

OrbitForge converts text descriptions into 3D scenes by leveraging frozen video generation models and Gaussian Splatting reconstruction. It generates a video from text, identifies missing viewpoints around a complete orbit, fills those gaps with the video model, and reconstructs everything into a consistent 3D scene—all without fine-tuning or slow step-by-step generation.

Sara Dorfman, Maya Vishnevsky, Omer Dahary et al.

By generating diverse prompts rather than diverse images from one prompt, you can create navigable design spaces where each variation is semantically meaningful and user-understandable, rather than random visual differences.

This paper solves the diversity problem in text-to-image generation by shifting variation from the model's random sampling to the text prompt itself.

Cong Han, Xiaohan Lan, Haibo Qiu et al.

MLLMs can be trained to adaptively switch between reasoning and code execution for numerical tasks—not just visual ones—using reinforcement learning with a specialized reward function that guides when to invoke tools.

This paper enhances multimodal AI models to reason through complex math and numerical problems by interleaving natural language thinking with executable code. The authors use reinforcement learning to train models to decide when and how to use code tools, achieving 6.1% accuracy improvement on benchmarks.

Sunil Wanjari, Manish Thakre, Aayushi Asole et al.

Combining multiple validated mental health screening tools with cognitive and personality data through a modular, interpretable system improves diagnostic accuracy and provides clinicians with more reliable decision support than using individual assessments alone.

PsyBridge is a decision-support system that combines clinical screening tools (PHQ-9, GAD-7), cognitive tests, and personality assessments into one interpretable framework for mental health evaluation.

Hengji Zhou, Ye Liu, Yufeng Liu et al.

You can generate personalized multimodal content on-demand by converting user behavior into preference signals, then using those signals to guide generation—avoiding the need to wait for or retrieve matching user-created content.

TailorMind generates personalized multimodal content (text + images) tailored to individual users by learning from their behavior patterns, even when no existing content matches their preferences. It combines collaborative filtering to understand user tastes with controllable generation to create novel, aesthetically pleasing content that stays true to authentic user patterns.

Shaghayegh Kolli, Timo Cavelius, Nafiseh Nikeghbal et al.

Social biases in vision-language models aren't random—they're driven by a small set of visual cues like age, body type, and clothing style. Understanding which specific visual features trigger biased judgments is crucial for building fairer AI systems.

This paper creates a controlled benchmark with 25K photorealistic images to measure how specific visual attributes (like age, body type, and fashion style) cause social biases in multimodal AI models. By keeping identity fixed and changing one visual cue at a time, researchers show that just 15 visual attributes account for 80% of bias variation across six major MLLMs.

Solène Debuysère, Nicolas Trouvé, Nathan Letheule et al.

This is the first large-scale, very-high-resolution SAR-optical-language dataset with complex-valued SAR data and pixel-level alignment, unlocking new benchmarks for multimodal foundation models that can learn from radar imagery the way they learn from optical images.

SARLO-80 is a large-scale dataset of 119,566 aligned SAR-optical-text triplets at 80cm resolution covering 257 locations worldwide. It preserves complex-valued SAR measurements and native acquisition geometry—unlike existing low-resolution datasets—enabling physically grounded multimodal learning for radar and optical image understanding.

Maria Ivanova, Pavel Zadorozhny, Rodion Levichev et al.

LLMs trained primarily on Python code don't generalize well to other languages—Multi-LCB exposes this critical gap and provides a rigorous way to measure cross-language coding ability.

Multi-LCB extends LiveCodeBench, a popular code-generation benchmark, from Python-only to 12 programming languages. The benchmark transforms competitive programming problems into equivalent tasks across languages while maintaining contamination controls, revealing that LLMs show Python overfitting and significant performance gaps across different languages.

Jinghong Lan, Wei Cheng, Yunuo Chen et al.

By mining community LoRAs as style-content anchors and using curriculum learning with targeted disentanglement mechanisms, you can scale dual-reference image generation while maintaining clean separation between style and content.

FreeStyle is a framework for generating images that combine the style of one image with the content of another. It uses community LoRA models as building blocks to create large training datasets and employs specialized techniques to prevent unwanted style or content leakage. The approach includes a new benchmark for evaluating dual-reference image generation.

Yusuf Salcan, Simon Ging, Robin Schirrmeister et al.

You can train medical vision-language models to perform spatial grounding (locating regions in images) alongside report generation without sacrificing language quality, using automatically-curated training data instead of expensive manual annotations.

This paper introduces RefRad2D, a large-scale bilingual dataset of 1.2M medical images paired with text, and RadGrounder, a vision-language model trained to simultaneously generate radiology reports, answer visual questions, and locate anatomical regions via bounding boxes or segmentation—all without manual spatial annotations.

Yupeng Xie, Chen Ma, Zhenyang Wang et al.

Instead of manually creating data visualizations or using pixel-level video generation, DataMagic separates data logic from rendering through a declarative spec, ensuring videos accurately represent underlying data while supporting interactive exploration.

DataMagic is a system that automatically transforms raw tabular data and natural language questions into narrative data videos with charts, voice narration, and animations. It uses a declarative specification to ensure data accuracy and a multi-agent architecture to generate and organize video scenes, making data insights more engaging and explorable than static dashboards.

Zhenghao Xing, Ruiyang Xu, Yuxuan Wang et al.

Active perception—where an AI agent decides what to observe rather than passively processing everything—enables better video understanding with lower computational cost and improves with more reasoning steps.

OmniAgent is an AI agent that watches videos intelligently by deciding what to pay attention to, rather than processing every frame. It combines audio, visual, and text understanding in a reasoning loop, building up memory of important moments. This approach scales better with video length and achieves top performance on video understanding benchmarks.

Michael Finkelson, Daniel Segal, Eitan Richardson et al.

You can generate multi-speaker audio with natural conversational qualities by using reference voices + free-form text prompts on foundation models, rather than building speech-only systems with rigid turn-based structure.

ScenA generates realistic multi-speaker audio scenes by conditioning a text-to-audio model on reference voice samples and natural language descriptions. Unlike systems that require structured turn-by-turn labels, it produces conversational audio with overlapping speech, background noise, and emotional sounds—mimicking real-world conversations.

Nikita Kachaev, Andrey Moskalenko, Matvey Skripkin et al.

VLA models trained for robotics lose significant commonsense and world knowledge compared to their base vision-language models, particularly on complex semantic tasks—a critical finding for building reliable embodied AI systems.

This paper introduces Act2Answer, a benchmark that measures whether vision-language-action (VLA) models—AI systems trained to understand images and perform robot actions—retain commonsense and factual knowledge after fine-tuning on robotics data.

Jiaqing Zhang, Sabyasachi Bandyopadhyay, Miguel Contreras et al.

Ambient sound and light sensors can be practical, non-invasive additions to ICU monitoring systems that meaningfully improve delirium risk prediction, with sound being the strongest signal.

This study uses ambient sound and light sensors in ICUs to predict which patients will develop delirium (acute confusion). Researchers tested neural network models on data from 309 patients across 9 hospitals, finding that sound levels alone predicted delirium with 80% accuracy, and combining sound with light improved short-term predictions.

Peiyang Xu, Bangzheng Li, Sijia Liu et al.

Training models to identify supporting evidence through context selection—not just answer correctness—improves long-horizon reasoning and multimodal performance without requiring more data.

ContextRL trains LLMs to better handle long contexts and multimodal inputs by rewarding models for selecting the correct supporting context from similar alternatives, rather than just supervising final answers. This indirect approach improves reasoning on coding tasks and visual question-answering by encouraging fine-grained evidence grounding.

Jisang Han, Seonghu Jeon, Jaewoo Jung et al.

Using 3D geometric reasoning as a shared foundation for both world prediction and action generation makes robot policies more accurate and efficient than 2D-based approaches, while requiring fewer parameters than large foundation models.

This paper introduces Geometric Action Model (GAM), a robot control system that uses a pretrained 3D geometry foundation model to understand both the physical world and predict robot actions.

Jiaju Han, Ben Zhang, Xuemeng Sun et al.

Combining infrared and RGB satellite imagery with modality-specific captions significantly improves vision-language models' ability to understand Earth observation data—infrared-aware text supervision is essential for effective multi-modal learning.

FusionRS is the first large-scale dataset pairing RGB and infrared satellite images with text descriptions for training AI models that understand both types of imagery together.

Jiayue Cao, Zhicong Lu, Xuehan Sun et al.

When training vision-language models to reason step-by-step, you need to explicitly enforce that the reasoning process logically leads to the final answer—not just optimize for getting the right answer.

This paper identifies and fixes a problem in multimodal AI models where their reasoning process doesn't match their final answer. The authors propose CORA, a method that adds a consistency check during training to ensure the model's thinking aligns with what it concludes, improving both accuracy and reasoning reliability.

Ines Nolasco, Jules Cauzinille, Marius Miron et al.

Pretrained audio models encode different acoustic features unevenly; combining multiple models or selecting them based on task-specific feature importance improves bioacoustics applications, especially for rare species where data is scarce.

This paper investigates what acoustic features are captured by pretrained audio embedding models used in bioacoustics research. By testing six models across different animal species using 88 speech-derived acoustic features, the authors show that no single model captures all important features—loudness is well-encoded while pitch (F0) is difficult to recover.

Seokju Cho, Ryo Hachiuma, Abhishek Badki et al.

Using executable code as an action interface lets vision-language models flexibly compose spatial reasoning operations and adapt to intermediate results, significantly improving performance on 3D/4D reasoning tasks without model-specific tuning.

SpatialClaw is a framework that helps AI agents reason about 3D space and object relationships by letting them write and execute Python code step-by-step. Instead of committing to a full analysis upfront or using rigid tool menus, the agent can see intermediate results and adapt its approach, achieving 59.9% accuracy across spatial reasoning tasks—11.2 points better than prior methods.

Pedro Correa, Olivier Perrotin, Samir Sadok et al.

Phonetic information in speech representations is crucial for accurate 3D facial animation; discrete token-based representations can serve as an effective bridge between speech and facial motion synthesis.

This paper investigates which speech representations work best for animating 3D faces from audio. The researchers compare four types of speech encodings—self-supervised learning features, neural codec outputs, and ASR-based representations—and find that representations capturing phonetic information produce the most accurate facial animations.

Dachun Kai, Jiayao Lu, Yueyi Zhang et al.

Event cameras can significantly boost video super-resolution quality by providing high-temporal-resolution texture cues and motion information, improving detail recovery by up to 1.55 dB PSNR on texture-rich videos.

This paper introduces EvTexture++, a framework that uses event camera data to improve video super-resolution by enhancing texture details and temporal consistency. Event cameras capture high-frequency visual information that helps recover fine details in upscaled videos and reduce flickering artifacts caused by motion.

Baochang Ren, Xinjie Liu, Xi Chen et al.

Vision-language-action models can now control lab robots by combining action token pretraining with flow matching, but success requires both lab-specific training data and support for multiple robot embodiments.

This paper introduces LabVLA, a vision-language-action model designed to control robots in scientific laboratories. The key innovation is a two-stage training approach: first pretraining the model to understand action tokens, then fine-tuning it with flow matching.

Tanmoy Kanti Halder, Akash Ghosh, Subhadip Baidya et al.

Specialized medical AI systems need language-specific and multimodal training to serve non-English-speaking populations effectively—this work shows how to build such systems with structured reasoning agents.

ArogyaSutra addresses healthcare AI gaps in rural India by introducing a multilingual medical dataset covering 21 clinical domains in English and seven Indian languages, plus a multi-agent framework that reasons through medical questions step-by-step using tool integration and memory mechanisms.

Shengqiang Zhang, Ruotong Liao, Volker Tresp et al.

By working with a VAR model's native bitwise predictions and residual code composition instead of token streams, you can edit images more precisely while preserving backgrounds—no retraining needed.

BitResEdit is a training-free editor for visual autoregressive image generators that uses two techniques: BitEdit guides bit-level predictions toward text descriptions, while ResEdit applies edits through the model's native residual code structure. This approach preserves unedited regions exactly while making precise, localized changes to match text prompts.

Soumyajit Mitra, Prabhat Pandey, Abhinav Jain et al.

Role-based conditioning significantly improves when voice agents decide to speak in group conversations—a critical capability for real-time multi-party voice applications like meeting assistants and collaborative AI systems.

This paper presents ModeratorLM, a voice agent that improves turn-taking in multi-party conversations by conditioning behavior on an explicitly assigned role (e.g., moderator, participant). The system uses a speech language model with streaming processing and optional chain-of-thought reasoning.

Cheng-Yu Yang, Shao-Yuan Lo, Yu-Lun Liu

Visual token importance changes as the model processes information deeper—tokens deemed unimportant early on may matter later, so recoverable routing outperforms permanent removal for vision-language tasks.

Vision-language models use thousands of visual tokens, making inference slow. Instead of permanently removing low-scoring tokens, this paper proposes Reroute: tokens can be temporarily skipped and re-evaluated later when they become important. The method works with existing token-reduction techniques and improves performance on grounding tasks without extra computational cost.

Zhi Wei Xu, Torbjörn E. M. Nordling

Robots can now reliably measure human heart rate from RGB cameras in varying lighting by combining spatial-temporal transformers with illumination-aware training, enabling non-contact physiological monitoring for human-robot interaction.

This paper develops a camera-based system for robots to measure human heart rate from video, even when lighting conditions change dramatically.

Ilay Kamai, Hugues Van Assel, Aviv Regev et al.

Before training a multimodal model, use the paper's diagnostic procedure to determine whether alignment, prediction, or neither will work for your specific data—saving wasted effort on approaches that will fail or harm performance.

This paper explains when to use cross-modal alignment versus cross-modal prediction for multimodal learning. Using a mathematical framework, the authors identify four regimes where each approach works best, fails, or actively hurts performance. They provide a practical diagnostic tool to test real datasets with minimal labeled data before committing to training.

Kevin Qinghong Lin, Batu EI, Yuhong Shi et al.

Building trustworthy AI journalism requires end-to-end orchestration of specialized agents plus explicit evidence-grounding: every number and claim must link back to verifiable data, code, or references.

Data2Story is a multi-agent system that automates data journalism by transforming raw data into complete multimedia news stories. It combines specialized agents for data analysis, writing, and design while ensuring every claim is traceable back to its source—making stories more transparent and verifiable than traditional approaches.

Mingxian Lin, Shengju Qian, Yuqi Liu et al.

Game benchmarks should measure agent improvement over time through iterative refinement, not just first-attempt performance—this reveals which VLMs can learn and adapt in interactive environments.

OmniGameArena is a benchmark for testing vision-language model agents in 12 Unreal Engine 5 games across different play modes (solo, competitive, cooperative). It introduces Improvement Dynamics Curve, which measures how agents improve when given multiple chances to refine their strategies through self-reflection, revealing performance evolution beyond single-attempt scores.

Badr AlKhamissi, Johannes Mehrer, Lara Marinov et al.

A single spatial organization principle governs how the brain integrates different types of information (vision, sound, language), and this principle can be captured in AI models to predict and discover new functional brain regions.

Researchers built Topo-Omni, a neural network model that maps how the brain organizes visual, auditory, and language processing in a single unified spatial layout. By adding a spatial smoothness constraint to a pretrained foundation model, they discovered that nearby regions specialize in related tasks—just like real brains do.

Yichen Gao, Yiqun Zhang, Zijing Wang et al.

Audio-language models encode audio evidence correctly but lose arbitration to text—this can be fixed at inference time by reweighting scores from audio-only and audio-text branches, improving accuracy by 17.8 points without retraining.

Audio-language models often prefer text over audio even when audio is clearly correct. This paper shows that the audio information is actually encoded in the model but gets overridden during decision-making. By removing conflicting text and measuring how the model's preference changes, researchers found that 64% of conflicts can be reversed.

Josef Bengtson, Yaroslava Lochman, Fredrik Kahl

This method lets you edit one image of an object and automatically propagate those changes—including major shape changes—consistently across all other camera angles without retraining.

GeM-NR enables consistent editing of multi-view images when geometry changes dramatically—like reshaping objects or altering scene structure. Unlike prior methods that only handle appearance changes, it works with any image editor (FLUX, Qwen, BrushNet) by aligning 3D geometry between edited and unedited views, then intelligently projecting and refining edits across viewpoints.

Zhifei Xie, Zihang Liu, Ze An et al.

Real-time audio AI is now possible with a single model that can listen, understand, and respond continuously—moving beyond offline systems that handle only one task at a time.

This paper introduces Audio-Interaction, a unified streaming model that listens to audio in real time and responds on the fly, rather than processing audio offline like current systems.

Mahtab Bigverdi, Lindsey Li, Weikai Huang et al.

Training vision-language models to generate intermediate visual representations of unseen spatial configurations works better than text-based reasoning for spatial tasks, and these representations remain interpretable without needing to generate actual images at inference time.

This paper introduces Imaginative Perception Tokens (IPT), a training method that helps vision-language models reason about spaces they can't directly see. Instead of forcing spatial reasoning through text, IPT teaches models to generate intermediate visual representations of what they would perceive from different viewpoints or through occluded spaces.

Quentin Fuxa, Dominik Macháček

AlignAtt can be applied to decoder-only LLMs by using prompt engineering, selective attention head selection, and runtime attention capture—enabling fast simultaneous translation without the encoder-decoder architecture that earlier methods relied on.

This paper adapts AlignAtt, a technique for controlling when a machine translation model reads source text, to work with decoder-only language models like Gemma-4. The system performs simultaneous speech translation by incrementally updating transcripts and deciding when to translate, achieving low latency while maintaining translation quality for English-to-German, Italian, and Chinese.

Hanjiang Hu, Yiyuan Pan, Jiaxing Li et al.

Real-time safety monitoring for physical tasks requires understanding intent—the same action is safe or dangerous depending on context, which VLESA solves with a goal-conditioned safety filter that works without retraining for new scenarios.

VLESA is a safety system that watches egocentric video to detect dangerous human actions and intervene in real-time. It understands that the same action can be safe or unsafe depending on what the person is trying to do, using a goal-aware safety filter trained with reinforcement learning to make context-aware safety decisions without retraining.

Seojeong Park, Jiho Choi, Junyong Kang et al.

Multimodal AI judges can be fooled into trusting text over images—training them on perceptually grounded examples significantly improves their ability to make consistent, verifiable evaluations.

This paper identifies and fixes a critical flaw in multimodal AI judges: they often trust plausible-sounding text over what they actually see in images. The authors create a dataset of carefully modified images and responses to train judges to rely on visual evidence, resulting in more reliable automated evaluation systems.

Yu-Cheng Shi, Zhen-Hao Xie, Jun-Tao Tang et al.

When continually training multimodal models on new tasks, routing decisions based only on semantic similarity fail—you also need to account for output format differences to prevent gradient interference and task confusion.

ProtoAda solves a key problem in continual learning for multimodal AI: when models learn new vision-language tasks sequentially, they often forget old ones or mix up tasks with different output formats (like coordinate prediction vs. text answers).

Haowen Hou, Zhen Huang, Zheming Liang et al.

Video MLLMs can be dramatically more efficient by encoding frames predictively: send full frames only when needed, use compact change descriptions otherwise. This cuts token usage to 1/7 while improving accuracy and reducing latency from 9+ seconds to 1.6 seconds.

AdaCodec is a new way to encode videos for AI models that intelligently decides when to send full frames versus compact descriptions of changes.

Utsav Dutta, Gerardo Pastrana, Sina Khoshfetrat Pakazad et al.

You can train a single time-series model on unlabeled sensor data using text descriptions of what each sensor measures, and the resulting embeddings transfer well to multiple downstream tasks without task-specific retraining.

CHARM is a new model that learns meaningful representations from multivariate time-series data (like sensor readings) by combining channel descriptions with a Transformer architecture. It uses a self-supervised training approach called JEPA that predicts future embeddings rather than raw values, making it robust to sensor noise.

Jusuk Lee, Seungjae Lee, Jonghun Shin et al.

Robot perception improves significantly when visual encoders are trained to understand dynamics and motion during pre-training, rather than relying on static image recognition—this upstream motion understanding boosts downstream policy performance by up to 22.5% in out-of-distribution settings.

DynaFLIP is a pre-training method that teaches robot vision systems to understand motion and dynamics, not just static scenes. By training on image-language-3D flow triplets from videos, it creates visual representations that capture action-relevant changes in the world, making robots better at manipulation tasks across different scenarios.

Xiaona Zhou, Muntasir Wahed, Tianjiao Yu et al.

You can build smaller, more interpretable anomaly detection systems by fine-tuning vision-language models on curated datasets with natural-language explanations rather than relying on large general-purpose models.

This paper creates VisAnomBench, a curated dataset of time-series anomalies with AI-generated explanations, and uses it to fine-tune VisAnomReasoner—a lightweight vision-language model for detecting and explaining unusual patterns in sequential data. The approach achieves significant improvements over existing methods while remaining parameter-efficient.

Ruixiang Jiang, Chang Wen Chen

Pre-capture planning for portraits is more effective than post-production editing: by reasoning about 3D scene geometry, lighting physics, and aesthetic principles before shooting, you can automatically generate better photography setups.

This paper tackles portrait photography planning before taking the photo—deciding where to position the subject, where to place the camera, and how to set up lighting in a 3D scene.

Chong Bao, Shichen Liu, Lijun Yu et al.

A single pretrained model can now generate realistic digital humans across all modalities (speech, movement, appearance) by treating them as a unified problem rather than separate tasks, making it practical to build avatar systems without specialized sub-models.

Archon is a unified AI model that generates digital humans across multiple modalities—text, audio, motion, and video—all from a single system. It uses a novel video compression technique to handle high-resolution talking videos efficiently, and a step-by-step reasoning approach that switches between modalities to improve output quality.

Jinzhou Wu, Zhengwu Ma, Jixing Li et al.

Multimodal training doesn't automatically make language models more human-like; visual pretraining helps selectively for visually-rich text, but language-internal representations remain the foundation for modeling human reading.

This paper compares language models trained only on text (LLMs) with models trained on both text and images (VLMs) to see if visual training makes AI better at matching how humans read. Using brain scans and eye-tracking data from real readers, the researchers found that VLMs don't universally outperform LLMs—language-only training remains crucial.

Viet Nguyen, Thao Nguyen, Vishal M. Patel et al.

Personal AI agents need dedicated visual memory systems that preserve image-specific information beyond captions—explicit entities and implicit user facts that text alone cannot capture.

This paper introduces a benchmark and system for helping AI agents remember personal information from images over long conversations. Most memory systems convert images to text captions, losing visual details about people, objects, and relationships.