이미지: METAL LAB 생성
Summary
- Apple Research published a paper comprehensively analyzing preference alignment methods for multimodal LLMs
- The study found that combining offline (DPO) and online (online-DPO) alignment approaches improves performance in certain situations
- It also proposes a new technique called 'BDHS' that creates preference data without additional annotation or external models, showing performance competitive with existing techniques
- 발표
- Apple ML Research, 2026년 8월 3일
- 논문 제목
- Understanding Alignment in Multimodal LLMs: A Comprehensive Study
- 핵심 제안 기법
- Bias-Driven Hallucination Sampling(BDHS)
- 비교 대상
- 오프라인 정렬(DPO), 온라인 정렬(online-DPO)
- 저자 규모
- Elmira Amirloo 등 11명(애플 소속)
AI that says something different from what it sees in an image — why does this happen?
A paper released by Apple Research on August 3 addresses why multimodal large language models (MLLMs — AI that understands images and text together) say things that differ from what's actually in a photo, and how this can be reduced. Text-only language models already suffer from "hallucination," the problem of fabricating statements that don't match facts, but models that also handle images face an additional layer of complexity. On top of stating incorrect facts, they can also fabricate content that isn't even present in the image. Apple took a direct look at "preference alignment," a training method used to reduce this.
Alignment is an old challenge, but still unfinished for multimodal models
Preference alignment is a training method where, among multiple answers an AI produces, the one humans prefer is selected and the model is gradually nudged toward that answer. This technique is already widely used in text-only models like ChatGPT, but the paper starts from the premise that it has been relatively less studied in multimodal models that also handle images. Various research teams have reported performance improvements using different datasets, base models, and alignment methods (DPO, PPO, etc.), but what actually drove those improvements has remained tangled and unclear. Apple's research team separated alignment algorithms into two categories — "offline" methods that train on pre-constructed pairs of answers, and "online" methods that generate answers in real time during training and adjust accordingly — and analyzed the effect of each separately. The results showed that combining both approaches improves performance in certain situations. The paper goes a step further, also examining how the construction methods of various publicly available multimodal preference datasets actually affect performance.
Z.ai releases image-to-text model GLM-OCR
Training data made without human hands: BDHS
The most notable proposal in this paper is a new data generation method called "Bias-Driven Hallucination Sampling (BDHS)." Existing methods often required humans to directly mark which answer was more accurate, or relied on separate external AI models to generate preference data. Both approaches cost time and money. BDHS generates preference data by leveraging the model's own biases, without requiring this additional work, and Apple stated that it achieved performance competitive with previously published alignment techniques across multiple benchmarks.
So what changes?
This research appears likely to serve as a practical reference for teams building multimodal AI. Until now, even when alignment performance improvements were announced, it was difficult to tell whether the improvement came from the dataset or the algorithm. Since Apple has separated each factor and presented comparative tables, follow-up researchers now have a basis for judging which combinations offer the best cost-to-performance ratio. In particular, the BDHS approach — generating preference data without manual human work — may also be worth considering for the open-source community looking to lower the cost of alignment training.
