ECCV 2026

PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation

Yuhang Wu1 Shuxiang Zhang2 Wee Hian Ching1 Chi Zhang1,3 Miao Liu1,†
1Tsinghua University 2Sun Yat-sen University 3Shanghai Qi Zhi Institute †Corresponding author
Dataset Code BibTeX
First profile-inclusive benchmark for personalized image generation
1,369 evaluation testcases across real users and synthetic agents
2-5 preferred images used as the personalization signal
91% persona-aware judge agreement with human annotations
PIPBench problem setup and examples
PIPBench evaluates personalized image generation from a short prompt and a small set of user-preferred images, with user profiles providing implicit preference context.

Abstract

Recent text-to-image models excel at following diverse prompts, but they remain largely blind to individual aesthetic preferences. PIPBench studies personalized image generation, where a model must align outputs with a user's implicit visual preferences given a few historically preferred images and a short prompt.

The benchmark introduces a profile-inclusive data construction pipeline that uses psychological and demographic profiling dimensions for real-user collection and scalable synthetic-agent generation. Experiments on representative methods reveal clear limitations in current preference-conditioning strategies and point to new opportunities for personalized text-to-image synthesis.

Key Contributions

What PIPBench Adds

The benchmark makes implicit visual preference measurable by pairing user profiles, preferred images, and prompt-conditioned targets.

01

Profile-inclusive benchmark

Real user profiles pair demographic, psychological, and contextual signals with selected preferred images.

02

Hybrid data engine

Survey-calibrated real-user records are complemented by consistency-filtered synthetic agents for scale and diversity.

03

Preference-aware evaluation

Automatic metrics are paired with persona-aware Elo judging, so subjective preference fit is evaluated directly.

04

Method diagnosis

Representative conditioning strategies expose open challenges in multi-reference reasoning and prompt-preference balance.

Problem Setting

Generate What This User Would Like

Each testcase asks a model to satisfy an explicit prompt while inferring latent visual taste from reference images.

short prompt preferred images preference-aligned image

Input Prompt

A short caption focuses on the primary visual content rather than exhaustive style instructions.

Preference Set

Up to five historically preferred images provide signals for color, composition, mood, subject, and style.

Target Output

The generated image should remain faithful to the prompt while matching the user's underlying preferences.

Benchmark Construction

Profile-Inclusive Data Pipeline

PIPBench combines costly real-user calibration with scalable synthetic-agent construction, using the same profiling schema to keep both parts comparable.

PIPBench data construction pipeline
Real users provide profiles and preferred images through surveys. Synthetic agents are sampled from the same profile schema and filtered for consistency and diversity before image prompts are generated.
1

Profile users

Collect psychological measures and personal context that can shape visual preferences.

2

Generate candidates

Use profile-conditioned LLM reasoning to produce image prompts and candidate sets.

3

Select preferences

Real users pick matching images; synthetic agents rank candidates automatically.

4

Build testcases

Each testcase pairs a short prompt with reference preferences and a target image.

1,369 testcases
1,876 images
251 agents/users
76 real users

Real-User Dataset

Built from a 19-item questionnaire and a second-stage preference selection process. Users choose 6-8 preferred images from generated candidate sets, producing human-calibrated records.

  • 134 valid profile responses after quality control
  • 650 real-user testcases from 645 images
  • Profiles include psychological measures and personal context

Synthetic-Agent Dataset

Scales benchmark coverage by sampling profile-consistent agents and ranking candidate images automatically, while preserving population diversity through rule checks.

  • 719 synthetic-agent testcases from 1,231 images
  • 175 synthetic agents with consistency and diversity filtering
  • Supports additional training data for preference encoders

Evaluation

Hybrid Metrics for Preference Alignment

Automatic metrics track instruction fidelity and reference alignment, while persona-aware Elo evaluates subjective fit under a user profile.

CLS-T

CLIP text-image similarity for prompt fidelity.

LPIPS-R

Perceptual distance to the user's reference images.

CLS-R / DIS-R

CLIP and DINO image-image similarities for reference alignment.

Persona Elo

Pairwise LLM-as-a-judge comparisons conditioned on full user profiles, with about 91% agreement against human annotations.

Win, tie, and loss analysis for persona-aware judges
Multiple judges reduce self-enhancement and position bias, while exposing differences between conservative and decisive persona-conditioned evaluations.

Results

Existing Methods Still Struggle with Preferences

Preference-aware methods generally improve alignment over a no-preference baseline, but multi-reference understanding and prompt-preference tradeoffs remain open problems.

Best real-user Elo 1765

GPT-5 fusion gains +338 Elo over the no-preference baseline.

Reference alignment 22.160

Best real-user DIS-R, an 83% relative increase over no-preference.

Multi-reference gap -167

Two-reference editing trails one-reference editing in Elo, showing current fusion limits.

Method Synthetic Agent Real-User
LPIPS-R ↓ CLS-R ↑ DIS-R ↑ LPIPS-R ↓ CLS-R ↑ DIS-R ↑ Elo ↑
No preference 0.7559 62.784 10.195 0.7448 62.761 12.099 1427
DreamBooth 0.7374 63.444 10.705 0.7265 63.751 12.720 1452
Qwen-Image-Edit (1-Ref) 0.7326 64.615 15.461 0.7209 65.802 18.459 1521
Qwen-Image-Edit (2-Ref) 0.7652 62.319 12.259 0.7511 65.023 16.821 1354
GPT-5 fusion 0.6977 68.119 17.085 0.6867 69.574 22.160 1765
Gemini 2.5 Pro fusion 0.7038 67.335 14.403 0.6910 69.174 20.090 1615
QwenVL2.5-70B fusion 0.7183 65.894 17.336 0.7092 66.835 20.282 1531
Fabric 0.7277 64.531 13.367 0.7222 64.590 15.463 1412

VLM condition fusion is strongest

GPT-5 fusion obtains the highest real-user Elo and the best preference-alignment metrics among representative methods.

More references are not automatically better

Qwen-Image-Edit with two references underperforms its one-reference variant, showing that current models struggle to jointly interpret multiple preference examples.

Profile-inclusive data matters

Training on profile-inclusive synthetic data improves real-user alignment over profile-free data, including a 52.4% persona-aware win rate.

Visualization

Qualitative Comparisons

Preference-aware models add style and semantic traits beyond the short prompt, but can also overfit to an individual reference or conflict with instruction fidelity.

PIPBench qualitative comparison across generation methods
Visual comparison of no-preference, joint-conditioning, VLM-fusion, and separate-conditioning methods on personalized image generation cases.

Resources

The dataset and repository are available for reproducing benchmark construction, evaluation, and future personalized image generation research.

Hugging Face Dataset GitHub Repository

BibTeX

@inproceedings{wu2026pipbench,
  title={PIPBench: A Profile-Inclusive Framework for Personalized Image Generation Evaluation},
  author={Wu, Yuhang and Zhang, Shuxiang and Ching, Wee Hian and Zhang, Chi and Liu, Miao},
  booktitle={European Conference on Computer Vision (ECCV)},
  year={2026},
  url={https://github.com/wuyuhang05/PIPBench}
}