Mobile GUI Agent Privacy Personalization with Trajectory Induced Preference Optimization
Abstract
Mobile GUI agents using MLLMs can execute complex tasks while addressing privacy personalization through trajectory-induced preference optimization that improves persona alignment and task executability.
Mobile GUI agents powered by Multimodal Large Language Models (MLLMs) can execute complex tasks on mobile devices. Despite this progress, most existing systems still optimize task success or efficiency, neglecting users' privacy personalization. In this paper, we study the often-overlooked problem of agent personalization. We observe that personalization can induce systematic structural heterogeneity in execution trajectories. For example, privacy-first users often prefer protective actions, e.g., refusing permissions, logging out, and minimizing exposure, leading to logically different execution trajectories from utility-first users. Such variable-length and structurally different trajectories make standard preference optimization unstable and less informative. To address this issue, we propose Trajectory Induced Preference Optimization (TIPO), which uses preference-intensity weighting to emphasize key privacy-related steps and padding gating to suppress alignment noise. Results on our Privacy Preference Dataset show that TIPO improves persona alignment and distinction while preserving strong task executability, achieving 65.60% SR, 46.22 Compliance, and 66.67% PD, outperforming existing optimization methods across various GUI tasks. The code and dataset will be publicly released at https://github.com/Zhixin-L/TIPO.
Community
We introduce a new framework for personalized mobile GUI agents: modeling personalization as persona-consistent trajectory selection, building a privacy-preference dataset with variable-length and structurally diverse trajectory annotations, and proposing TIPO, which enables more stable preference optimization and delivers strong empirical gains.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation (2026)
- PSPA-Bench: A Personalized Benchmark for Smartphone GUI Agent (2026)
- BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs (2026)
- Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History (2026)
- Do Phone-Use Agents Respect Your Privacy? (2026)
- PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments (2026)
- Long Context, Less Focus: A Scaling Gap in LLMs Revealed through Privacy and Personalization (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2604.11259 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper