UI Trees or Pixels? Studying Modality Choices in Android Action-Taking Models

Chris Gresla

2025-02-06

How should a model that operates the Android OS be architected?

Abstract

We study how choices of screen representation modality impact the performance of action-taking models on the Android OS. Several recent works probe the design space of these action-taking models, ^{1 2 3 4 5} but the key architectural question of what screen representation modality should be used as a base for device manipulation model perception remains underexplored. We study systems based on textual and visual modalities and find that visual models which are built to percieve their environment through device screenshots provide a compelling architecture for action models. These multimodal models perform on par with text-only action models, but they learn richer relations between spatial actions and screens, require substantially less tokens to process the same information, continually learn whilst text-only approaches overfit, and present a general base for other applications that require processing visual inputs. Our findings inform the design of future action-taking models, suggesting that integrating information across modalities is paramount to effective intelligence.

Introduction

The proliferation of Large Language Models (LLMs) and Visual Language Models (VLMs) has paved the way for the development of new paradigms for interacting with technology. LLMs/VLMs contain broadly applicable, representational knowledge - these representations are key components in prompt-based agentic systems ^{6 7 8 9 10 11} and are crucial starting checkpoints in the training of models for various applications. Device manipulation is emerging as a significant downstream application of LLMs. These systems are capable of interacting with digital devices, such as computers and mobile phones, with the same inputs that humans provide. Mobile operating systems, such as Android,¹¹ provide a unique location for language models to interact with and assist humans. At the system level, an operating system has a cornucopia of data on a user: full of device interaction traces, user preferences, and highly informative contexts. Responsible and effective use of this information could allow LLMs to perform highly-personalized tasks on behalf of users, drastically improving the usage experience of mobile devices and driving a new paradigm of human-ai interaction.

To date, action-taking model architecture have primarily been developed with multimodal representations. These models percieve their environment through screenshots of the devices they are manipulating. On Android in particular, there is a second option available for the representing screen states, accessibility trees (UI Trees) which encode screen information purely in text. ¹² When developing on device action models, it is not clear which representation provides a better foundation for such an agentic system. Therefore in this study we investigate the key question of:

“What representational modality provides the best base for action-taking models?”

Addressing this question is essential as on-device action-taking models need to comprehensively understand the semantics of screen states in order to operate reliably while being deployed in heavily constrained computational settings.

We explore this question by training an ensemble of models suitable for on-device, Android action-taking applications and compare the characteristics of these models, trained on textual and visual screen representations. We conduct our experiments with the AndroidControl dataset¹³ and use a unified, coordinate-based action space in our primary experiments. We compare models based on their abilities to generate suitable actions in the pursuit of completing goals and in the architectural characteristics of both methods. Additionally, we examine the training dynamics of our models, finding insights related to how the choice of representation affects convergence. This study analyzes the characteristics of models for given choices of screen representational modalities.

Preliminaries

Problem setting. The development of action-taking models for the Android Operating System (OS), requires an answer to the unique question what screen representation modality is the most effective. Several action-taking models have been developed and use either textual representations, such as UI trees, or visual representations, like screenshots, to interpret device states in action-taking trajectories. However, no works study both representations and compare the effects of both against one another. As such it remains unclear which modality provides a more robust foundation for developing action models. We investigate this problem in our study and address the crucial architectural question which underpins the design of capable action-taking models.

The Dataset and Model Training. To enable our study we require 1- goal-conditioned device manipulation data with screen representations for both of the modalities of interest and 2- comparable base models suited for training action-models which percieve the environment through each of the modality choices.

To fulfill the first requirement, we leverage the AndroidControl dataset, introduced in Li, Wei, et al..¹³ This dataset contains goal-conditioned episodes of user-device interactions which were collected by human annotators. Each episode starts with an initial screen state and a goal that needs to be accomplished; the goals are then completed within episodes through a sequence of steps. For each step in an episode, the state of the device is represented with either: an Android accessibility tree (textual representation of the screen) or with a screenshot (rasterized image). Additionally, an action is applied to the device which brings the episode closer to completion and advances the episode to the next step. This dataset contains both modalities of screen representations for all steps in all of the dataset’s episodes, providing ideal pairings of textual and visual screen states for our experiment. The full AndroidControl dataset contains 15,283 episodes, and a total of 99,131 individual steps. We provide a sample from an episode with all of the attirbutes that we use in the appendix.

For specific models to train on the textual and visual modalities, we choose Llama-3.1-8B-Instruct¹⁴ and Idefics3-8B-Llama3¹⁵ respectively. These two models are derivatives of the widely used Llama 3 architecture, with Idefics3 being a variant of Llama-3.1-8B-Instruct, adapted to the multimodal setting with the integration of the SigLIP vision encoder.¹⁶ Neither of these models are trained specifically for goal-conditioned device manipulation tasks, and preliminary evaluations we conducted that showed these two models performed similarly without task-specific training on their respective modalities.

Our datasets are derived from processing the AndroidControl dataset, ¹³ tailored for the two distinct models. The models’ task is to predict the appropriate function calls for a given step within an episode. Each step-level sample comprises a screen representation $s$ (either a screenshot or a UI Tree), the sequence of prior actions within the episode $z$, a high-level goal $g$, and the ground-truth action $a$.

For the text modality, the screen representation $s$ is constructed by concatenating the language model tokens of the tokenized UI Tree. If the UI Tree comprises $t$ tokens, the screen representation is $s = [e_1, e_2, \cdots, e_t] \in \mathbb{R}^{n \times d_{text}}$, where $d_{text}$ denotes the dimensionality of the language model’s hidden size

In the visual modality, the representation $s$ consists of sequence vision encoder states projected into the language model’s embedding space. Given a device screenshot $x_{image} \in \mathbb{R}^{H \times W}$, the multimodal model’s vision encoder $f_v$ processes the image to yield hidden states $x_{features} \in \mathbb{R}^{m \times d_{v}} = f_v(x_{image})$, where $d_{v}$ is the vision encoder’s hidden dimension. These features are projected into the language model’s embedding space using the perceiver resampler $f_{p}$, ¹⁵ resulting in the representation $s \in \mathbb{R}^{m \times d_{text}} = f_\text{p}(x_{features})$. It is important to note that the number of tokens $m$ required to represent an image in the input sequence may differ from the number of tokens $t$ needed to represent the corresponding UI Tree.

For each step across all episodes, a prompt $p = (s, z, g) = (e_1, e_2, \cdots, e_n)$ of $n$ token embeddings is constructed to encapsulate the screen’s state and the episode’s context. The model is then supervised to generate the tokens for the ground-truth action $a = (a_1, a_2, \cdots, a_T)$ conditioned on $p$ by employing the standard cross-entropy loss. ¹⁷ Specifically, we optimize the model parameters $\theta$ by minimizing the loss function:

Where $T$ is the length of the sequence of tokens representing the action $a$, $C$ is the language model’s vocabulary size, $y_{t, i}$ is the correct token for at the $t$-th sequence position, and $p(a_{t, i} | a_{<t}, s, z, g; \theta)$ is the predicted probability for token $a_{t, i}$ at position $t$ under model parameters $\theta$.

Model Training Details. In this work, we finetune two models on textual and visual screen representations. Our text-only model is based on Llama-3.1-8B-Instruct¹⁴ (text-model), and is trained on UI trees. Our visual model is based on Idefics3-8B-Llama3 ¹⁵ (visual-model), and is trained on screenshots. In both cases we use LoRA¹⁸ to finetune the models – we adapt all linear layers (including the vision tower for the visual-model) with a rank of 16. We conducted independent hyperparameter sweeps for both models, searching over values of: learning rate, weight decay, warmup period, learning rate scheduler, lora alpha, and lora dropout. The final training jobs for both models use the best hyperparameters observed in the respective sweep runs, we trained both models to 30,000 optimization steps (~6.5 epochs) at an effective batch size of 16.

Experimental Setup

The goal of our study is to determine which screen representation modality serves as a better base for building capable on-device action models. Specifically, our primary interest is to characterize the difference in action-taking performance between models developed for the two representations.

Dataset and Action Space. In light of computational restrictions early on in this project, we use a subset of the AndroidControl dataset,¹³ filtering out samples with sequence lengths greater than 20,000 tokens. After filtering down the samples in the dataset, we hold out the steps from 1,451 episodes as out test set, and use the steps from the remaining 13,053 episodes as our training set for both models.

Here we use the set of actions that are captured in the AndroidControl dataset.¹³ Following, ¹³ we add a status action to the final step of all episodes, which allows the model to signify the completion of an episode with its outcome being successful or infeasible. We employ the following set of actions in our dataset:

Additionally, we provide an illustrative example of one specific sample from our dataset in the appendix.

Evaluation Metrics. As with prior works that train action models in Android environments,^{13 19 20} we compare model performance through stepwise accuracy. This metric compares model generated actions with ground truth actions from the dataset — where a correctly generated action contains the same function name and arguments. Similarly to,¹³ we relax the requirements for generated actions, marking semantically equivalent, but not exact, model generations as successful. Specifically, we permit a 3% area around a ground-truth, coordinate based action’s coordinate values. We also consider a relaxation criterion for generated strings in input_text actions. Specifically, we allow model generated values of text where the lowercased string is within a Levenshtein distance²¹ that is less than or equal to three compared to the lowercased ground truth text string.

We conduct our evaluations through inferencing model checkpoints on samples drawn from the test set and reporting the raw, relaxed stepwise accuracy scores. We provide the models with no few-shot examples and sample all completions with greedy decoding.

Additionally, we analyze the training dynamics of the two approaches. The training and validation loss are compared between both of the modality models’ training runs. Comparisons of the relative trends in these values provide insight into how each representation impacts the learning trajectory. Although trained on different modalities, the supervision between the two approaches is the same — making this relative comparison of trends for the best hyperparameter configurations informative for how the modality impacts learning.

We do not consider episodic-level accuracy in this work, as we filter out steps based on the length of the UI Trees, resulting in several episodes being non-contiguous. This is a key limitation of this work, as ultimately episodic-level accuracy is the key metric for action model performance. However in this preliminary architectural study, we believe that stepwise accuracy in conjunction with other key values provides sufficent information to determine which modality is more favorable as a base for action-taking models.

Results

Our main results are displayed in the following figure and table:

We find that for the dataset considered, the text-model has the best absolute value of stepwise accuracy of 64.76% at 21,000 optimization steps. The best performance of the vision model is just slightly behind at 63.55% and occurs later in the vision model’s training at 28,000 optimization steps. For the two best scores, we conduct a homoscedastic t-test and find that the difference between the best model performances is not statistically significant at any standard value of $\alpha \in \set{0.01, 0.05, 0.10}$.

This finding is quite interesting, as for the dataset and set of actions we consider, the screen representation’s modality does not appear to drastically impact stepwise accuracy performance. We then examined the errors for individual functions for each of the best checkpoint evaluation results:

Here we note that the visual-model tended to select functions more appropriately across the board than with text-model, whereas the text-model was more adept in its predictions for text based values (lower error rates for input_text and open_app). The click function was by far the most problematic for both models, as most errors occur in click predictions. This function is the most common in our dataset and is also the function which is most often incorrectly predicted in lieu of other ground-truth functions.

We inspect the incorrect click functions for both models and find that the difference in “misclicks”, the failed clicks due to incorrect coordinates, are quite different between these two models. The vision-model has a 1.328x smaller average absolute deviation in its predicted coordinates, when compared to the text-model, along with a smaller standard deviation. Of these misclicks, we observe two distinct failure modes in both models: completely incorrect clicks and near-misses. We include several illustrative examples of these fail cases in the Appendix.

For the completely incorrect clicks, we identified a distinct pattern wherein one coordinate (either x or y) deviated substantially from the ground-truth values, whereas the other coordinate demonstrated a high degree of accuracy, often aligning precisely or closely with the corresponding coordinate in the ground-truth values. We examined the locations of these misclicks by overlaying the predicted coordinates on the screenshots, frequently these incorrect coordinates lay on completely unrelated regions of the screen. We hypothesize that this odd abrupt deviation in the coordinate values is due to an uneven or biased distribution over the the supervised coordinate values of x and y, we leave further investigation of this failure mode and the resolution of it to future work.

For the cases of near-misses that we observe, we see that a model will predict a coordinate that is relatively close to the ground-truth, but selects a related but incorrect element on the screen or slightly misses the boundary of the ground-truth element. We note that these types of errors tend to occur in complex UIs with many elements, such as the calculator application or in the photos application.

While both representations of screens contain similar information, they represent that information in different ways. We find that near-misses for both modalities tend to occur in a manner that reflects the representation. Specifically, the text-model’s misclicks frequently select similar or related elements which share text with the ground-truth element in the UI Tree and the vision-model’s misclicks are slightly off spatially, occasionally missing the bounding box by a few pixels. We present a few samples that display these failure modes in the appendix.

Training Dynamics and Efficiency. As part of our study, we analyze the graphs corresponding to the train and validation losses. These charts are taken from the main training jobs of the text and visual models.

Some notable points on these curves are:

• The text-model appears to start overfitting to the training set as early as 20,000 optimization steps, with a pronounced difference between the training and validation losses.
• Throughout the training job, the vision-model generally had a slightly higher training loss and a far more “spiky” curve, which appears to smooth out towards the end of training.
• The vision-model’s train and validation losses closely track one another and continue to learn beyond the point where the text-model starts to overfit.
• The best observed validation loss for the text-model is .1167 at 17,000 optimization steps, whereas the vision-model’s best validation loss of 0.0263 occurs at the end of training.

We find that critically, multimodal models need less tokens to represent the same amount of information. Below we plot the token distributions for the text and visual models’ test datasets:

The distribution of input tokens required to represent a sample is significantly smaller for the visual domain than it is for the textual domain – typically 2 to 20x smaller. This significant reduction is due to using the visual encoder as a compressor of the base image. The test set accuracy demonstrates that the reduction in sequence length afforded by the compressed image representation performs on par with the UI Trees.

Conclusion

Our study finds that the representation modality of the screen states on Android devices does not have a significant effect on the stepwise accuracy of action models. However the choice does have significant implications for efficiency and training dynamics, with the visual modality offering comparable performance for far fewer tokens and avoids overfitting whilst continuing to learn. In resource constrained environments, where action models need to be effective and efficient, the visual modality presents a compelling architecture for the development of capable agentic systems on mobile devices.

Thank you to Abhay Kashyap, Nicole Fitzerald, Sudhanshu Ranjan, Daniel Bulhosa Solórzano, and Nate Harada for invaluable feedback on preliminary drafts of this post.

Appendix

Example of a Data Sample. This is one data sample from AndroidControl that we used to train the model:

• Goal, $g$: for all steps in this episode, we condition the model with this task objective: Using the ASOS app, add the adidas Training Rapidmove trainers in white to my wishlist.

• Prior Actions, $z$: for any prior steps in an episode we include the sequence of prior actions into the model’s prompt as a newline joined string. This particular sample does not have any prior actions as it is the first step in the episode.

• Screen Representation, $s$: is either the text of a UI Tree or a screenshot, the corresponding artifacts for this sample are:

  1. UI Tree:

     windows {
       bounds_in_screen {
         right: 1080
         bottom: 136
       }
       id: 7
       layer: 1
       window_type: TYPE_SYSTEM
       tree {
         nodes {
           bounds_in_screen {
             right: 1080
             bottom: 136
           }
           class_name: "android.widget.FrameLayout"
           package_name: "com.android.systemui"
           text_selection_start: -1
           text_selection_end: -1
           window_id: 7
           is_enabled: true
     
     <A lot more of these UI tree nodes, which we have truncated for brevity>
     .....
     <this ui tree ends with the following>
     
           window_id: 133
           is_enabled: true
           is_visible_to_user: true
           actions {
             id: 4
           }
           actions {
             id: 8
           }
           actions {
             id: 64
           }
           actions {
             id: 16908342
           }
           depth: 20
           drawing_order: 1
         }
       }
     }

  2. Screenshot:

     

• Action, $a$: for this sample the ground truth action is: {"action_type": "click", "X": 461, "y": 1223} — which in this case, for the screen representation and goal, the model should click on the “Heart” button.²²

Illustrative Misclicks

In each of the following figures, the ground-truth coordinates for the click are colored in green, the text-model’s predicted coordinates are colored in red and the vision model’s predictions are in blue.

Completely Incorrect

Near-Misses

Quite Positive Model Predictions

Supplementary Training Curves

Here we display the learning curves of the two models, relative to each other.

References