META-GUI is a dataset for training a Multi-modal convErsaTional Agent on mobile GUI. It consists of 1125 dialogues, 4684 dialogue turns and 18337 data points in total. Each data point contains screenshot history, action history, dialogue history, items appeared on the current screen and actions to be performed.
NewsOct 6, 2022 Our paper is accepted by EMNLP 2022, dataset and baseline are listed below. Dataset is available on Download from Amazon.com Baseline is available on META-GUI Baseline
Contact UsIf you have any questions about this dataset, please contact slt19990817@sjtu.edu.cn, galaxychen@sjtu.edu.cn or chenlusz@sjtu.edu.cn
Three metrics, i.e. Action Completion Rate (CR), Turn Completion Rate, and Reply BLEU score, are used to evaluate on the test set of META-GUI. Please refer to the paper to find more details about evaluation metrics.
| Rank | Model | Action CR | Turn CR | Reply BLEU score |
|---|---|---|---|---|
| 1 Oct 06, 2022 |
m-BASH
Shanghai Jiao Tong University (Sun et al., EMNLP'22) code |
82.74 | 56.88 | 63.11 |
| 2 Oct 06, 2022 |
BERT
Shanghai Jiao Tong University (Sun et al., EMNLP'22) code |
78.42 | 52.08 | 62.19 |
| 3 Oct 06, 2022 |
LayoutLM
Shanghai Jiao Tong University (Sun et al., EMNLP'22) code |
67.76 | 38.12 | 50.43 |
| 4 Oct 06, 2022 |
LayoutLMv2
Shanghai Jiao Tong University (Sun et al., EMNLP'22) code |
64.48 | 36.88 | 58.20 |