Code generation from a graphical user interface via attention-based encoder–decoder model

Code generation from graphical user interface images is a promising area of research. Recent progress on machine learning methods made it possible to transform user interface into the code using several methods. The encoder–decoder framework represents one of the possible ways to tackle code generat...

Full description

Saved in:
Bibliographic Details
Published in:Multimedia systems 2022-02, Vol.28 (1), p.121-130
Main Authors: Chen, Wen-Yin, Podstreleny, Pavol, Cheng, Wen-Huang, Chen, Yung-Yao, Hua, Kai-Lung
Format: Article
Language:English
Subjects:
Coders
Computer Communication Networks
Computer Graphics
Computer Science
Cryptology
Data Storage Representation
Graphical user interface
Machine learning
Multimedia Information Systems
Operating Systems
Regular Paper
User interface
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Code generation from graphical user interface images is a promising area of research. Recent progress on machine learning methods made it possible to transform user interface into the code using several methods. The encoder–decoder framework represents one of the possible ways to tackle code generation tasks. Our model implements the encoder–decoder framework with an attention mechanism that helps the decoder to focus on a subset of salient image features when needed. Our attention mechanism also helps the decoder to generate token sequences with higher accuracy. Experimental results show that our model outperforms previously proposed models on the pix2code benchmark dataset.
ISSN:0942-4962
1432-1882
DOI:10.1007/s00530-021-00804-7