DialMAT: Dialogue-Enabled Transformer with Moment-Based Adversarial Training

This paper focuses on the DialFRED task, which is the task of embodied instruction following in a setting where an agent can actively ask questions about the task. To address this task, we propose DialMAT. DialMAT introduces Moment-based Adversarial Training, which incorporates adversarial perturbat...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-11
Main Authors: Kaneda, Kanta, Korekata, Ryosuke, Wada, Yuiga, Nagashima, Shunya, Kambara, Motonari, Iioka, Yui, Matsuo, Haruka, Imai, Yuto, Nishimura, Takayuki, Sugiura, Komei
Format: Article
Language:English
Subjects:
Datasets
Feature extraction
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This paper focuses on the DialFRED task, which is the task of embodied instruction following in a setting where an agent can actively ask questions about the task. To address this task, we propose DialMAT. DialMAT introduces Moment-based Adversarial Training, which incorporates adversarial perturbations into the latent space of language, image, and action. Additionally, it introduces a crossmodal parallel feature extraction mechanism that applies foundation models to both language and image. We evaluated our model using a dataset constructed from the DialFRED dataset and demonstrated superior performance compared to the baseline method in terms of success rate and path weighted success rate. The model secured the top position in the DialFRED Challenge, which took place at the CVPR 2023 Embodied AI workshop.
ISSN:2331-8422