TabNAS: Rejection Sampling for Neural Architecture Search on Tabular Datasets

The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc. Neural architecture search (NAS) for tabular datasets is an important...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2022-10
Main Authors: Yang, Chengrun, Bender, Gabriel, Liu, Hanxiao, Pieter-Jan Kindermans, Udell, Madeleine, Lu, Yifeng, Le, Quoc, Huang, Da
Format: Article
Language:English
Subjects:
Algorithms
Constraints
Datasets
Energy consumption
Machine learning
Searching
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:The best neural architecture for a given machine learning problem depends on many factors: not only the complexity and structure of the dataset, but also on resource constraints including latency, compute, energy consumption, etc. Neural architecture search (NAS) for tabular datasets is an important but under-explored problem. Previous NAS algorithms designed for image search spaces incorporate resource constraints directly into the reinforcement learning (RL) rewards. However, for NAS on tabular datasets, this protocol often discovers suboptimal architectures. This paper develops TabNAS, a new and more effective approach to handle resource constraints in tabular NAS using an RL controller motivated by the idea of rejection sampling. TabNAS immediately discards any architecture that violates the resource constraints without training or learning from that architecture. TabNAS uses a Monte-Carlo-based correction to the RL policy gradient update to account for this extra filtering step. Results on several tabular datasets demonstrate the superiority of TabNAS over previous reward-shaping methods: it finds better models that obey the constraints.
ISSN:2331-8422