End-to-end data-dependent routing in multi-path neural networks

Neural networks are known to give better performance with increased depth due to their ability to learn more abstract features. Although the deepening of networks has been well established, there is still room for efficient feature extraction within a layer, which would reduce the need for mere para...

Full description

Saved in:
Bibliographic Details
Published in:Neural computing & applications 2023-06, Vol.35 (17), p.12655-12674
Main Authors: Tissera, Dumindu, Wijesinghe, Rukshan, Vithanage, Kasun, Xavier, Alex, Fernando, Subha, Rodrigo, Ranga
Format: Article
Language:English
Subjects:
Algorithms
Artificial Intelligence
Computational Biology/Bioinformatics
Computational Science and Engineering
Computer Science
Data Mining and Knowledge Discovery
Feature extraction
Image Processing and Computer Vision
Mathematical analysis
Neural networks
Original Article
Parameters
Probability and Statistics in Computer Science
Resource allocation
Tensors
Widening
Citations: Items that this one cites
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Neural networks are known to give better performance with increased depth due to their ability to learn more abstract features. Although the deepening of networks has been well established, there is still room for efficient feature extraction within a layer, which would reduce the need for mere parameter increment. The conventional widening of networks by having more filters in each layer introduces a quadratic increment of parameters. Having multiple parallel convolutional/dense operations in each layer solves this problem, but without any context-dependent allocation of input among these operations: The parallel computations tend to learn similar features making the widening process less effective. Therefore, we propose the use of multi-path neural networks with data-dependent resource allocation from parallel computations within layers, which also lets an input be routed end-to-end through these parallel paths. To do this, we first introduce a cross-prediction -based algorithm between parallel tensors of subsequent layers. Second, we further reduce the routing overhead by introducing feature-dependent cross-connections between parallel tensors of successive layers. Using image recognition tasks, we show that our multi-path networks show superior performance to existing widening and adaptive feature extraction, even ensembles and deeper networks at similar complexity.
ISSN:0941-0643
1433-3058
DOI:10.1007/s00521-023-08381-8