Knowledge-Based Exploration for Reinforcement Learning in Self-Organizing Neural Networks

• Main Authors: Teng, Teck-Hou, Tan, Ah-Hwee
• Format: Conference Proceeding
• Language: English
• Subjects: Computing methodologies > Artificial intelligence > Distributed artificial intelligence > Multi-agent systems; Computing methodologies > Artificial intelligence > Knowledge representation and reasoning; Computing methodologies > Machine learning; Directed Exploration; Reinforcement Learning; Rule-Based System; Self-Organizing Neural Network; Theory of computation > Logic
• DOI: 10.1109/WI-IAT.2012.154

Exploration is necessary during reinforcement learning to discover new solutions in a given problem space. Most reinforcement learning systems, however, adopt a simple strategy, by randomly selecting an action among all the available actions. This paper proposes a novel exploration strategy, known as Knowledge-based Exploration, for guiding the exploration of a family of self-organizing neural networks in reinforcement learning. Specifically, exploration is directed towards unexplored and favorable action choices while steering away from those negative action choices that are likely to fail. This is achieved by using the learned knowledge of the agent to identify prior action choices leading to low $Q$-values in similar situations. Consequently, the agent is expected to learn the right solutions in a shorter time, improving overall learning efficiency. Using a Pursuit-Evasion problem domain, we evaluate the efficacy of the knowledge-based exploration strategy, in terms of task performance, rate of learning and model complexity. Comparison with random exploration and three other heuristic-based directed exploration strategies show that Knowledge-based Exploration is significantly more effective and robust for reinforcement learning in real time.