Order Statistics Based Estimator for Renyi's Entropy

Several types of entropy estimators exist in the information theory literature. Most of these estimators explicitly involve estimating the density of the available data samples before computing the entropy. However, the entropy-estimator using sample spacing avoids this intermediate step and compute...

Full description

Saved in:
Bibliographic Details
Main Authors: Hegde, A., Tian Lan, Erdogmus, D.
Format: Conference Proceeding
Language:English
Subjects:
Brain modeling
Entropy
Independent component analysis
Information theory
Mutual coupling
Mutual information
Order Statistics
Performance analysis
Sequences
Statistical distributions
Statistics
Online Access:Request full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Several types of entropy estimators exist in the information theory literature. Most of these estimators explicitly involve estimating the density of the available data samples before computing the entropy. However, the entropy-estimator using sample spacing avoids this intermediate step and computes the entropy directly using the order-statistics. In this paper, we extend our horizon beyond Shannon's definition of entropy and analyze the entropy estimation performance at higher orders of alpha, using Renyi's generalized entropy estimator. We show that the estimators for higher orders of alpha better approximate the true entropy for an exponential family of distributions. Practical application of this estimator is demonstrated by computing mutual information between functionally coupled systems. During the estimation process, the joint distributions are decomposed into sum of their marginals by using linear ICA
ISSN:1551-2541
2378-928X
DOI:10.1109/MLSP.2005.1532924