Observer agreement for event sequences: methods and software for sequence alignment and reliability estimates

When sequences of discrete events, or other units, are independently coded by two coders using a set of mutually exclusive and exhaustive codes, but the onset times for the codes are not preserved, it is often unclear how pairs of protocols should be aligned. Yet such alignment is required before Co...

Full description

Saved in:
Bibliographic Details
Published in:Behavior research methods 2007-02, Vol.39 (1), p.39-49
Main Authors: Quera, Vicenç, Bakeman, Roger, Gnisci, Augusto
Format: Article
Language:English
Subjects:
Agreements
Algorithms
Behavior
Codes
Computer science
Costs
Dynamic programming
Eye movements
False alarms
Humans
Life Change Events
Methods
Models, Psychological
Observer Variation
Psychology - methods
Psychology - statistics & numerical data
Software
Studies
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:When sequences of discrete events, or other units, are independently coded by two coders using a set of mutually exclusive and exhaustive codes, but the onset times for the codes are not preserved, it is often unclear how pairs of protocols should be aligned. Yet such alignment is required before Cohen's kappa, a common agreement statistic, can be computed. Here we describe a method--based on the Needleman and Wunsch (1970) algorithm originally devised for aligning nucleotide sequences--for optimally aligning such sequences; we also offer the results of a simulation study of the behavior of alignment kappa with a number of variables, including number of codes, varying degrees of observer accuracy, sequence length, code variability, and parameters governing the alignment algorithm. We conclude that (1) under most reasonable circumstances, observer accuracies of 90% or better result in alignment kappas of .60 or better; (2) generally, alignment kappas are not strongly affected by sequence length, the number of codes, or the variability in the codes' probability; (3) alignment kappas are adversely affected when missed events and false alarms are possible; and (4) cost matrices and priority orders used in the algorithm should favor substitutions (i.e., disagreements) over insertions and deletions (i.e., missed events and false alarms). Two computer programs were developed: Global Sequence Alignment, or GSA, for carrying out the simulation study, and Event Alignment, or ELign, a user-oriented program that computes alignment kappa and provides the optimal alignment given a pair of event sequences.
ISSN:1554-351X
1554-3528
DOI:10.3758/BF03192842