Effects of Behavioral Anchors on Peer Evaluation Reliability

This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergradu...

Full description

Saved in:
Bibliographic Details
Published in:Journal of engineering education (Washington, D.C.) D.C.), 2005-07, Vol.94 (3), p.319-326
Main Authors: Ohland, Matthew W., Layton, Richard A., Loughry, Misty L., Yuhasz, Amy G.
Format: Article
Language:English
Subjects:
assessment
Behavioral Objectives
behaviorally anchored rating scale
College students
Comparative studies
Educational evaluation
Engineering
Engineering Education
Peer Evaluation
Reliability
Student Behavior
Thinking Skills
Citations: Items that this one cites
Items that cite this one
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
cited_by cdi_FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53
cites cdi_FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53
container_end_page 326
container_issue 3
container_start_page 319
container_title Journal of engineering education (Washington, D.C.)
container_volume 94
creator Ohland, Matthew W.
Layton, Richard A.
Loughry, Misty L.
Yuhasz, Amy G.
description This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergraduate engineering classes over four years show that the use of behavioral anchors significantly improves the inter‐rater reliability of the single‐item instrument. The inter‐rater reliability (based on four raters) of the behaviorally anchored instrument was 0.78, which was not significantly higher than that of the ten‐item instrument (0.74), but it was substantially more parsimonious. The results of this study add to the body of knowledge on evaluating students' performance in teams. This is critical since the ability to function in multidisciplinary teams is a required student learning outcome of engineering programs.
doi_str_mv 10.1002/j.2168-9830.2005.tb00856.x
format article
fullrecord <record><control><sourceid>proquest_cross</sourceid><recordid>TN_cdi_proquest_journals_217949172</recordid><sourceformat>XML</sourceformat><sourcesystem>PC</sourcesystem><sourcerecordid>873131741</sourcerecordid><originalsourceid>FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53</originalsourceid><addsrcrecordid>eNqVkF9LwzAUxYMoOKffoey99aZpk1ZEmFqnUvwzFMGXkGYJa63rTLq5fXtTOvZuXm7uveecCz-ERhgCDBCeV0GIaeKnCYEgBIiDtgBIYhpsDtBgvzpEAww09SNG4BidWFsBQAqUDdBlprWSrfUa7V2ruViXjRG1N17IeWPcdOG9KGW8bC3qlWhL109VXYqirMt2e4qOtKitOtvVIXq_y95u7v38efJwM859SRJCfUExJZhIHBeKSU0ZCBYTiItEQSILSaSYSS2EAJmGSkVy5l5KtC6cKhIxGaJRn7s0zc9K2ZZXzcos3EkeYpZGKWahE130Imkaa43SfGnKb2G2HAPvYPGKd0R4R4R3sPgOFt8481Vv_i1rtf2Hkz9mmfu4AL8PKG2rNvsAYb44ZYTF_ONpwnPyOc3p7Ssn5A8YKIDx</addsrcrecordid><sourcetype>Aggregation Database</sourcetype><iscdi>true</iscdi><recordtype>article</recordtype><pqid>217949172</pqid></control><display><type>article</type><title>Effects of Behavioral Anchors on Peer Evaluation Reliability</title><source>Social Science Premium Collection (Proquest) (PQ_SDU_P3)</source><source>Wiley-Blackwell Read &amp; Publish Collection</source><source>Education Collection</source><creator>Ohland, Matthew W. ; Layton, Richard A. ; Loughry, Misty L. ; Yuhasz, Amy G.</creator><creatorcontrib>Ohland, Matthew W. ; Layton, Richard A. ; Loughry, Misty L. ; Yuhasz, Amy G.</creatorcontrib><description>This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergraduate engineering classes over four years show that the use of behavioral anchors significantly improves the inter‐rater reliability of the single‐item instrument. The inter‐rater reliability (based on four raters) of the behaviorally anchored instrument was 0.78, which was not significantly higher than that of the ten‐item instrument (0.74), but it was substantially more parsimonious. The results of this study add to the body of knowledge on evaluating students' performance in teams. This is critical since the ability to function in multidisciplinary teams is a required student learning outcome of engineering programs.</description><identifier>ISSN: 1069-4730</identifier><identifier>EISSN: 2168-9830</identifier><identifier>DOI: 10.1002/j.2168-9830.2005.tb00856.x</identifier><identifier>CODEN: JEEDEQ</identifier><language>eng</language><publisher>Oxford, UK: Blackwell Publishing Ltd</publisher><subject>assessment ; Behavioral Objectives ; behaviorally anchored rating scale ; College students ; Comparative studies ; Educational evaluation ; Engineering ; Engineering Education ; Peer Evaluation ; Reliability ; Student Behavior ; Thinking Skills</subject><ispartof>Journal of engineering education (Washington, D.C.), 2005-07, Vol.94 (3), p.319-326</ispartof><rights>2005 American Society for Engineering Education</rights><rights>Copyright American Society for Engineering Education Jul 2005</rights><lds50>peer_reviewed</lds50><woscitedreferencessubscribed>false</woscitedreferencessubscribed><citedby>FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53</citedby><cites>FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53</cites></display><links><openurl>$$Topenurl_article</openurl><openurlfulltext>$$Topenurlfull_article</openurlfulltext><thumbnail>$$Tsyndetics_thumb_exl</thumbnail><linktopdf>$$Uhttps://www.proquest.com/docview/217949172/fulltextPDF?pq-origsite=primo$$EPDF$$P50$$Gproquest$$H</linktopdf><linktohtml>$$Uhttps://www.proquest.com/docview/217949172?pq-origsite=primo$$EHTML$$P50$$Gproquest$$H</linktohtml><link.rule.ids>314,780,784,21378,21394,27924,27925,33611,33877,43733,43880,74221,74397</link.rule.ids></links><search><creatorcontrib>Ohland, Matthew W.</creatorcontrib><creatorcontrib>Layton, Richard A.</creatorcontrib><creatorcontrib>Loughry, Misty L.</creatorcontrib><creatorcontrib>Yuhasz, Amy G.</creatorcontrib><title>Effects of Behavioral Anchors on Peer Evaluation Reliability</title><title>Journal of engineering education (Washington, D.C.)</title><description>This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergraduate engineering classes over four years show that the use of behavioral anchors significantly improves the inter‐rater reliability of the single‐item instrument. The inter‐rater reliability (based on four raters) of the behaviorally anchored instrument was 0.78, which was not significantly higher than that of the ten‐item instrument (0.74), but it was substantially more parsimonious. The results of this study add to the body of knowledge on evaluating students' performance in teams. This is critical since the ability to function in multidisciplinary teams is a required student learning outcome of engineering programs.</description><subject>assessment</subject><subject>Behavioral Objectives</subject><subject>behaviorally anchored rating scale</subject><subject>College students</subject><subject>Comparative studies</subject><subject>Educational evaluation</subject><subject>Engineering</subject><subject>Engineering Education</subject><subject>Peer Evaluation</subject><subject>Reliability</subject><subject>Student Behavior</subject><subject>Thinking Skills</subject><issn>1069-4730</issn><issn>2168-9830</issn><fulltext>true</fulltext><rsrctype>article</rsrctype><creationdate>2005</creationdate><recordtype>article</recordtype><sourceid>ALSLI</sourceid><sourceid>CJNVE</sourceid><sourceid>M0P</sourceid><recordid>eNqVkF9LwzAUxYMoOKffoey99aZpk1ZEmFqnUvwzFMGXkGYJa63rTLq5fXtTOvZuXm7uveecCz-ERhgCDBCeV0GIaeKnCYEgBIiDtgBIYhpsDtBgvzpEAww09SNG4BidWFsBQAqUDdBlprWSrfUa7V2ruViXjRG1N17IeWPcdOG9KGW8bC3qlWhL109VXYqirMt2e4qOtKitOtvVIXq_y95u7v38efJwM859SRJCfUExJZhIHBeKSU0ZCBYTiItEQSILSaSYSS2EAJmGSkVy5l5KtC6cKhIxGaJRn7s0zc9K2ZZXzcos3EkeYpZGKWahE130Imkaa43SfGnKb2G2HAPvYPGKd0R4R4R3sPgOFt8481Vv_i1rtf2Hkz9mmfu4AL8PKG2rNvsAYb44ZYTF_ONpwnPyOc3p7Ssn5A8YKIDx</recordid><startdate>200507</startdate><enddate>200507</enddate><creator>Ohland, Matthew W.</creator><creator>Layton, Richard A.</creator><creator>Loughry, Misty L.</creator><creator>Yuhasz, Amy G.</creator><general>Blackwell Publishing Ltd</general><scope>BSCLL</scope><scope>AAYXX</scope><scope>CITATION</scope><scope>0-V</scope><scope>3V.</scope><scope>4T-</scope><scope>7XB</scope><scope>88B</scope><scope>88I</scope><scope>8A4</scope><scope>8AF</scope><scope>8FE</scope><scope>8FG</scope><scope>8FK</scope><scope>8G5</scope><scope>ABJCF</scope><scope>ABUWG</scope><scope>AFKRA</scope><scope>ALSLI</scope><scope>AZQEC</scope><scope>BEC</scope><scope>BENPR</scope><scope>BGLVJ</scope><scope>CCPQU</scope><scope>CJNVE</scope><scope>DWQXO</scope><scope>GNUQQ</scope><scope>GUQSH</scope><scope>HCIFZ</scope><scope>L6V</scope><scope>M0P</scope><scope>M2O</scope><scope>M2P</scope><scope>M7S</scope><scope>MBDVC</scope><scope>PQEDU</scope><scope>PQEST</scope><scope>PQQKQ</scope><scope>PQUKI</scope><scope>PRINS</scope><scope>PTHSS</scope><scope>Q9U</scope><scope>S0W</scope><scope>S0X</scope></search><sort><creationdate>200507</creationdate><title>Effects of Behavioral Anchors on Peer Evaluation Reliability</title><author>Ohland, Matthew W. ; Layton, Richard A. ; Loughry, Misty L. ; Yuhasz, Amy G.</author></sort><facets><frbrtype>5</frbrtype><frbrgroupid>cdi_FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53</frbrgroupid><rsrctype>articles</rsrctype><prefilter>articles</prefilter><language>eng</language><creationdate>2005</creationdate><topic>assessment</topic><topic>Behavioral Objectives</topic><topic>behaviorally anchored rating scale</topic><topic>College students</topic><topic>Comparative studies</topic><topic>Educational evaluation</topic><topic>Engineering</topic><topic>Engineering Education</topic><topic>Peer Evaluation</topic><topic>Reliability</topic><topic>Student Behavior</topic><topic>Thinking Skills</topic><toplevel>peer_reviewed</toplevel><toplevel>online_resources</toplevel><creatorcontrib>Ohland, Matthew W.</creatorcontrib><creatorcontrib>Layton, Richard A.</creatorcontrib><creatorcontrib>Loughry, Misty L.</creatorcontrib><creatorcontrib>Yuhasz, Amy G.</creatorcontrib><collection>Istex</collection><collection>CrossRef</collection><collection>ProQuest Social Sciences Premium Collection【Remote access available】</collection><collection>ProQuest Central (Corporate)</collection><collection>Docstoc</collection><collection>ProQuest Central (purchase pre-March 2016)</collection><collection>Education Database (Alumni Edition)</collection><collection>Science Database (Alumni Edition)</collection><collection>Education Periodicals</collection><collection>STEM Database</collection><collection>ProQuest SciTech Collection</collection><collection>ProQuest Technology Collection</collection><collection>ProQuest Central (Alumni) (purchase pre-March 2016)</collection><collection>Research Library (Alumni Edition)</collection><collection>Materials Science &amp; Engineering Collection</collection><collection>ProQuest Central (Alumni)</collection><collection>ProQuest Central</collection><collection>Social Science Premium Collection (Proquest) (PQ_SDU_P3)</collection><collection>ProQuest Central Essentials</collection><collection>eLibrary</collection><collection>ProQuest Central</collection><collection>Technology Collection</collection><collection>ProQuest One Community College</collection><collection>Education Collection</collection><collection>ProQuest Central</collection><collection>ProQuest Central Student</collection><collection>Research Library Prep</collection><collection>SciTech Premium Collection</collection><collection>ProQuest Engineering Collection</collection><collection>Education Database</collection><collection>ProQuest Research Library</collection><collection>ProQuest Science Journals</collection><collection>Engineering Database</collection><collection>Research Library (Corporate)</collection><collection>ProQuest One Education</collection><collection>ProQuest One Academic Eastern Edition (DO NOT USE)</collection><collection>ProQuest One Academic</collection><collection>ProQuest One Academic UKI Edition</collection><collection>ProQuest Central China</collection><collection>Engineering collection</collection><collection>ProQuest Central Basic</collection><collection>DELNET Engineering &amp; Technology Collection</collection><collection>SIRS Editorial</collection><jtitle>Journal of engineering education (Washington, D.C.)</jtitle></facets><delivery><delcategory>Remote Search Resource</delcategory><fulltext>fulltext</fulltext></delivery><addata><au>Ohland, Matthew W.</au><au>Layton, Richard A.</au><au>Loughry, Misty L.</au><au>Yuhasz, Amy G.</au><format>journal</format><genre>article</genre><ristype>JOUR</ristype><atitle>Effects of Behavioral Anchors on Peer Evaluation Reliability</atitle><jtitle>Journal of engineering education (Washington, D.C.)</jtitle><date>2005-07</date><risdate>2005</risdate><volume>94</volume><issue>3</issue><spage>319</spage><epage>326</epage><pages>319-326</pages><issn>1069-4730</issn><eissn>2168-9830</eissn><coden>JEEDEQ</coden><abstract>This paper presents comparisons of three peer evaluation instruments tested among students in undergraduate engineering classes: a single‐item instrument without behavioral anchors, a ten‐item instrument, and a single‐item behaviorally anchored instrument. Studies using the instruments in undergraduate engineering classes over four years show that the use of behavioral anchors significantly improves the inter‐rater reliability of the single‐item instrument. The inter‐rater reliability (based on four raters) of the behaviorally anchored instrument was 0.78, which was not significantly higher than that of the ten‐item instrument (0.74), but it was substantially more parsimonious. The results of this study add to the body of knowledge on evaluating students' performance in teams. This is critical since the ability to function in multidisciplinary teams is a required student learning outcome of engineering programs.</abstract><cop>Oxford, UK</cop><pub>Blackwell Publishing Ltd</pub><doi>10.1002/j.2168-9830.2005.tb00856.x</doi><tpages>8</tpages></addata></record>
fulltext fulltext
identifier ISSN: 1069-4730
ispartof Journal of engineering education (Washington, D.C.), 2005-07, Vol.94 (3), p.319-326
issn 1069-4730
2168-9830
language eng
recordid cdi_proquest_journals_217949172
source Social Science Premium Collection (Proquest) (PQ_SDU_P3); Wiley-Blackwell Read & Publish Collection; Education Collection
subjects assessment
Behavioral Objectives
behaviorally anchored rating scale
College students
Comparative studies
Educational evaluation
Engineering
Engineering Education
Peer Evaluation
Reliability
Student Behavior
Thinking Skills
title Effects of Behavioral Anchors on Peer Evaluation Reliability
url http://sfxeu10.hosted.exlibrisgroup.com/loughborough?ctx_ver=Z39.88-2004&ctx_enc=info:ofi/enc:UTF-8&ctx_tim=2024-12-26T14%3A01%3A39IST&url_ver=Z39.88-2004&url_ctx_fmt=infofi/fmt:kev:mtx:ctx&rfr_id=info:sid/primo.exlibrisgroup.com:primo3-Article-proquest_cross&rft_val_fmt=info:ofi/fmt:kev:mtx:journal&rft.genre=article&rft.atitle=Effects%20of%20Behavioral%20Anchors%20on%20Peer%20Evaluation%20Reliability&rft.jtitle=Journal%20of%20engineering%20education%20(Washington,%20D.C.)&rft.au=Ohland,%20Matthew%20W.&rft.date=2005-07&rft.volume=94&rft.issue=3&rft.spage=319&rft.epage=326&rft.pages=319-326&rft.issn=1069-4730&rft.eissn=2168-9830&rft.coden=JEEDEQ&rft_id=info:doi/10.1002/j.2168-9830.2005.tb00856.x&rft_dat=%3Cproquest_cross%3E873131741%3C/proquest_cross%3E%3Cgrp_id%3Ecdi_FETCH-LOGICAL-c3836-a616313c15be7cf670a75305b8e08cbc3cadcfaaa0c92ee4cdddd93ffb5304a53%3C/grp_id%3E%3Coa%3E%3C/oa%3E%3Curl%3E%3C/url%3E&rft_id=info:oai/&rft_pqid=217949172&rft_id=info:pmid/&rfr_iscdi=true