Design sensitivity and statistical power in acceptability judgment experiments

Abstract

Previous investigations into the validity of acceptability judgment data have focused almost exclusively on type I errors (or false positives) because of the consequences of such errors for syntactic theories (Sprouse & Almeida 2012; Sprouse et al. 2013). The current study complements these previous studies by systematically investigating the type II error rate (false negatives), or equivalently, the statistical power, of a wide cross-section of possible acceptability judgment experiments. Though type II errors have historically been assumed to be less costly than type I errors, the dynamics of scientific publishing mean that high type II error rates (i.e., studies with low statistical power) can lead to increases in type I error rates in a given field of study. We present a set of experiments and resampling simulations to estimate statistical power for four tasks (forced-choice, Likert scale, magnitude estimation, and yes-no), 50 effect sizes instantiated by real phenomena, sample sizes from 5 to 100 participants, and two approaches to statistical analysis (null hypothesis and Bayesian). Our goals are twofold (i) to provide a fuller picture of the status of acceptability judgment data in syntax, and (ii) to provide detailed information that syntacticians can use to design and evaluate the sensitivity of acceptability judgment experiments in their own research.

Keywords

Acceptability judgments, statistical power, experimental syntax, linguistic methodology

How to Cite

Sprouse, J. & Almeida, D., (2017) “Design sensitivity and statistical power in acceptability judgment experiments”, Glossa: a journal of general linguistics 2(1): 14. doi: https://doi.org/10.5334/gjgl.236

Download

Download PDF

2585

Views

1333

Downloads

27

Citations

Share

Authors

Jon Sprouse (Department of Linguistics, University of Connecticut)
Diogo Almeida (Psychology Program, Division of Science, New York University, Abu Dhabi)

Download

Issue

Publication details

Dates

Licence

Creative Commons Attribution 4.0

Identifiers

Peer Review

This article has been peer reviewed.

File Checksums (MD5)

  • PDF: 2900770a9189179a85c12aae9bd3f83d