<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE article PUBLIC "-//NLM//DTD JATS (Z39.96) Journal Publishing DTD v1.0 20120330//EN" "http://jats.nlm.nih.gov/publishing/1.0/JATS-journalpublishing1.dtd">
<!--<?xml-stylesheet type="text/xsl" href="article.xsl"?>-->
<article article-type="research-article" dtd-version="1.0" xml:lang="en" xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<front>
<journal-meta>
<journal-id journal-id-type="issn">2397-1835</journal-id>
<journal-title-group>
<journal-title>Glossa: a journal of general linguistics</journal-title>
</journal-title-group>
<issn pub-type="epub">2397-1835</issn>
<publisher>
<publisher-name>Ubiquity Press</publisher-name>
</publisher>
</journal-meta>
<article-meta>
<article-id pub-id-type="doi">10.5334/gjgl.396</article-id>
<article-categories>
<subj-group>
<subject>Research</subject>
</subj-group>
</article-categories>
<title-group>
<article-title>Quantifying sentence acceptability measures: Reliability, bias, and variability</article-title>
</title-group>
<contrib-group>
<contrib contrib-type="author" corresp="yes">
<contrib-id contrib-id-type="orcid">http://orcid.org/0000-0001-6135-8389</contrib-id>
<name>
<surname>Langsford</surname>
<given-names>Steven</given-names>
</name>
<email>steven.langsford@adelaide.edu.au</email>
<xref ref-type="aff" rid="aff-1">1</xref>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Perfors</surname>
<given-names>Amy</given-names>
</name>
<xref ref-type="aff" rid="aff-2">2</xref>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Hendrickson</surname>
<given-names>Andrew T.</given-names>
</name>
<xref ref-type="aff" rid="aff-3">3</xref>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Kennedy</surname>
<given-names>Lauren A.</given-names>
</name>
<xref ref-type="aff" rid="aff-1">1</xref>
</contrib>
<contrib contrib-type="author">
<name>
<surname>Navarro</surname>
<given-names>Danielle J.</given-names>
</name>
<xref ref-type="aff" rid="aff-4">4</xref>
</contrib>
</contrib-group>
<aff id="aff-1"><label>1</label>The University of Adelaide, School of psychology, AU</aff>
<aff id="aff-2"><label>2</label>University of Melbourne, School of psychological sciences, AU</aff>
<aff id="aff-3"><label>3</label>Tilburg University, Department of cognitive science and artificial intelligence, NL</aff>
<aff id="aff-4"><label>4</label>The University of New South Wales, School of psychology, AU</aff>
<pub-date publication-format="electronic" date-type="pub" iso-8601-date="2018-03-16">
<day>16</day>
<month>03</month>
<year>2018</year>
</pub-date>
<pub-date pub-type="collection">
<year>2018</year>
</pub-date>
<volume>3</volume>
<issue>1</issue>
<elocation-id>37</elocation-id>
<history>
<date date-type="received" iso-8601-date="2017-04-03">
<day>03</day>
<month>04</month>
<year>2017</year>
</date>
<date date-type="accepted" iso-8601-date="2017-11-20">
<day>20</day>
<month>11</month>
<year>2017</year>
</date>
</history>
<permissions>
<copyright-statement>Copyright: &#x00A9; 2018 The Author(s)</copyright-statement>
<copyright-year>2018</copyright-year>
<license license-type="open-access" xlink:href="http://creativecommons.org/licenses/by/4.0/">
<license-p>This is an open-access article distributed under the terms of the Creative Commons Attribution 4.0 International License (CC-BY 4.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited. See <uri xlink:href="http://creativecommons.org/licenses/by/4.0/">http://creativecommons.org/licenses/by/4.0/</uri>.</license-p>
</license>
</permissions>
<self-uri xlink:href="http://www.glossa-journal.org/articles/10.5334/gjgl.396/"/>
<abstract>
<p>Understanding and measuring sentence acceptability is of fundamental importance for linguists, but although many measures for doing so have been developed, relatively little is known about some of their psychometric properties. In this paper we evaluate within- and between-participant test-retest reliability on a wide range of measures of sentence acceptability. Doing so allows us to estimate how much of the variability within each measure is due to factors including participant-level individual differences, sample size, response styles, and item effects. The measures examined include Likert scales, two versions of forced-choice judgments, magnitude estimation, and a novel measure based on Thurstonian approaches in psychophysics. We reproduce previous findings of high between-participant reliability within and across measures, and extend these results to a generally high reliability within individual items and individual people. Our results indicate that Likert scales and the Thurstonian approach produce the most stable and reliable acceptability measures and do so with smaller sample sizes than the other measures. Moreover, their agreement with each other suggests that the limitation of a discrete Likert scale does not impose a significant degree of structure on the resulting acceptability judgments.</p>
</abstract>
<kwd-group>
<kwd>syntax</kwd>
<kwd>acceptability</kwd>
<kwd>measurement</kwd>
<kwd>representation</kwd>
<kwd>reliability</kwd>
<kwd>power</kwd>
<kwd>sample size</kwd>
</kwd-group>
</article-meta>
</front>
<body>
<sec>
<title>1 Introduction</title>
<p>Acceptability judgments have formed a large part of the study of language since at least Chomsky (<xref ref-type="bibr" rid="B7">1965</xref>). They are one of many sources of evidence, alongside corpus linguistics (<xref ref-type="bibr" rid="B62">Sampson 2007</xref>), psychological experiments (<xref ref-type="bibr" rid="B53">Noveck &amp; Reboul 2008</xref>), and neuroscience techniques (<xref ref-type="bibr" rid="B67">Shalom &amp; Poeppel 2007</xref>), that each offer distinct and complementary information about language (<xref ref-type="bibr" rid="B1">Arppe &amp; J&#228;rvikivi 2007</xref>). One major factor in the popularity of acceptability judgments is the way they allow theories to be tested against artificial constructions that passive observation would rarely or never provide (<xref ref-type="bibr" rid="B63">Sch&#252;tze 1996</xref>). For instance, acceptability judgments can differentiate between constructions that are ungrammatical and those that are rare or missing but still grammatical.</p>
<p>Acceptability judgments come in a number of possible forms, each with their own advantages and disadvantages. The main differences between different forms are in the kind of response required from the participant. People can be offered a discrete rating scale, a real-valued scale, or be asked to make a relative comparison between items. The choice of what response options to offer is critical in two important respects: it determines the statistical tests available to researchers, and it may also significantly influence people&#8217;s interpretation of the task. For these reasons, the characteristics of different kinds of acceptability measures are well studied. We know that acceptability judgment data are influenced by details such as the selection of participants (<xref ref-type="bibr" rid="B12">D&#261;browska 2010</xref>), sample size (<xref ref-type="bibr" rid="B46">Mahowald et al. 2016</xref>), task structure (<xref ref-type="bibr" rid="B23">Featherston 2008</xref>), participant engagement (<xref ref-type="bibr" rid="B30">H&#228;ussler &amp; Juzek 2016</xref>), and data processing decisions (<xref ref-type="bibr" rid="B36">Juzek 2015</xref>).</p>
<p>Most of the existing literature focuses on the question of to what extent acceptability judgment data can be used to adjudicate about individual phenomena or effects of linguistic interest (e.g., by presenting pairs of sentences that capture a specific contrast relevant to a particular theoretical claim). However, one might be interested in evaluating the range of acceptability measures along other dimensions as well. To what extent do acceptability judgments from different elicitation tasks support claims about larger-scale generalizations across many different sentences and phenomena? To what extent do different measures of acceptability agree with each other about specific items or sentences? To what extent is each measure robust to differences within individuals at different time points? This paper focuses on exploring these questions.</p>
<p>In the work presented here, we attempt to quantify the extent to which acceptability judgment data from a variety of different elicitation tasks supports different kinds of claims: claims about the global structure of acceptability across a large set of diverse sentences, claims based on the magnitude of acceptability differences, and claims made at the level of single items or sentences. We accomplish this by quantifying the relative contribution of multiple factors &#8211; individual participant differences, sample size, task structure, and response-style mitigation in data processing &#8211; to the empirical reliability of acceptability scores over specific items (rather than over specific effects) for different measures. We chose to focus on reliability because reliability places a ceiling on how appropriate acceptability judgments are as a test of linguistic theories. If acceptability judgments for some measure or in the presence of some factor are not reliable, we should be cautious about relying on them. Moreover, understanding what factors influence the reliability of a measure can be informative about exactly what that measure reflects.</p>
<p>Our approach aims to differentiate between possible sources of bias and variance. It is currently unclear what proportion of the variability seen in acceptability judgment data is due to lapses of attention, idiolect differences between participants, differences in interpretation of acceptability scales, or interference from simultaneously presented items.</p>
<p>A standard response to the diversity of potential sources of variability is to give them all equal status as <italic>noise independent of the linguistic effect</italic> and ask what can be concluded about true linguistic effects (focused on specific phenomena) in the presence of this noise, regardless of its source. An extensive literature explores this question, looking at the chance of identifying an effect where none exists (<xref ref-type="bibr" rid="B78">Sprouse &amp; Almeida 2011</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>), the chance of failing to identify an effect that is truly present (<xref ref-type="bibr" rid="B80">Sprouse and Almeida 2017a</xref>), and differences in sensitivity of different measures compared on a particular known effect (<xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>). The consensus of such studies is that acceptability judgments are highly reliable across replications (<xref ref-type="bibr" rid="B81">Sprouse &amp; Almeida 2017b</xref>).</p>
<p>As this literature shows, differentiation between different sources of bias and variance is not strictly necessary in order to test specific linguistic <bold>effects</bold>, which are the primary currency of linguistic research. Many measures of sentence acceptability have good psychometric properties when they are used for such a purpose (e.g., testing whether a set of sentences licensed under some linguistic theory have different acceptability than a set of sentences that are not licensed). If such differentiation is not necessary, why are we attempting to do so here?</p>
<p>The first reason is that such differentiation is important if we want to use acceptability judgments to explore questions that are not focused on hypothesis testing about specific linguistic effects. For instance, it is quite possible that the nature of the elicitation task may impose structure on the <italic>overall distribution</italic> of acceptability scores across multiple kinds of sentences. Thus, understanding to what extent different tasks do this is important for investigations of the global structure of acceptability in language. Such investigations would include issues like the extent to which clustering structure may be apparent in acceptability judgments (<xref ref-type="bibr" rid="B33">Hofmeister et al. 2013</xref>; <xref ref-type="bibr" rid="B40">Lau et al. 2016</xref>), whether there are dialect or language differences in global acceptability structure, or whether low acceptability sentences show greater variability than high acceptability ones. Indeed, global acceptability judgments (if they are reliable) may even provide a means to differentiate between dialects or evaluate the knowledge or fluency of individual speakers.</p>
<p>The second reason we are interested in distinguishing between different sources of variability is the expectation that some of these sources fall under an experimenter&#8217;s control and can be minimized. Different elicitation tasks may vary in their vulnerability to particular sources of variability, which affects their relative quality as scientific instruments. In general, a task that is more difficult might be expected to incur greater variability due to distraction or mistaken responding. Tasks with a small number of unambiguous response options, such as forced choice tasks, may be less vulnerable to response style variability than tasks with flexible free response options that are open to differences of interpretation, such as magnitude estimation. Conversely, forced choice tasks may be more vulnerable to item neighborhood effects, with sentences potentially processed differently in the context of a contrast rather than in isolation.</p>
<p>How much do these tasks vary and how large are these different sources of variation? Our goal is to provide a quantitative answer to this question.</p>
<p>The many possible sources of bias and variability cannot be completely disentangled, since they are generally all present in some unknown degree in every response. We give quantitative bounds for the distinct contribution of certain sources of variability in two different ways.</p>
<p>First, we contrast between and within participant test-retest reliability. Between-participant test-retest reliability is an important metric of measure quality in its own right, since no strong conclusions can be drawn from the results of a measure if it is liable to give different answers to the same question on different occasions (<xref ref-type="bibr" rid="B39">Kline 2013</xref>; <xref ref-type="bibr" rid="B58">Porte 2013</xref>; <xref ref-type="bibr" rid="B5">Brandt et al. 2014</xref>; <xref ref-type="bibr" rid="B16">DeVellis 2016</xref>). While distinct in the way it avoids appealing to a ground truth, between-participant test-retest reliability is closely related to error-rate reliability, if the underlying truth is considered stable over the time scales involved. As such, it is widely reported in existing work on the reliability of acceptability judgment data (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>; <xref ref-type="bibr" rid="B80">Sprouse &amp; Almeida 2017a</xref>). However test-retest reliability within the same participant can offer additional information, especially when contrasted with between-participant reliability. This contrast, which has no analogue in error rates, is informative about the composition of the variability: variability inherent to the construct itself and random noise due to inattention or other error can be expected in both, while individual differences in response style and subjective acceptability only contribute to the variability of between-participant replications. As a result, between-participant replications are expected to be less reliable, and the size of the reliability gap quantifies the combined impact of these particular sources of variability. Even further decomposition into the source of this within/between reliability gap is possible as well. For instance, the variability due to response style differences can be estimated by examining the effect of data pre-processing steps (e.g., <italic>z</italic>-transformation of scores) known to mitigate this particular source of variability.</p>
<p>Second, we contrast these within and between participant test-retest reliability results for measures based on different tasks. The tasks differ primarily in the kind of response options offered, which could potentially impose structure on results. For example, asking people to give responses on a discrete Likert scale might force them to collapse distinct acceptabilities onto one response if there are too few options or encourage them to make spurious distinctions if there are too many (<xref ref-type="bibr" rid="B63">Sch&#252;tze 1996</xref>; <xref ref-type="bibr" rid="B6">Carifio &amp; Perla Likert 2008</xref>; <xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>). The comparisons involved in forced choice judgments could also direct people&#8217;s attention to specific syntactic details, particularly when the two sentences are related, as is typical of a well-controlled test pair. This might lead to different acceptability ratings than if each sentence was considered in isolation (<xref ref-type="bibr" rid="B8">Cornips &amp; Poletto 2005</xref>). Contrasts between measures are therefore useful both in identifying the best-performing measures (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>; <xref ref-type="bibr" rid="B80">Sprouse &amp; Almeida 2017a</xref>) and to test the degree of agreement between them (<xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>; <xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>; <xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>).</p>
<p>However, from the perspective of decomposing sources of bias and variance, distinct tasks may also be differently vulnerable to different sources of variability. As a result, we may be able to use them to cross-check against each other&#8217;s potential biases.</p>
<p>The structure of this paper is as follows. We first give a detailed introduction to the measures considered in this paper, the processing steps and statistical tests associated with each, and the series of experiments within which we collect the data. When reporting the results our primary focus is on test-retest reliability; it is first evaluated in terms of raw score correlation of all sentences in a dataset, then in terms of the decisions yielded by each measure on particular contrasts of interest. For each of these we compare within and between participant reliability and examine the impact of sample size. We conclude by examining the mutual agreement between the measures, with reference to expert judgments in the published literature. In the discussion, we address limitations of this work, consider recommendations for researchers interested in measuring sentence acceptability, and discuss future directions.</p>
<sec>
<title>1.1 The measures</title>
<p>Early work on the reliability of formal measures was prompted by concerns about the practice of &#8220;armchair linguistics&#8221;, which considered phrases or sentences as the primary unit of evidence on which linguistic theories were built, taking for granted that the acceptability status of these sentences would be immediately obvious to a native speaker. With reference to previously discredited introspective approaches in psychology (<xref ref-type="bibr" rid="B14">Danziger 1980</xref>), critics pointed out that the intuitions of a linguist about a sentence they constructed themselves to demonstrate a particular point of syntax might not be the same as those of the broader language community (<xref ref-type="bibr" rid="B71">Spencer 1973</xref>; <xref ref-type="bibr" rid="B63">Sch&#252;tze 1996</xref>; <xref ref-type="bibr" rid="B85">Wasow &amp; Arnold 2005</xref>; <xref ref-type="bibr" rid="B12">D&#261;browska 2010</xref>).</p>
<p>Proponents of informal approaches argued in response that linguists were mainly concerned with phenomena that gave very large effect sizes, making multiple opinions on a particular acceptability difference redundant (<xref ref-type="bibr" rid="B56">Phillips &amp; Lasnik 2003</xref>; <xref ref-type="bibr" rid="B24">Featherston 2009</xref>; <xref ref-type="bibr" rid="B55">Phillips 2009</xref>). This approach defended the legitimacy of the large literature built on such informal tests, but left open the question of how to decide what counts as an obvious case (<xref ref-type="bibr" rid="B44">Linzen &amp; Oseki 2015</xref>).</p>
<p>Recent systematic work comparing expert and naive judgments has largely supported the argument that the majority of claims published in the linguistics literature are consistent with the results of formal tests against the judgments of large numbers of naive native speakers (<xref ref-type="bibr" rid="B10">Culbertson &amp; Gross 2009</xref>; <xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>). However the same program of research has shown that even for contrasts with large effect sizes, formal tests offer more information than informal ones. As well as giving an objective measure of whether a test sentence is more or less acceptable than a control to a language community, a formal test can also give an indication of the size of the difference, and the relative acceptability of both sentences on a global acceptability scale (<xref ref-type="bibr" rid="B76">Sprouse and Sch&#252;tze 2017</xref>). It has also been argued that as a result of much productive work on large effects, smaller effects have become increasingly important to further progress (<xref ref-type="bibr" rid="B27">Gibson &amp; Fedorenko 2013</xref>; <xref ref-type="bibr" rid="B28">Gibson et al. 2013</xref>).</p>
<p>One potential drawback of formal methods is their higher cost in time and participant-hours. However, as Myers (<xref ref-type="bibr" rid="B50">2009</xref>) points out, more representative samples and quantitative replicability need not be prohibitively expensive or complicated. Moreover, cost depends in part on the measurement task as well as the question being asked. For instance, many fewer judgments are required for a forced-choice task on an &#8220;obvious&#8221; effect (<xref ref-type="bibr" rid="B46">Mahowald et al. 2016</xref>) than for answering finer-grained questions about statistical power or sensitivity (<xref ref-type="bibr" rid="B80">Sprouse &amp; Almeida 2017a</xref>).</p>
<p>Our goal in this paper was to evaluate all of the most commonly used formal measures of sentence acceptability, as well as variants on them, in order to isolate and expose the impact of task-specific assumptions. The primary distinction between existing measures is whether they ask participants to give each sentence a rating on a scale of some sort (a rating task) or make a choice between two sentences (a choice task). The two rating tasks we consider are <sc>LIKERT</sc> scales and Magnitude Estimation (ME), while the two choice tasks involve either deciding between two related sentences (T<sc>ARGET PAIRS</sc>) or two random sentences (<sc>RANDOM PAIRS</sc>). This yields four separate tasks, but for two we separately evaluate alternative statistical methods for transforming the raw results, giving six distinct measures. One task for which we consider multiple analyses is magnitude estimation, where scores can be log transformed (ME(<sc>LOG</sc>)) or both log and <italic>z</italic>-transformed (ME(<italic>z</italic>-<sc>SCORE</sc>)). The other is the judgments involving random sentence pairs, which can either be used directly or input into a T<sc>HURSTONE</sc> model based on a standard measurement approach in psychophysics.</p>
<p>The six measures, ME(<italic>z</italic>-<sc>SCORE</sc>), ME(<sc>LOG</sc>), L<sc>IKERT</sc>, T<sc>HURSTONE</sc>, T<sc>ARGET PAIRS</sc>, and <sc>RANDOM PAIRS</sc> are described in more detail in the Method section. One reason for this choice of tasks is to reflect current practice: L<sc>IKERT</sc>, T<sc>ARGCHOICE</sc>, and ME are probably the most common instruments for eliciting acceptability judgments (<xref ref-type="bibr" rid="B57">Podesva &amp; Sharma 2014</xref>). However another consideration is their diversity of assumptions. In particular, L<sc>IKERT</sc> and ME each supply a particular rating scale, while the choice tasks do not. A key contribution of this paper is the presentation of the T<sc>HURSTONE</sc> model, which allows comparisons between these perspectives by inferring scale structure from choice data (<xref ref-type="bibr" rid="B83">Thurstone 1927</xref>). The T<sc>HURSTONE</sc> model is capable of representing a wide range of latent acceptability structures: the degree of consistency between the structure inferred from choice task data and rating task data gives an indication of the extent to which the researcher-supplied scales impose structure on people&#8217;s responses.</p>
</sec>
<sec>
<title>1.2 Measure evaluation</title>
<p>In this paper we systematically investigate three criteria for evaluating each of the six measures: test-retest reliability, agreement, and robustness to sample size.</p>
<p>Measure agreement is an important check of validity for diverse measures claiming to reflect the same underlying construct. Here we are also interested in the vulnerability of different measures to different sources of noise, with the goal of allowing researchers to minimize the variability in results that are due to controllable properties of the elicitation task rather than the linguistic construct of interest. Although robustness to sample size is not directly related to the decomposition of measure variability and bias that is the main focus of this paper, we include it as important information for readers interested in the implications of this work for study design.</p>
<p>Test-retest <bold>reliability</bold> can be defined at various levels from responses (when repeating questions within-participants) to items (an aggregation of many responses) to effects (which aggregate over many theoretically-related items). Here we are primarily concerned with the item level, for several reasons. First, effect-level reliability is already well studied. Second, including only one item per effect (as we do) allows us to maximize variability across items and thus creates a much more stronger test of each <italic>measure</italic>. If a measure is highly reliable even across an extremely varied sentence set, this is more informative than finding that it is reliable along a more narrow set of stimuli. Finally, item-level reliability is not itself well-studied, yet is theoretically important: if people&#8217;s judgments about specific items are reliable for a given measure, a much wider range of theoretical claims about language are open to study with this data type.</p>
<p>The assessment of reliability depends in part on the nature of the hypothesis being tested. Some researchers might be particularly interested in a <italic>decision</italic> problem: determining whether people make different judgments for two different sentences or kinds of sentences. Others might be interested in an <italic>estimation</italic> problem, being able to accurately position sentences relative to each other on an acceptability scale. In this paper we evaluate reliability using both kinds of assessment. For a decision problem, we rely on statistical significance testing of the difference between acceptability scores produced by a particular measure for the two sentences. This allows us to precisely characterize our uncertainty in the estimate of the difference for each pair of sentences, and compare that degree of uncertainty across measures in a principled way. For estimation problems, we calculate correlations between scores from different time periods or people. Reliability at this level of detail is relevant to claims about the overall structure of acceptability, for example whether or not it exhibits strong clustering (<xref ref-type="bibr" rid="B72">Sprouse 2007</xref>).</p>
<p>A secondary factor we focus on is <bold>sensitivity to sample size.</bold> We do this by systematically repeating our reliability analyses with the judgments derived from different sample sizes of participants and comparing this to the results from the full sample. This is directly useful in estimating the sample size required for a target level of reliability in studies using these measures. It also gives an indication of how efficiently these measures are able to extract information from responses; this is useful because different methods might take different numbers of trials to produce reliable answers (<xref ref-type="bibr" rid="B42">Li et al. 2016</xref>; <xref ref-type="bibr" rid="B80">Sprouse &amp; Almeida 2017a</xref>).</p>
<p>Our final factor of interest is the <bold>agreement</bold> between measures. This is of interest not only because substantial agreement suggests that the measures reflect genuine acceptability judgments rather than superficial measure-specific behavior, but also because such agreement provides converging evidence about the nature of those judgments. Cross-measure agreement is better studied than reliability (<xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>; <xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>; <xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>), but still has not been investigated within the full array of measures we consider. It is therefore valuable as a replication and extension of previous work.</p>
</sec>
</sec>
<sec sec-type="methods">
<title>2 Method</title>
<sec>
<title>2.1 Sentences</title>
<p>In order for the comparisons to be fair, all of the measures are evaluated on the same set of sentences. Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>) selected these sentences from a subset of English data points published in <italic>Linguistic Inquiry</italic> between 2001 and 2010.</p>
<p>Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>) subdivide these sentences into 148 distinct linguistic phenomena, roughly corresponding to 150 distinct sources (with two instances where different sources discussed the same construction). Each linguistic phenomenon was then represented by multiple items (eight instances). Since our focus is not on the content of any particular linguistic claim, we selected one matched pair of acceptable/unacceptable items at random from the 150 distinct sources to create a set of 300 sentences. This decision limits our ability to make claims about the status of any particular phenomenon, since each is represented by a single item. However, our focus is on the reliability and variability inherent to specific <italic>measures</italic>, and for this the diversity of sentences is a significant advantage: it is important to evaluate them over the full range of sentence acceptability levels and effect sizes. In addition, we can also estimate the variability inherent to individual items. The full list of sentences is in Appendix A.</p>
</sec>
<sec>
<title>2.2 Measures</title>
<p>Our reliability and sample size analyses involve comparing the six different measures of sentence acceptability described above. When analyzing agreement, we additionally include informal expert judgments from the published literature (I<sc>NFORMAL</sc>). The procedures for deriving scores and significance tests for each measure are given below, followed by the details of data collection. Table <xref ref-type="table" rid="T1">1</xref> summarizes this information.</p>
<table-wrap id="T1">
<label>Table 1</label>
<caption>
<p>Method summary: we examined four different tasks, two choice tasks and two rating tasks, analyzing two of these in two different ways for a total of six distinct measures. For each of these measures, we evaluate the set of acceptability scores for all sentences (which supports comparisons using Pearson correlations) as well as decisions made on pairs of sentences (which allows us to focus on targeted contrasts between two particular sentences in a hypothesis-testing framework).</p>
</caption>
<table>
<tr>
<th align="left" style="background-color:#f3f3f4;">Task</th>
<th align="left" style="background-color:#f3f3f4;">Measure</th>
<th align="left" style="background-color:#f3f3f4;">Sentence Score</th>
<th align="left" style="background-color:#f3f3f4;">Hypothesis test</th>
</tr>
<tr>
<td align="left" colspan="4"><hr/></td>
</tr>
<tr>
<td align="left">Targeted contrasts</td>
<td align="left">Target pairs</td>
<td align="left">Proportion endorsements</td>
<td align="left">Difference of proportions</td>
</tr>
<tr>
<td align="left">Random contrasts</td>
<td align="left">Random pairs</td>
<td align="left">Proportion endorsements</td>
<td align="left">Difference of proportions</td>
</tr>
<tr>
<td align="left">Random contrasts</td>
<td align="left">Thurstone</td>
<td align="left">Mean posterior acceptability</td>
<td align="left">Credible interval</td>
</tr>
<tr>
<td align="left">Magnitude estimation</td>
<td align="left">ME (log)</td>
<td align="left">Mean of log responses</td>
<td align="left">t-test</td>
</tr>
<tr>
<td align="left">Magnitude estimation</td>
<td align="left">ME (<italic>z</italic>-score)</td>
<td align="left">Mean of <italic>z</italic>-transformed log responses</td>
<td align="left">t-test</td>
</tr>
<tr>
<td align="left">Likert rating</td>
<td align="left">Likert</td>
<td align="left">Mean of <italic>z</italic>-transformed ratings</td>
<td align="left">t-test</td>
</tr>
</table>
</table-wrap>
<sec>
<title>2.2.1 Informal</title>
<p>The I<sc>NFORMAL</sc> measure captures the binary judgments presented in the <italic>Linguistic Inquiry</italic> journal for each of the sentences in question. For each of the 150 pairs, one sentence was judged to be acceptable and one was unacceptable (as noted with a judgment diacritic like * or ? in the journal). We include this measure because of the intense interest in comparing informal and formal methods (<xref ref-type="bibr" rid="B22">Featherston 2007</xref>; <xref ref-type="bibr" rid="B48">Munro et al. 2010</xref>; <xref ref-type="bibr" rid="B51">Myers 2012</xref>; <xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>; <xref ref-type="bibr" rid="B27">Gibson &amp; Fedorenko 2013</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>), although our main focus in this paper is on evaluating the test-retest reliability and mutual consistency of the formal methods.</p>
<p>One important caveat for the interpretation of the comparison with I<sc>NFORMAL</sc> results in this paper is the fact that each phenomenon is represented here by a single example sentence, rather than the multiple items as is the usual practice for formal studies (<xref ref-type="bibr" rid="B50">Myers 2009</xref>). For our purposes (i.e., investigating item-level reliability and especially the extent to which acceptability judgment data supports tests of global structure), this feature of the item set is an advantage: to the extent that different instances of the same phenomenon have similar acceptability, using one item per phenomenon gives the maximum variability over the item set and maximum coverage over the acceptability space. However it also means that there is some risk that any specific phenomenon in question will be represented by an atypical example.</p>
<p>Since typically only a single instance of an I<sc>NFORMAL</sc> judgment is available for any particular item, test-retest reliability does not apply, so they are assessed only in terms of cross-measure agreement.</p>
</sec>
<sec>
<title>2.2.2 Likert</title>
<p>In a typical L<sc>IKERT</sc> task, each sentence is presented with a series of possible acceptability rating options. This task is widely used in the psychological literature (<xref ref-type="bibr" rid="B43">Likert 1932</xref>; <xref ref-type="bibr" rid="B29">Hartley 2014</xref>) and is generally considered fairly intuitive. L<sc>IKERT</sc> scales are one of the most widely-used formal measures of linguistic acceptability (<xref ref-type="bibr" rid="B65">Sch&#252;tze &amp; Sprouse 2014</xref>) and have been shown to substantially agree with informal judgments (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>), with experts and non-linguists coming to largely the same conclusions (<xref ref-type="bibr" rid="B10">Culbertson and Gross 2009</xref>). In our experiments, following Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>) and Mahowald et al. (<xref ref-type="bibr" rid="B46">2016</xref>), we aggregated L<sc>IKERT</sc> scores by first converting each individual participants&#8217; responses to <italic>z</italic>-scores. The acceptability score for each sentence was thus the average of all <italic>z</italic>-scores associated with it. This normalization scheme mitigates the impact of individual differences in response style.</p>
</sec>
<sec>
<title>2.2.3 Magnitude estimation</title>
<p>ME tasks were developed to estimate the relative magnitude of differences between items by supplying interval data (<xref ref-type="bibr" rid="B2">Bard et al. 1996</xref>). In this procedure, adapted from psychophysics (<xref ref-type="bibr" rid="B82">Stevens 1956</xref>), participants are given an initial reference item to calibrate their judgments and then asked to compare other items by assigning them any positive real number. Although unable to provide true ratio data as initially claimed (<xref ref-type="bibr" rid="B73">Sprouse 2008</xref>; <xref ref-type="bibr" rid="B86">Weskott &amp; Fanselow 2008</xref>; <xref ref-type="bibr" rid="B74">Sprouse 2011a</xref>), ME is still commonly used (<xref ref-type="bibr" rid="B9">Cowart 1997</xref>; <xref ref-type="bibr" rid="B37">Keller 2003</xref>; <xref ref-type="bibr" rid="B21">Featherston 2005</xref>; <xref ref-type="bibr" rid="B49">Murphy et al. 2006</xref>; <xref ref-type="bibr" rid="B34">Johnson 2011</xref>; <xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>; <xref ref-type="bibr" rid="B19">Erlewine &amp; Kotek 2016</xref>). Interpreted as a linear scaling task rather than a direct recording of people&#8217;s mental representations, it is distinct from other measures in the extreme freedom it gives for arbitrarily precise responses, although whether that extra variability actually encodes information about linguistic effects has been questioned (<xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>). ME has been shown to agree with other forms of acceptability judgment (<xref ref-type="bibr" rid="B38">Keller &amp; Asudeh 2001</xref>; <xref ref-type="bibr" rid="B87">Weskott &amp; Fanselow 2009</xref>).</p>
<p>The typical aggregation scheme for ME data in linguistics, following Bard et al. (<xref ref-type="bibr" rid="B2">1996</xref>), is to average the log of the raw scores associated with each item (<xref ref-type="bibr" rid="B69">Sorace 2010</xref>; <xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>). Originally, this was because the log transform is natural for ratio data, which is the form requested in the instructions to participants. Recent work has shown that participants are in general unable to produce responses conforming to the properties of true ratio scales (<xref ref-type="bibr" rid="B74">Sprouse 2011a</xref>), and it may in fact be impossible to do so since acceptability does not have a clearly defined zero point. We adopt the log transformation here primarily because it has historically been a standard approach for reducing the impact of the outliers typical of ME data. Moreover, other possibilities, such as trimming the data or Windsorizing, would remove information.</p>
<p>In order to evaluate the role played by response style differences, we additionally investigate the impact of also applying a <italic>z</italic>-transformation, which is sometimes recommended for ME scores for that purpose (<xref ref-type="bibr" rid="B21">Featherston 2005</xref>; <xref ref-type="bibr" rid="B78">Sprouse &amp; Almeida 2011</xref>; <xref ref-type="bibr" rid="B25">Fukuda et al. 2012</xref>). The <italic>z</italic>-transformation mitigates response style differences in two ways. First, participant ratings are scored relative to their mean rating (which compensates for individual differences in which part of the scale people use) and distances are expressed in standard deviation units (which compensates for individual differences in the range of the scale that they use).</p>
<p>By contrasting the test-retest reliability of ME data both with and without the <italic>z</italic>-transform applied, it is possible to see how effective it is in mitigating response style differences. Specifically, the <italic>z</italic>-transformation is predicted to improve reliability in the between-participant replication to a much greater extent than the within-participant replication, since response style differences are a between-participant source of variability. To the extent that it is effective, this contrast gives an indication of the degree to which variation in ME scores can be attributed to variability in people&#8217;s usage of the scale.<xref ref-type="fn" rid="n1">1</xref></p>
</sec>
<sec>
<title>2.2.4 Target pairs</title>
<p>The T<sc>ARGET PAIRS</sc> judgment task asks people to select the more acceptable sentence of two candidates specifically chosen to isolate a particular contrast of theoretical interest. This is perhaps the simplest measure. By focusing only on the differences that are of theoretical interest, this measure increases the statistical power for determining the differences within those targeted pairs, but sacrifices the ability to compare pairs to one another. The T<sc>ARGET PAIRS</sc> comparison is widely used (<xref ref-type="bibr" rid="B61">Rosenbach 2003</xref>; <xref ref-type="bibr" rid="B50">Myers 2009</xref>) and has been shown to substantially agree with informal judgments (<xref ref-type="bibr" rid="B78">Sprouse &amp; Almeida 2011</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>).</p>
<p>We consider acceptability scores in the T<sc>ARGET PAIRS</sc> task to be the proportion of times the preferred option was chosen, without distinguishing between responses indicating equal acceptability or the alternative option. Unlike the other aggregate scores, this measure does not capture global structure, since decisions regarding each pair are isolated by design. The primary outcome of interest for this measure is the outcome of the significance test of the estimated proportion (<italic>&#80;&#770;</italic>) with respect to the number of judgments (<italic>N</italic>). This was calculated by determining if the 95% confidence interval around the estimated proportion included random guessing (0.5). If the interval did not include 0.5, the null hypothesis that people did not prefer one sentence over the other was rejected. The standard formula for calculating a confidence interval around a proportion was used:
<inline-formula>
<alternatives>
<mml:math id="Eq001-mml"><mml:mrow><mml:mover accent='true'><mml:mi>p</mml:mi><mml:mo>&#x005E;</mml:mo></mml:mover><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:mo>&#x00B1;</mml:mo><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:msub><mml:mi>Z</mml:mi><mml:mrow><mml:mi>c</mml:mi><mml:mi>r</mml:mi><mml:mi>i</mml:mi><mml:mi>t</mml:mi></mml:mrow></mml:msub><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:mo>&#x00D7;</mml:mo><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:msqrt><mml:mrow><mml:mstyle scriptlevel='+1'><mml:mfrac><mml:mrow><mml:mover accent='true'><mml:mi>p</mml:mi><mml:mo>&#x005E;</mml:mo></mml:mover><mml:mo stretchy='false'>(</mml:mo><mml:mn>1</mml:mn><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:mo>&#x2212;</mml:mo><mml:mtext>&#x2009;&#x2009;</mml:mtext><mml:mover accent='true'><mml:mi>p</mml:mi><mml:mo>&#x005E;</mml:mo></mml:mover><mml:mo stretchy='false'>)</mml:mo></mml:mrow><mml:mi>N</mml:mi></mml:mfrac></mml:mstyle></mml:mrow></mml:msqrt></mml:mrow></mml:math>
<tex-math id="M2">
\documentclass[10pt]{article}
\usepackage{wasysym}
\usepackage[substack]{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage[mathscr]{eucal}
\usepackage{mathrsfs}
\usepackage{pmc}
\usepackage[Euler]{upgreek}
\pagestyle{empty}
\oddsidemargin -1.0in
\begin{document}
\[\hat p\,\, \pm \,\,{Z_{crit}}\,\, \times \,\,\sqrt {{\textstyle{{\hat p(1\,\, - \,\,\hat p)} \over N}}} \]
\end{document}
</tex-math>
<graphic xlink:href="/article/id/5005/file/66406/"/>
</alternatives>
</inline-formula> where <italic>Z<sub>crit</sub></italic> = 1.96.</p>
</sec>
<sec>
<title>2.2.5 Thurstone</title>
<p>The T<sc>HURSTONE</sc> measure, which has a long history in psychophysics (<xref ref-type="bibr" rid="B83">Thurstone 1927</xref>; <xref ref-type="bibr" rid="B60">Roberts et al. 1999</xref>; <xref ref-type="bibr" rid="B20">Fabrigar &amp; Paik 2007</xref>), is used to make inferences about the subjective perception of stimuli based on forced-choice comparison data.</p>
<p>The basic idea is to ask people to make acceptability judgments about a random subset of pairs drawn from a large set of stimuli (for us, this corresponds to asking people to give forced-choice judgments on two sentences sampled at random from the full set of 300 sentences). It is important that the pairs are random rather than the theoretically-motivated pairs as in the T<sc>ARGET PAIRS</sc> task because comparing each sentence to many others imposes strong statistical constraints on the set of possible orderings of all of the sentences (<xref ref-type="bibr" rid="B83">Thurstone 1927</xref>). Distances on the inferred acceptability scale are given meaning by the model&#8217;s mapping between acceptability differences and probability of endorsing a particular response. The observed responses constrain the plausible outcomes for responses in unobserved comparisons, assuming transitivity of acceptability. As a result, only a small subset of all possible pairs is necessary to make inferences about the acceptability of all of the sentences relative to each other. Technical details for the T<sc>HURSTONE</sc> measure are described thoroughly in Appendix B.</p>
<p>The T<sc>HURSTONE</sc> model represents the acceptability of each sentence as a distribution over an inferred acceptability scale. To derive an overall acceptability score from this measure, we simply take the mean of each distribution as representing the acceptability score for that sentence. For a decision rule corresponding to the significance test in other measures, we constructed a credible interval over the difference between sentences. The distribution of credible differences was generated by repeatedly sampling from each posterior and taking the difference of those samples. The result was considered inconclusive if the range between 0.025 and 0.975 quantiles included 0, otherwise the observed difference was considered significant.</p>
<p>The T<sc>HURSTONE</sc> model has a well-established record of performance in other domains that require inferring latent acceptability orderings, such as product preferences in marketing research (<xref ref-type="bibr" rid="B54">O&#8217;Mahony et al. 2003</xref>; <xref ref-type="bibr" rid="B18">Ennis 2016</xref>). It is also a prominent tool in the <italic>wisdom of crowds</italic> literature, where it is used to define a meaningful consensus aggregating over individual judgments that cannot be simply be averaged together (<xref ref-type="bibr" rid="B47">Miller et al. 2009</xref>; <xref ref-type="bibr" rid="B66">Selker et al. 2017</xref>). Previous work on experimental syntax methodology has identified forced choice tasks as a particularly sensitive and reliable method of eliciting acceptability judgments (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>; <xref ref-type="bibr" rid="B65">Sch&#252;tze &amp; Sprouse 2014</xref>), while noting that they are restricted in the way they give limited ordinal information about only the particular sentences involved in the contrast at hand. The T<sc>HURSTONE</sc> method retains main benefits of this task type, which are the simple unambiguous response options and the way individual items can target arbitrarily small acceptability differences, while also aggregating information over all responses to derive a real-valued acceptability score that is directly comparable over all items. By providing real-valued data on a psychologically meaningful scale (<xref ref-type="bibr" rid="B52">Nosofsky 1992</xref>; <xref ref-type="bibr" rid="B4">Borg &amp; Groenen 2005</xref>), T<sc>HURSTONE</sc> modeling draws on much of the same motivation that originally drove the adoption of ME (<xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>). By shifting the responsibility for quantifying acceptability from participants to a measurement model, it avoids problems associated with the difficulty people have using the number line in the requested way (<xref ref-type="bibr" rid="B73">Sprouse 2008</xref>; <xref ref-type="bibr" rid="B74">2011a</xref>).</p>
</sec>
<sec>
<title>2.2.6 Random pairs</title>
<p>The T<sc>HURSTONE</sc> model requires choice task data over random pairs rather than the theoretically related pairs that are usually compared in a choice task. This means that we have a dataset &#8211; the raw scores on R<sc>ANDOM PAIRS</sc> &#8211; which provides a baseline against which to compare the T<sc>HURSTONE</sc> and the T<sc>ARGET PAIRS</sc> measures. Analyzing the R<sc>ANDOM PAIRS</sc> measure may be helpful in both determining how much of the performance of the T<sc>HURSTONE</sc> measure depends on the model, as well as in quantifying the impact the choice of contrast sentence has in the T<sc>ARGET PAIRS</sc> task.</p>
<p>As in the T<sc>ARGET PAIRS</sc> measure, the proportion of trials in which a sentence was endorsed over the alternative or the both-equal option was taken as its overall acceptability score. Unlike T<sc>ARGET PAIRS</sc>, this is an estimate of global acceptability across the whole set of sentences considered, albeit a noisy measure that depends on the randomly sampled set of alternative sentences each sentence appeared with. The significance test applied was the same test of proportion equality applied to T<sc>ARGET PAIRS</sc>.</p>
</sec>
</sec>
<sec>
<title>2.3 General procedure</title>
<p>To examine within and between participant reliability, three data sets were needed, an <sc>INITIAL</sc> reference set, followed by a <sc>WITHIN PARTICIPANTS</sc> replication and a <sc>BETWEEN PARTICIPANTS</sc> replication. Participants involved in the <sc>BETWEEN PARTICIPANTS</sc> replication gave the series of acceptability judgments used in the <sc>INITIAL</sc> dataset. They then performed a short distractor task designed to interfere with their ability to remember their answers to particular items, after which they repeated the same set of acceptability judgments (in a different random order) to create the <sc>BETWEEN PARTICIPANTS</sc> data set. A second group of participants was recruited to supply the <sc>BETWEEN PARTICIPANTS</sc> data set: the same procedure was used, except that these participants did not see the distractor task or give a second set of judgments.</p>
<p>In order to keep the time commitment per participant under approximately 30 minutes, we divided the four tasks into two groups that were presented to the same set of participants, with the R<sc>ANDOM PAIRS</sc> task grouped with the L<sc>IKERT</sc> rating task, and the T<sc>ARGET PAIRS</sc> task grouped with the ME task. With these groupings each participant saw one choice task and one rating task, which minimized possible fatigue due to always making the same type of judgment or interference between similar task types. Furthermore, requiring participants to complete more than one task increases the time and attention expended between responses to identical items. This decreases the chance that responses reflect an explicit memory of the first judgment for the <sc>WITHIN PARTICIPANTS</sc> replication.</p>
<p>In the first set of trials (the first half of the experiment for the <sc>INITIAL/WITHIN PARTICIPANTS</sc> group, the entirety of the study for the <sc>BETWEEN PARTICIPANTS</sc> group) participants saw two blocks (one rating task and one choice task of 42 trials each. The order of tasks within a block was randomized for each participant, and the order of items within each task was randomized on each presentation of a block. Each block contributed 40 trials to the data analysis. The additional two questions were attention checks designed to have a clear correct answer, used only to exclude participants whose incorrect responses indicated either inattention or misunderstanding of the task (see Appendix A). Participants involved in the <sc>BETWEEN PARTICIPANTS</sc> study completed at this point, while those involved in the <sc>WITHIN PARTICIPANTS</sc> study then did the distractor task, followed by a repetition of the exact same trials, with the same task order as the initial presentation but a re-drawn random order of items within each task. No sentences were repeated in different items for any one participant. Each participant thus saw only a random subset of the 300 sentences, but across participants all sentences were seen a similar number of times.</p>
<p>The distractor task was based on a change blindness demonstration (<xref ref-type="bibr" rid="B68">Simons &amp; Rensink 2005</xref>). We chose it because it is non-linguistic and known to be a very attention-grabbing task (<xref ref-type="bibr" rid="B59">Rensink et al. 1997</xref>). During it, people were shown two images that were identical except for one difficult-to-identify discrepancy: for instance, one showed a city street in which the window of one of the buildings was present in one image and absent in the other. The images were presented sequentially and repeatedly for 800ms each with an 800 ms white mask in between. Participants were asked to identify the discrepancy and click on it. Once they had done so or thirty seconds had elapsed, they were shown another pair of alternating images. There were six such images. Because the point of this task was just to provide a break between the acceptability judgment tasks, performance was not analyzed.</p>
<p>In all conditions participants saw the same general set of instructions, shown below:</p>
<disp-quote>
<p>This study will ask you some questions about the acceptability of sentences. There&#8217;s no objective standard for what makes a phrase feel &#8220;more acceptable&#8221;, but we&#8217;re confident that you&#8217;ll know it when you see it. Some phrases are natural while others are clumsy or just plain wrong, and we expect you&#8217;ll find it pretty easy to judge how acceptable a phrase is, even across very different topics. There are two different types of question. Some of the questions will ask you to give a sentence an acceptability rating. Others will ask you to compare two sentences and say which one is more acceptable.</p>
</disp-quote>
<p>All participants were asked to answer two multiple choice questions to make sure they understood the instructions (see Appendix A) before beginning the experiment. Those who did not answer both questions correctly were returned to the instructions page and could not begin until both were answered correctly.</p>
</sec>
<sec>
<title>2.4 Task-specific procedures</title>
<sec>
<title>2.4.1 Random pairs blocks</title>
<p>In these blocks, people were presented with three vertically arranged options. Each was surrounded by a blue border under the title &#8220;Which of the two sentences is most acceptable?&#8221; The first two options were sentences randomly drawn from the full pool of 300. The third option read &#8220;These two sentences are equally acceptable.&#8221; Participants clicked on a sentence to choose it, as shown in Figure <xref ref-type="fig" rid="F1">1(b)</xref>. As in the L<sc>IKERT</sc> blocks, a progress marker indicating the item and block number was displayed, and no feedback was given.</p>
<fig id="F1">
<label>Figure 1</label>
<caption>
<p><bold>Example trials for each of the four question types.</bold> In one version of the experiment ((a) and (b)), participants saw blocks of sentences presented in random order in a L<sc>IKERT</sc> task and a choice task in which the sentences were randomly drawn from the entire sentence pool (R<sc>ANDOM PAIRS</sc>). In the other version ((c) and (d)), the blocks were either in a magnitude estimation (ME) or typical choice task in which the sentence pairs were theoretically motivated (T<sc>ARGET PAIRS</sc>). For each measure, the associated panel reflects the appearance of a typical trial.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66407/"/>
</fig>
</sec>
<sec>
<title>2.4.2 Target pairs blocks</title>
<p>These trials were exactly the same as the R<sc>ANDOM PAIRS</sc> trials in the other version of the experiment. The only difference is that the sentences were both in the pair of theoretical interest rather than randomly selected from the entire set; an example is shown in Figure <xref ref-type="fig" rid="F1">1(d)</xref>.</p>
</sec>
<sec>
<title>2.4.3 Likert blocks</title>
<p>In these blocks, on each trial people saw a single sentence surrounded by a blue border under the title &#8220;Please rate the acceptability of this sentence.&#8221; Under the sentence was a row of five unmarked buttons labeled &#8220;Bad&#8221; at the far left and &#8220;Good&#8221; on the far right, as shown in Figure <xref ref-type="fig" rid="F1">1(a)</xref>. Below this was a progress marker giving the trial and block number. Clicking any of the response buttons disabled them for 500 ms and displayed the next sentence to be judged. No feedback was given.</p>
</sec>
<sec>
<title>2.4.4 Magnitude estimation blocks</title>
<p>In these blocks, people saw six pages of seven sentences each. On each page the top of the screen contained a fixed title banner that remained in position when the page was scrolled. It consisted of some reminder instructions (&#8220;If this sentence gets an acceptability rating of one hundred &#8230; [reference sentence] &#8230; what should these get?&#8221;). The reference sentence, following Sprouse (<xref ref-type="bibr" rid="B75">2011b</xref>), was &#8220;Who said that my brother was kept tabs on by the FBI?&#8221;), and was surrounded by a black border that also contained a non-editable text box in the lower right corner that was pre-filled with the reference value 100. This was followed by the test items, which were surrounded by a blue border and contained an editable text box in the lower right corner initially containing a question mark. An example is shown in Figure <xref ref-type="fig" rid="F1">1(c)</xref>.</p>
<p>Test items were arranged vertically with seven to a page with approximately two or three test sentences visible at once on the screen and the remaining sentences visible by scrolling. Each set of seven sentences was followed by progress marker and a next button which presented a fresh set of seven sentences, with no option to return to a previously rated set. Input was restricted to positive numbers, and no feedback was given, other than a prompt to give positive number ratings in order to continue if an unparsable or empty input was detected when the next button was clicked.</p>
<p>In order to ensure that people understood the ME task, before they rated any sentences each participant practiced the task on line lengths. They were required to give ratings for six different test lines (relative to a reference line length of 100). There were five test lines presented in random order, with lengths ranging between &#215;0.01 and &#215;2.5 of the reference line. Although the exact lengths of test lines were randomized to avoid encouraging participants to only use round numbers, there was one example each of very short (length &#8764; 25% of the reference line), short (&#8764;75%), roughly equal (&#8764;125%), long (&#8764;175%), and very long (&#8764;225%) lines.<xref ref-type="fn" rid="n2">2</xref> During these practice trials there was feedback on every response, and people did not continue to the next trial until their estimates were within 30 of the correct answer. Participants successfully completing this practice were considered to have understood the process of ME.</p>
</sec>
<sec>
<title>2.4.5 Participants</title>
<p>There were four rounds of recruitment to cover the two pairs of tasks (L<sc>IKERT</sc>/T<sc>ARGET PAIRS</sc> and ME/T<sc>ARGET PAIRS</sc>) in two presentation formats (a two-session format giving <sc>INITIAL</sc> and <sc>WITHIN PARTICIPANTS</sc> data, and a single-session format giving <sc>BETWEEN PARTICIPANTS</sc> data).</p>
<disp-quote>
<p><bold>Two-session</bold> L<sc>IKERT</sc> <bold>and</bold> R<sc>ANDOM PAIRS</sc> 150 adults were recruited via Amazon Mechanical Turk. Participants were paid US$3.00 for an average of 33 minutes work. They ranged in age from 20 to 65 (mean: 34.6) and 81 of them (55%) were male. Fifteen people were excluded from the analysis: three had non-compatible browsers so their data failed to save, one reported being a non-native English speaker, and 11 gave at least one incorrect response to the attention check questions. Of the 135 remaining participants, 133 were from the US and two were from India. Three reported speaking additional languages other than English but all 135 included participants reported being English native speakers.</p>
<p><bold>Two-session</bold> ME <bold>and</bold> T<sc>ARGET PAIRS</sc> 160 adults were recruited via Amazon Mechanical Turk. Participants were paid US$4.00 for an average of 38 minutes work. They ranged in age from 19 to 66 (mean: 34.0) and 91 of them (57%) were male. Twenty-five people were excluded from the analysis: one reported being a non-native English speaker, two were found to have participated in the previous round, and 22 gave at least one incorrect response to the attention check questions. Of the 135 remaining participants, 132 were from the US, with one each from India, Chile, and Ireland. One reported speaking an additional language other than English but all 135 included participants reported being English native speakers.</p>
<p><bold>Single-session</bold> L<sc>IKERT</sc> <bold>and</bold> R<sc>ANDOM PAIRS</sc> 150 adults were recruited via Amazon Mechanical Turk. Participants were paid US$1.60 for an average of 21 minutes work. They ranged in age from 22 to 69 (mean: 34.5) and 93 of them (62%) were male. Twenty-three people were excluded from the analysis: two had participated in a previous round, four reported being non-native English speakers, and 17 gave at least one incorrect response to the attention check questions. Of the 127 remaining participants, 125 were from the US, one was from Dominica, and one was from India. Two reported speaking additional languages other than English but all 127 included participants reported being English native speakers.</p>
<p><bold>Single-session</bold> ME <bold>and</bold> T<sc>ARGET PAIRS</sc> 151 adults were recruited via Amazon Mechanical Turk. Participants were paid US$3.00 for an average of 31 minutes work. They ranged in age from 18 to 70 (mean: 34.8) and 89 of them (59%) were male. Fourteen people were excluded from the analysis: four reported being non-native English speakers, and 10 gave at least one incorrect response to the attention check questions. Of the 137 remaining participants, 135 were from the US with one participant from Canada and one from the United Kingdom. Three reported speaking additional languages other than English but all 137 included participants reported being English native speakers.</p>
</disp-quote>
</sec>
</sec>
</sec>
<sec>
<title>3 Results</title>
<p>We begin by examining the test-retest reliability of the scores derived from each measure. For these analyses, we use the Pearson correlation between scores drawn from the relevant data sets: <sc>INITIAL</sc> and <sc>WITHIN PARTICIPANTS</sc> for <bold>within participant reliability</bold> or <sc>INITIAL</sc> and <sc>BETWEEN PARTICIPANTS</sc> for <bold>between participant reliability.</bold> Reliability at this level of detail may be required to test claims involving comparisons over more than two items, such as whether or not acceptability exhibits strong clustering, or claims expressed in terms of the degree of difference between items rather than the binary presence or absence of a difference (<xref ref-type="bibr" rid="B70">Sorace &amp; Keller 2005</xref>; <xref ref-type="bibr" rid="B28">Gibson et al. 2013</xref>).</p>
<sec>
<title>3.1 Global measures</title>
<sec>
<title>3.1.1 Reliability</title>
<p>We quantify the global reliability of a measure across different data sets using the Pearson correlation between acceptability estimates. Correlations between scores obtained between scores based on the <sc>INITIAL</sc> dataset and those based on the <sc>WITHIN PARTICIPANTS</sc> replication data are shown in Figure <xref ref-type="fig" rid="F2">2</xref>, with the score based on <sc>INITIAL</sc> responses on the x-axis and scores based on <sc>WITHIN PARTICIPANTS</sc> replication on the y-axis. The strong linear relationships obtained show that all measures were highly reliable. Test-retest correlations were large and statistically significant for every measure. L<sc>IKERT</sc> scores and T<sc>ARGET PAIRS</sc> were the most reliable measures.</p>
<fig id="F2">
<label>Figure 2</label>
<caption>
<p><bold>Within-participant reliability measured by correlations between sentence acceptability rankings.</bold> All of the formal measures aggregate responses into an acceptability score for each sentence. For each, the <italic>x</italic>-axis reflects the score using that measure in the <sc>INITIAL</sc> data, while the <italic>y</italic>-axis reflects the score from that measure in the <sc>WITHIN PARTICIPANTS</sc> data. The <italic>r</italic> values indicate Pearson&#8217;s correlation coefficient, and the stars (*) indicate significance at <italic>p &lt;</italic> 0.001. All measures are both highly linear and highly significant, suggesting that all these measures have good within-participant reliability.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66408/"/>
</fig>
<p>Correlations between scores obtained in <sc>INITIAL</sc> data and <sc>BETWEEN PARTICIPANTS</sc> data are shown in Figure <xref ref-type="fig" rid="F3">3</xref>, with scores obtained from the <sc>INITIAL</sc> data on the x-axis and scores obtained from the <sc>BETWEEN PARTICIPANTS</sc> replication on the y-axis. As in the <sc>WITHIN PARTICIPANTS</sc> case, all measures were highly reliable, with all correlations large and statistically significant. However, each correlation is somewhat lower than the within-participant counterpart. This extra variation must be driven by those factors unique to the <sc>BETWEEN PARTICIPANTS</sc> case: either individual differences among the participants in the two participant pools or item effects due to the re-drawing of the items shown to participants (within-participants tests used identical items each time).</p>
<fig id="F3">
<label>Figure 3</label>
<caption>
<p><bold>Between-participant reliability measured by correlations between sentence acceptability rankings.</bold> All of the formal measures aggregate responses into an acceptability score for each sentence. For each, the <italic>x</italic>-axis reflects the ranking derived using that measure in the <sc>INITIAL</sc> data. The <italic>y</italic>-axis reflects the score from that measure in the <sc>BETWEEN PARTICIPANTS</sc> data. The <italic>r</italic> values indicate Pearson&#8217;s correlation coefficient, and the stars (*) indicate significance at <italic>p &lt;</italic> 0.001. Between-participant reliability is naturally lower than within-participant reliability for all measures, but the relationship between scores derived from the two data sets are still linear and highly significant. All these measures show good between-participant reliability.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66409/"/>
</fig>
<p>Given that all the measures seem to be relatively reliable, we next seek to test whether the relative differences in reliability can be considered significant. One way to test the significance of the differences observed between these correlations is to bootstrap 95% intervals around them. We used the R package <italic>boot</italic> (<xref ref-type="bibr" rid="B15">Davison &amp; Hinkley 1997</xref>) to generate adjusted bootstrap percentile intervals (BCa) around the <italic>r</italic><sup>2</sup> estimate of variance explained in re-test scores given only scores from the <sc>INITIAL</sc> data set, assuming linearity.</p>
<p>Results are shown in Figure <xref ref-type="fig" rid="F4">4</xref>. There is a significant difference in correlations between within-participant and between-participant reliability for the ME and R<sc>ANDOM PAIRS</sc> measures but not the other ones.</p>
<fig id="F4">
<label>Figure 4</label>
<caption>
<p><bold>Comparing reliability correlations with bootstrapped <italic>r</italic><sup>2</sup>.</bold> The correlations indicating score reliability were compared across scores and across between/within contrasts by bootstrapping a 95% BCa interval. These intervals are of the <italic>r</italic><sup>2</sup> for the linear model predicting scores in the second data set (<sc>WITHIN PARTICIPANTS</sc> or <sc>BETWEEN PARTICIPANTS</sc>) from scores derived from the <sc>INITIAL</sc> dataset, and are based on 1000 samples. The results show that ME scores and R<sc>ANDOM PAIRS</sc> scores are significantly impacted by participant and item effects where the other scores are not. T<sc>ARGET PAIRS</sc> is the single most reliable measure. Of the measures allowing global comparisons, the most reliable is L<sc>IKERT</sc>.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66410/"/>
</fig>
<p>The relatively large drop in correlation when moving to from within to between-participant data for the ME scores is most likely driven by individual differences in response styles, as shown by the large reduction in this gap when mitigating response style differences using ME(<italic>z</italic>-score). In contrast, there is limited scope for response style differences in the R<sc>ANDOM PAIRS</sc> task, so the large drop in reliability when moving from within to between-participants data is likely to reflect the fact that new items were drawn, which gave each sentence a different set of comparison sentences.</p>
<p>What is most noteworthy about these effects is the fact that the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> scores do not appear to suffer from them. Despite L<sc>IKERT</sc> ratings being vulnerable in theory to response style differences, these results suggest they do not appear to be a major source of variation in practice. Although the T<sc>HURSTONE</sc> acceptability estimates are based on exactly the same responses the R<sc>ANDOM PAIRS</sc> endorsement proportions derive from, they do not show strong item effects, which is a testament to the robust nature of the T<sc>HURSTONE</sc> approach.</p>
</sec>
<sec>
<title>3.1.2 Sample size dependence</title>
<p>The analyses so far yield estimates of between- and within-participant reliability of global sentence acceptability judgments for each measure, but all involve our full sample of judgments derived from all included participants. Although even this quantity of judgments is relatively cheap and straightforward using platforms such as Amazon Mechanical Turk, it is important to understand how robust reliability is when sample sizes are lower. By repeatedly dropping some subset of participants at random from the full sample and re-running all analyses on the retained participants only, we obtained estimates for the number of participants required for a given level of reliability up to the level achieved in the full sample. These required-sample-size estimates are directly useful for researchers planning future studies, and also give an indication of how efficiently each measure extracts information from its input. All measures can be expected to asymptote to some maximum level of reliability given the underlying variability of responses, with more efficient measures approaching this maximum more quickly.</p>
<p>We explored sample size by performing a sub-sampling procedure in which only a subset of participants were drawn without replacement<xref ref-type="fn" rid="n3">3</xref> from the total population (of around 150) at sample sizes ranging from 30 to 120 in increments of 10. Only the subset of judgments was used to derive the reliability estimates. We carried out 30 repetitions of the sub-sampling procedure at each sample size and averaged them to estimate the reliability measures at that sample size. Although this smooths out variation associated with the random choice of participants retained, it does not fully reflect the variability expected at each sample size because the repetitions cannot be totally independent. Especially when the sub-sample is a large proportion of the full sample, there is extensive overlap in the data retained across iterations. Sub-sampling was also constrained to only allow samples where every item appeared at least once so that an acceptability score was always computable for each sentence and the targeted comparisons were guaranteed to be feasible.</p>
<p>As Figure <xref ref-type="fig" rid="F5">5</xref> shows, reliability decreased for every measure with decreasing sample size, but less reliable measures also showed larger decreases and the drops were higher for between-participant reliability. Reassuringly, the relative ordering of measures did not change and most became reasonably close to their performance on the full dataset at samples between 50 and 100 people. These results also suggest that the most reliable measures are most efficient, as they approach their maximum reliability more quickly in the number of responses.</p>
<fig id="F5">
<label>Figure 5</label>
<caption>
<p><bold>Global reliability measured by Pearson correlation.</bold> Pearson correlations (y-axis) between acceptability estimates based on different data sets were used to quantify the reliability of each measure over different sample sizes (x-axis). Results to the right of the dotted vertical bar are based on the full sample, results to the left are averages of 30 samples of the designated size drawn from the full sample. For all measures, smaller sample sizes are less reliable, but gains in reliability from increasing sample size become progressively smaller. <bold>Within-participant reliability</bold> (left panel) shows variability in estimates based on responses from the same people to the same items, and can be interpreted as variability deriving from the difficulty of the task and the inherently probabilistic nature of people&#8217;s responses. <bold>Between-participant reliability</bold> (right panel) is subject to the same sources of noise plus individual differences and variability introduced when re-drawing the items presented, so contrasting within and between participant reliability indicates the vulnerability of each measure to these extra factors. These results show that R<sc>ANDOM PAIRS</sc> and ME scores are particularly vulnerable to participant and item differences, and that T<sc>ARGET PAIRS</sc> and L<sc>IKERT</sc> ratings are consistently most reliable.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66411/"/>
</fig>
</sec>
<sec>
<title>3.1.3 Discussion</title>
<p>Overall, all of the measures have high test-retest reliability, especially L<sc>IKERT</sc>, T<sc>HURSTONE</sc>, and T<sc>ARGET PAIRS</sc>; the most reliable judgments are obtained by T<sc>ARGET PAIRS</sc>. This task is unusual in not offering acceptability scores that are comparable across all sentences: of the measures that do offer global comparisons, L<sc>IKERT</sc> scores are most reliable. R<sc>ANDOM PAIRS</sc> and the two ME scores were the least reliable. Contrasting the within and between-participant <italic>r</italic><sup>2</sup> values suggests one possible reason: these scores are particularly vulnerable to individual response style or item effects. Of these two possibilities, individual differences in response style is likely to be the major contributing factor for ME, as shown by the way the <italic>z</italic>-transformation improves reliability and reduces the gap between within and between participant reliability. The R<sc>ANDOM PAIRS</sc> measure is more likely to be showing item effects. There is little scope for response style differences in a choice task, but the measure is clearly sensitive to the changing identity of the alternative choices, which were re-drawn for the new participants.</p>
<p>In principle, L<sc>IKERT</sc> scales are also vulnerable to response style differences, and since the T<sc>HURSTONE</sc> scores are based on the same input as random-pairs they are exposed to the same item effects. However both measures include protection against these influences: <italic>z</italic>-transformation in the case of L<sc>IKERT</sc> scores and the inferred latent scale for the T<sc>HURSTONE</sc> scores. These results suggest that in practice these protections are effective.</p>
<p>Examining subsets of participants shows that the relative reliability of the different measure types does not change with sample size, and that the most reliable measures were also the least impacted by the number of participants. For the sentences considered here, reliability scores approached their maximum values at approximately 100 participants, which with 40 trials per participant and 300 items corresponds to an average of 13.3 trials per item. The degree of variability in responses might be expected to vary with the particular sentences used, so this relation between reliability and number of trials per item holds only to the extent that the sentences considered here represent a typical range of acceptability for research targets.</p>
</sec>
</sec>
<sec>
<title>3.2 Decision measures</title>
<p>Global reliability is useful when testing claims applicable to diverse collections of sentences, but some hypotheses are most naturally tested with targeted contrasts between particular pairs of sentences. Does each measure yield the same <italic>decision</italic> about which item of a pair is more acceptable? This sort of targeted comparison can expose changes in acceptability due to a particular syntactic manipulation while controlling for other factors like length, plausibility, and complexity. The global scores discussed above do allow pair-wise contrasts based simply on the difference between two acceptability scores, but for targeted contrasts researchers would typically conduct a measure-specific significance test instead. These are preferable because they take full advantage of a researcher&#8217;s knowledge of the test structure to appropriately characterize the variability associated with the acceptability estimates, which in turn offers control of the long-run Type 1 error rate.</p>
<p>Of course, if a researcher&#8217;s primary goal was to evaluate a particular theoretical claim, they would present participants with multiple item pairs that all instantiate the syntactic manipulation of interest, rather than the one-item-per-effect that we have evaluated here. It is nevertheless interesting for us to evaluate the decision reliability of <italic>items</italic> as we do here, for several reasons. First, if items are highly reliable across tests or people, that is both noteworthy and highly reassuring about whether <italic>effects</italic> might also be reliable. Second, looking at item-level decision reliability is still informative about the overall reliability of each measure, and can tell us about the sources of variability within each measure.</p>
<p>We thus investigate the reliability of each of the measures with regard to the decisions a researcher would draw based on a significance test for a contrast of interest. The particular significance tests we used differ for each measure as described in the <italic>Measures</italic> section in Table <xref ref-type="table" rid="T1">1</xref>: some involve two-tailed independent sample t-tests, while for others the structure of the data requires more complex analyses. We consider only the 150 targeted contrasts used in the targeted pairs task, reflecting the particular linguistic phenomena under investigation in the original <italic>Linguistic Inquiry</italic> articles. Since we are interested in the reliability of decisions rather than the content of any particular decision, we did not control for multiple comparisons in any of these tests, mimicking the situation that would obtain if each contrast was being studied independently. As in the previous analysis, we contrast within and between-participant reliability.</p>
<p>We examine three outcomes relevant to the test-retest reliability and sensitivity of such decisions: the number of <italic>inconsistent</italic> decisions across time points, the number of those inconsistencies which involve decision <italic>reversals</italic>, and the <italic>null decision count</italic>. Each individual decision admits three possible outcomes: option A is more acceptable, option B is more acceptable, or the null hypothesis of no difference cannot be rejected. For each measure, we evaluate the number of decisions (of 150 pairs) which were <bold>inconsistent</bold> (i.e., at one time option A was selected by the measure but at the other time either option B or the null was). An inconsistent measure indicates that an error of some sort (either Type 1 or Type 2) was made at some point, but it in most cases it is impossible to determine what kind of error it was. Flipping from option A to option B, which we call a <bold>reversal</bold> indicates a Type 1 error, and is quite rare: no measure produces a reversal on the full dataset. Flipping from a null to a non-null result could be either a Type 1 error (if the non-null result was incorrect) or a Type 2 error (if the null result was incorrect).</p>
<p>An indication of the sensitivity of a measure is given by the number of <bold>null decisions</bold> (i.e., the measure was unable to reject the null hypothesis in the <sc>INITIAL</sc> data set): it would be possible for a measure to never produce an inconsistent decision, but only because it was unable to ever reject the null hypothesis, which would not be a very interesting measure.</p>
<p>The raw numbers of inconsistent and null decisions are shown in Table <xref ref-type="table" rid="T2">2</xref>. No significant differences in the number of inconsistent measures was found either within participants (&#967;<sup>2</sup>(5) = 10.933, <italic>p</italic> = 0.0527) or between participants (&#967;<sup>2</sup>(5) = 8.0842, <italic>p</italic> = 0.15), but there were significant differences in the number of null decisions (&#967;<sup>2</sup>(5) = 37.35, <italic>p</italic> &lt; 0.001). In particular, T<sc>ARGET PAIRS</sc> had notably fewer null responses: it was a more sensitive measure. There were no reversals across data sets for any measure in the full sample, but some did occur at smaller sample sizes.</p>
<table-wrap id="T2">
<label>Table 2</label>
<caption>
<p><bold>Decision reliability measured by agreement on targeted contrasts.</bold> The reliability of each measure was quantified based on the number of decisions, out of a 150 total, that suggested different conclusions at different time points. All inconsistent decisions here were significant at one time point and null in the other: sign reversals appeared only at smaller sample sizes. To test if high reliability was based on insufficient power resulting in consistent null decisions, the total number of null results is also shown. There was no significant difference in the number of inconsistent decisions across measures for within- or between- participant datasets, however there was a significant difference in the number of null decisions, with the T<sc>ARGET PAIRS</sc> measure showing the fewest null decisions.</p>
</caption>
<table>
<tr>
<th align="left" style="background-color:#f3f3f4;"></th>
<th align="center" style="background-color:#f3f3f4;">Likert</th>
<th align="center" style="background-color:#f3f3f4;">ME (<italic>z</italic>-score)</th>
<th align="center" style="background-color:#f3f3f4;">ME (log)</th>
<th align="center" style="background-color:#f3f3f4;">Thurstone</th>
<th align="center" style="background-color:#f3f3f4;">Random pairs</th>
<th align="center" style="background-color:#f3f3f4;">Target pairs</th>
</tr>
<tr>
<td align="left" colspan="7"><hr/></td>
</tr>
<tr>
<td align="center" colspan="7">Inconsistent decisions</td>
</tr>
<tr>
<td align="left">Within</td>
<td align="right">11</td>
<td align="right">15</td>
<td align="right">18</td>
<td align="right">12</td>
<td align="right">26</td>
<td align="right">13</td>
</tr>
<tr>
<td align="left">Between</td>
<td align="right">17</td>
<td align="right">21</td>
<td align="right">28</td>
<td align="right">18</td>
<td align="right">27</td>
<td align="right">15</td>
</tr>
<tr>
<td align="center" colspan="7">Null decisions</td>
</tr>
<tr>
<td align="left">Initial</td>
<td align="right">24</td>
<td align="right">35</td>
<td align="right">44</td>
<td align="right">31</td>
<td align="right">49</td>
<td align="right">11</td>
</tr>
</table>
</table-wrap>
<sec>
<title>3.2.1 Sample size dependence</title>
<p>As in the previous analysis, we examined the impact of sample size on decision reliability by analyzing 30 random subsets of the full sample for each increment of 10 participants between 30 and 120. Results are shown in Figure <xref ref-type="fig" rid="F6">6</xref>. As expected, within-participant reliability is generally higher than between-participant reliability. T<sc>ARGET PAIRS</sc> is highly reliable and less sensitive to sample size, although at sample sizes of 75 and above it enjoys no particular advantage over the most reliable global scores, L<sc>IKERT</sc> and T<sc>HURSTONE</sc>. A similar sample size analysis for the null results is shown in Figure <xref ref-type="fig" rid="F7">7</xref>, whose left panel indicates the number of sentence pairs for which each measure concluded the evidence was insufficient to reject the null in the <sc>INITIAL</sc> data set.</p>
<fig id="F6">
<label>Figure 6</label>
<caption>
<p><bold>Decision reliability measured by agreement on targeted contrasts.</bold> The reliability of each measure was quantified as the number of decisions resulting in inconsistent outcomes (y-axis) tested across various sample sizes (x-axis). Outcomes were considered inconsistent if a contrast was considered null at one time point but significant at another, or the two results were significant in opposite directions: either case guarantees an error, although the first scenario is ambiguous as to error type. Results to the right of the dotted vertical bar are based on the full sample, results to the left are averages of 30 samples of the designated size drawn from the full sample. For all measures, smaller sample sizes are less reliable. Within-participant reliability (left panel) shows variability in estimates based on responses from the same people to the same items, and can be interpreted as variability deriving from the difficulty of the task and the inherently probabilistic nature of people&#8217;s responses. Between-participant reliability (right panel) is subject to the same sources of noise plus individual differences and variability introduced when re-drawing the selection of items presented. Contrasting within and between participant reliability indicates the vulnerability of each measure to these extra factors. Between-participant reliability also gives a direct measure of how well acceptability scores might be expected to replicate. These results show that R<sc>ANDOM PAIRS</sc> and ME scores are particularly vulnerable to participant and item differences, and that T<sc>HURSTONE</sc> and L<sc>IKERT</sc> scores are consistently reliable and approach the reliability of the T<sc>ARGET PAIRS</sc> method.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66412/"/>
</fig>
<fig id="F7">
<label>Figure 7</label>
<caption>
<p><bold>Sensitivity tests by sample size.</bold> The number of non-significant differences declared by each decision rule (left panel) and the number of times significant effects appear to reverse between samples (right panel). Each measure evaluated the same 150 contrasts between target-pairs of interest. Results to the right of the dotted vertical bar are based on the full sample, results to the left are averages of 30 samples of the designated sample size drawn from the full sample. T<sc>ARGET PAIRS</sc> stands out as rejecting the null most often, and at a similar rate across a range of sample sizes. Taken together, these results highlight the unique properties of the T<sc>ARGET PAIRS</sc> measure, which almost always arrives at a decision even at small sample sizes, but with decisions that may not be stable under repeated measurement. Decision reversals are shown for the between-participants test, the within-participants analogue shows the same qualitative pattern with roughly half the number of decision reversals at each sample size for T<sc>ARGET PAIRS</sc>. The other measures are less sensitive in the sense of producing more null decisions, but more conservative in the way repeated samples only ever disagree on the magnitude of an effect, never its sign. Among these conservative measures, L<sc>IKERT</sc> ratings are the most likely to detect acceptability differences.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66413/"/>
</fig>
<p>Overall, then, T<sc>ARGET PAIRS</sc> appears to be both highly reliable and extremely sensitive, yielding relatively few inconsistent decisions (Figure <xref ref-type="fig" rid="F6">6</xref>) even with very low numbers of null decisions. That said, the main drawback of the measure is evident upon comparing the effect of sample size on the number of decision <bold>reversals</bold> across measures: the number of decisions on which the measure indicates one option was significantly more acceptable at one time but the same measure indicates that the <italic>other</italic> option was significantly more acceptable at the other time. Reversals are one kind of inconsistency, but are singled out here because they reflect a larger and more consequential difference than inconsistencies that involve being unable to reject the null in one sample but not another. As the right panel of Figure <xref ref-type="fig" rid="F7">7</xref> indicates, only T<sc>ARGET PAIRS</sc> yields decision reversals. Especially at small sample sizes, it may show a statistically significant preference for one item of a pair only to prefer the <italic>other</italic> item with more data. Thus, its sensitivity comes at a cost &#8211; being more likely to completely flip in the direction of a statistically significant judgment.</p>
</sec>
<sec>
<title>3.2.2 Discussion</title>
<p>The relative reliability of the targeted contrast decisions derived from each measure is consistent with the ordering observed in the score correlation analysis, with T<sc>ARGET PAIRS</sc> and L<sc>IKERT</sc> the most reliable measures. The consistency of decisions is however distinct from the consistency of the underlying scores because of the way it depends on the threshold for rejecting the null. When the null hypothesis is true, the number of agreements across replications is completely determined by the alpha level of the test, but in the presence of a real effect it also depends on the sensitivity of the measure and the true effect size. In this study the effect sizes are constant across measures, so differences between measures reflect their sensitivity. A measure&#8217;s sensitivity ultimately depends on the information content of responses, and the extent to which information is lost by the process of aggregating responses to produce an acceptability score. Unlike the alpha level, this is not a property of the decision rule and can only be estimated empirically. It is determined by the allowable range of variability in responses and the extent to which observed variability is systematic. An ideal measure would be high on both, but the two properties conflict in the sentence acceptability context to the extent that increasing the flexibility of response options makes the task more difficult. The different tasks considered here represent different trade-offs in intuitive ease-of-use (helping participants respond systematically) and expressiveness (widening the range of response options). As previous authors have noted (<xref ref-type="bibr" rid="B86">Weskott &amp; Fanselow 2008</xref>; <xref ref-type="bibr" rid="B25">Fukuda et al. 2012</xref>), the greater expressiveness of ME&#8217;s free responses appears to be offset by increases in unsystematic variation. As in the score correlations discussed above, contrasting between and within participant reliability suggests that this extra noise is introduced by individual participants&#8217; idiosyncratic use of the scale, and can be mitigated by <italic>z</italic>-transformation. L<sc>IKERT</sc> appears to be effective in the compromise it achieves between allowing variability in responding and constraining unsystematic variation.</p>
<p>T<sc>ARGET PAIRS</sc> was found to achieve a very high power on the limited contrasts it considers, in that it arrived at fewer null decisions and was relatively insensitive to sample size. The risk of producing significant results in the wrong direction, as shown by the decision reversals, is a consequence of this high power along with the fact that controlling Type 1 error rates does not entail controlling error magnitudes. The design of the decision rule used allows for the possibility of significant findings in the wrong direction so long as the rate of such outcomes obeys the specified limits (<xref ref-type="bibr" rid="B26">Gelman &amp; Tuerlinckx 2000</xref>; <xref ref-type="bibr" rid="B11">Cumming and Maillardet 2006</xref>).</p>
<p>The practice of testing several instances of any one phenomenon of interest (<xref ref-type="bibr" rid="B64">Sch&#252;tze 2011</xref>) provides protection against these potential sign errors, since measurement error will be randomly distributed across individual items. The cost is inflation of the item set size, which may be a relatively minor burden compared to increasing the number of participants to the levels required for comparable power with a more conservative measure such as L<sc>IKERT</sc> or T<sc>HURSTONE</sc>.</p>
<p>Although item effects may be driving the higher level of sign errors for T<sc>ARGET PAIRS</sc>, it is unlikely that they are responsible for the relatively greater number of null decisions yielded by our L<sc>IKERT</sc> measure than were reported in other work (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>; <xref ref-type="bibr" rid="B31">H&#228;ussler et al. 2016</xref>; <xref ref-type="bibr" rid="B46">Mahowald et al. 2016</xref>). On closer examination, it probably emerges because our dataset involved fewer responses per effect. For instance, the Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>) results are based on 12 or 13 responses per item, with 8 items per phenomenon giving roughly 100 responses per effect. Our data involve approximately 20 responses per item (with some variability due to the random draw of items), but since there is only one item per phenomenon this is also 20 responses for each effect. With the smaller number of responses there are naturally more null decisions. Despite our much smaller Ns per item, the fact that we still had relatively few null decisions is overall reassuring, especially in light of the fact that a researcher investigating a specific effect would test multiple items.</p>
</sec>
</sec>
<sec>
<title>3.3 Agreement between measures</title>
<p>Having investigated the within and between-participant reliability of all of our measures, a natural question is whether they give the same answers as <italic>each other</italic>. Indeed, the question of how well different measures compare to informal judgments (as published in scholarly journals or textbooks) is much of the focus of related research in the literature (<xref ref-type="bibr" rid="B22">Featherston 2007</xref>; <xref ref-type="bibr" rid="B48">Munro et al. 2010</xref>; <xref ref-type="bibr" rid="B51">Myers 2012</xref>; <xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>; <xref ref-type="bibr" rid="B27">Gibson &amp; Fedorenko 2013</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>).</p>
<p>Here we build on that by also incorporating comparisons to the I<sc>NFORMAL</sc> measure of acceptability, though that is not our main focus. Instead, we explore a much wider set of comparison measures between all of the formal measures. We examine both global structure and decision agreement, using the same tests of agreement between measures as the reliability analyses, but for agreement between measures rather than across data sets.</p>
<p>Figure <xref ref-type="fig" rid="F8">8</xref> summarizes these results, presenting both correlations between acceptability estimates and the percentage of decisions (out of 150 total) where the measures arrived at the same conclusions. The scores used in this analysis were from the <sc>INITIAL</sc> dataset for L<sc>IKERT</sc> and T<sc>ARGET PAIRS</sc> measures, and from the <sc>BETWEEN PARTICIPANTS</sc> data for T<sc>HURSTONE</sc>, R<sc>ANDOM PAIRS</sc>, and ME scores. This avoids comparing unrelated measures on data derived from the same participants, potentially inflating their agreement. Related measures (the two versions of ME, or the two analyses of the random pairs choice task) are derived from the same data sets, so any disagreements between them are consequences of the different analysis only.</p>
<fig id="F8">
<label>Figure 8</label>
<caption>
<p><bold>Agreement between different measures.</bold> The lower left of this table presents Pearson correlation as a measure of global agreement between scores. The upper right presents the percentage of targeted contrasts for which different measures arrived at the same conclusion (A is more acceptable, B is more acceptable, or the evidence is inconclusive). The exception to this format is the informal expert judgment measure of acceptability, which presents only the number of decisions where informal judgments were consistent with the formal measure, the number that were inconsistent, and the number of decisions where the formal measure found the evidence inconclusive. Overall inter-measure agreement is strong. Among the formal measures, the highest agreement obtains between the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> scores. Agreement with the informal judgments is also high, with T<sc>ARGET PAIRS</sc> notable as the only measure indicating an appreciable number of contrary conclusions.</p>
</caption>
<graphic xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="/article/id/5005/file/66414/"/>
</fig>
<sec>
<title>3.3.1 Discussion</title>
<p>There was substantial agreement between measures. Measures based on the same responses (T<sc>HURSTONE</sc>/R<sc>ANDOM PAIRS</sc> and ME(<sc>LOG</sc>)/ME(<italic>z</italic>-score)) were highly correlated (<italic>r</italic> &#8776; 0.96). Between measures based on different responses, correlations ranged between 0.74 and 0.95, with the highest agreement appearing between the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> measures. This highest level of agreement is comparable to the reliability between participants for these measures, indicating that switching from one measure to the other introduces no more variation than using the same measure twice, despite substantial differences in the presentation of the task.</p>
<p>All measures are largely consistent with the informal judgments. Almost all differences were observed in the expected direction, although on average 23% of these differences were deemed too small to be considered statistically significant. The measures were also found to be highly consistent with existing quantitative studies of these items, with the Likert acceptability scores reported here correlating at <italic>r</italic> = .94 with those obtained by Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>). Although item-level agreement is high, the rate of null decisions is higher here than previously reported for these measures in Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>). This is primarily because the decisions reported here are based on more responses per item but many fewer responses per effect, due to the way each effect was represented by a single sentence pair.</p>
<p>Although these design decisions limit conclusions about any particular effect, because they were constant over measures, contrasts can still be drawn between the measures. In particular, this comparison highlights a striking difference between T<sc>ARGET PAIRS</sc> and the other measures, as it identified 14 contrasts in the opposite of the predicted direction, a disagreement rate of 9.3%. There are several possible explanations for these inconsistencies. It is possible that they represent measurement errors, although the sub-sampling analysis suggests that it is unlikely that re-measurement at the sample size considered here would ever be expected to reverse more than one decision. More likely is that in most cases these are genuine effects counter to the predicted direction, but they are extremely small and T<sc>ARGET PAIRS</sc> is the only measure with enough power to identify them. This interpretation is supported by the fact that most of these contrary decisions by T<sc>ARGET PAIRS</sc> are identified as null results by the other measures.</p>
<p>Measurement error seems unlikely for cases where two different measures agree with each other and informal judgments, so there may also be instances where contrasting highly similar sentences directs people&#8217;s attention to features of the sentences that are less salient when the two items are presented separately. One example of such an effect is the pair of sentences <italic>There are leaves burnt</italic> and <italic>There are leaves green</italic>. This pair of sentences, constructed by Sprouse et al. (<xref ref-type="bibr" rid="B77">2013</xref>), follows a pattern designed to demonstrate that (English) passives have event arguments (<xref ref-type="bibr" rid="B3">Basilico 2003</xref>). The option endorsed in the <sc>INITIAL</sc> T<sc>ARGET PAIRS</sc> data is <italic>There are leaves green</italic>.<xref ref-type="fn" rid="n4">4</xref> One possible explanation for this counterintuitive result is that it arises from people recognizing that the sentences are identical except for the words <italic>burnt</italic> and <italic>green</italic>, then responding to a strong association between <italic>green</italic> and <italic>leaves</italic>. The interpretation that the choice is somehow induced by the particular contrast of these two sentences is supported by the way the measures that do not present the two sentences together agree that <italic>There are leaves burnt</italic> is the more acceptable option. The practice of using multiple items targeting each phenomenon under study would be an effective defense against this kind of potentially misleading result, since it depends heavily on the <italic>green/leaves</italic> association, which would be unlikely to have an analogue in other sentences targeting this particular passive construction. Across all three data sets collected, only 19 contrasts are involved in a decision result conflicting with expert judgment at any point. These are presented in Appendix A.</p>
</sec>
</sec>
</sec>
<sec>
<title>4 Summary and conclusions</title>
<p>Our main focus in this work is the test-retest reliability survey of the most common tasks used to measure sentence acceptability. All tasks considered here showed high reliability, with even the least reliable measure, T<sc>ARGET PAIRS</sc>, producing large positive correlations across re-test data sets. By contrasting within-participant reliability with between-participant reliability on the same sentences with the same measures, we estimated what proportion of the variability observed can be attributed to factors unique to the between-participant replication. In all cases between-participant reliability was lower, and this reliability drop was particularly pronounced for ME and R<sc>ANDOM PAIRS</sc>, suggesting these measures are particularly vulnerable to variability across people or how items are paired together. The T<sc>ARGET PAIRS</sc> and L<sc>IKERT</sc> ratings showed not only the highest within-participant reliability but also had the least amount of decrease in reliability when comparing between- to within-participant correlations. This pattern is a hallmark of well-calibrated measurement instruments.</p>
<p>Secondly, we ask to what extent acceptability estimates depend on the particular assumptions of each measurement tool, and whether the conclusions a researcher would reach would change based on the measurement task they used. Here we find high consistency between measures, including near-uniform agreement with expert judgment. The least accurate global score (R<sc>ANDOM PAIRS</sc>) was still highly correlated (<italic>r</italic> &#8776; <italic>.9</italic>) with the most accurate global score (L<sc>IKERT</sc>). Where disagreements occurred between the measures, it was usually in the magnitude rather than the direction of the difference, with the less reliable scores more likely to not reject the null for closely matched pairs.</p>
<p>This overall consistency is striking given the structural differences between these tasks, especially between the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> tasks. Both these measurement tasks incorporate strong assumptions, and in different domains have not always agreed with each other (<xref ref-type="bibr" rid="B60">Roberts et al. 1999</xref>; <xref ref-type="bibr" rid="B17">Drasgow et al. 2010</xref>).</p>
<p>Specifically, the assumptions made by the L<sc>IKERT</sc> task center around people&#8217;s interpretation of the scale, which may impose structure on responses (<xref ref-type="bibr" rid="B63">Sch&#252;tze 1996</xref>; <xref ref-type="bibr" rid="B6">Carifio &amp; Perla 2008</xref>) or be vulnerable to differences in response style (<xref ref-type="bibr" rid="B41">Lee et al. 2002</xref>; <xref ref-type="bibr" rid="B35">Johnson 2003</xref>). The T<sc>HURSTONE</sc> measure avoids these issues by removing the researcher-supplied scale and forcing a discrete choice, but instead assumes transitivity of acceptability, which is known to be violated in similar preference-ranking tasks (<xref ref-type="bibr" rid="B84">Tversky 1969</xref>). Such violations have been observed in sentence acceptability judgments (<xref ref-type="bibr" rid="B13">Danks &amp; Glucksberg 1970</xref>; <xref ref-type="bibr" rid="B32">Hindle &amp; Ivan 1975</xref>).</p>
<p>A core contribution of this paper is that these measures provide converging evidence in the domain of sentence acceptability: theoretically motivated concerns about the restrictions a fixed L<sc>IKERT</sc> response scale imposes on participants turn out not to matter in practice, with the scale-free T<sc>HURSTONE</sc> measure based on choice task data arriving at essentially identical acceptability estimates. Although the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> acceptability scores agree, L<sc>IKERT</sc> scores are marginally more reliable and have the advantage of more easily accepting additional sentences into an existing set of comparisons.</p>
<p>Despite the close agreement between measures, T<sc>ARGET PAIRS</sc> stands out as having noteworthy decision reliability. It showed the highest power, yielding very few null results, but as a result was also the only measure vulnerable to complete reversals of a significant decision. This pattern is characteristic of high-powered tests, where significant differences observed under high-noise/low information conditions tend to entail exaggerated estimates of effect size (<xref ref-type="bibr" rid="B45">Loken and Gelman 2017</xref>). While T<sc>ARGET PAIRS</sc> is the highest performing measure in terms of test-retest consistency, and maintains this performance at small sample sizes, the relatively few errors it produces at low sample sizes can be of a qualitatively different and potentially much more misleading kind. Relatedly, the T<sc>ARGET PAIRS</sc> measure had by far the highest disagreement with the informal expert ratings of any measure, endorsing the informally dispreferred sentence on 14 of the items (9.3%) while the other measures endorsed at most two. When using the T<sc>ARGET PAIRS</sc> measure it is critical for researchers to include multiple pairs of target sentences within the same construct to increase decision reliability.</p>
<p>We find that ME tasks produce acceptability scores that are consistent with the other measures but somewhat less reliable. Contrasting the within and between participant test-retest reliability shows that this greater variability is likely to be due to variation in participant response styles, which appears as noise in the final measure. This source of variability can be mitigated somewhat by processing the scores using a transformation sensitive to response style, such as the <italic>z</italic>-transform. However, this is less effective than offering restricted responses in the task itself, as the L<sc>IKERT</sc> and T<sc>HURSTONE</sc> measures do. In general, although ME measures performed overall better than we expected them to, they were still consistently inferior to most of the alternatives.</p>
<p>Although we expect these results to be indicative of the relative test-retest reliability of these measures, the particular reliability results we observed can be expected to depend to some extent on factors such as the specific sentences and the number of trials per participant, which were controlled across measures to ensure the comparisons were fair. For the rating tasks, reliability can be expected to be a function of the number of trials per item, so the analysis over participant sample sizes gives some indication of how reliability might be expected to change with different sentence set sizes. The situation is less clear for the T<sc>HURSTONE</sc> and R<sc>ANDOM PAIRS</sc> measures, which may be sensitive to the diversity of contrasts presented as well as the average number of presentations per sentence. By choosing to hold the set of sentences constant we ensured that each measure was tested on the same range of effect sizes, but this does limit the generalizability of our reliability results. However, we believe these 150 sentences are representative of the kinds of sentences commonly used for sentence grammaticality judgments.</p>
<p>Although individually these measures make a range of assumptions that could be considered strong limitations, the high agreement between them suggests that these measure-specific assumptions do not have a strong impact on acceptability judgments. We find that if multiple items targeting the same contrast are used, none of the methods considered here have an appreciable chance of giving a strongly misleading result (although there are differences in efficiency, with ME measures requiring more trials for any given level of reliability).</p>
<p>While we find that the most common measurement tasks are all reasonably effective, the L<sc>IKERT</sc> task performed especially well. In addition to achieving relatively high test-retest reliability, our results also suggest that the L<sc>IKERT</sc> measure admits a stronger interpretation of sentence acceptability scores than is usually attributed to it. Our findings suggest that the interpretation of L<sc>IKERT</sc> data need not be constrained by concerns that the limited response scale may impose structure on the data, or that the subjective distance between response options is unknown and may vary between people. The structure suggested by the L<sc>IKERT</sc> data is in high agreement with the structure suggested by the T<sc>HURSTONE</sc> measure. Since the latter is both agnostic about the underlying structure of acceptability and capable of recovering various clustered or gradient but non-linear distributions of acceptability, this high agreement suggests that the nature of the L<sc>IKERT</sc> scale is not significantly shaping the structure of acceptability judgments it yields. The minimal difference between within-participant test-retest reliability and between-participant test-retest reliability suggests that the <italic>z</italic>-transformation offers effective protection against potential differences in the interpretation of the scale.</p>
<p>One interesting aspect of our results hinges on the fact that our dataset involved only one item per effect. This was intentional since it thus made the item set maximally variable and offered a stronger test of each measure. Our results indicating that many of these measures can reliably reflect <italic>global</italic> acceptability, rather than just effect-level acceptability, is gratifying and reassuring. It is also interesting that our <italic>item</italic>-level reliability is so high, differing from other work measuring effect-level reliability primarily in yielding slightly higher numbers of null decisions at lower sample sizes (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>; <xref ref-type="bibr" rid="B31">H&#228;ussler et al. 2016</xref>; <xref ref-type="bibr" rid="B46">Mahowald et al. 2016</xref>). Aside from this, we found item-level reliability that was nearly as good as effect-level reliability incorporating multiple items. Taken together with the high item-level variability observed around effects in other work (<xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>), this may suggest that people are surprisingly consistent on specific items but that the effect-level phenomena within any given item can at least sometimes be obscured by lexical choices or other superficial differences between sentences.</p>
<p>In terms of design recommendations for researchers interested in efficiently obtaining results that replicate with high confidence, we replicate previous results pertaining to the reliability of effects defined as ordinal relationships between sentence classes and extend them to include recommendations for ensuring the reliability of distances between individual items. We reproduce here both the general finding that acceptability judgments are highly reliable in between-participant replications (<xref ref-type="bibr" rid="B79">Sprouse &amp; Almeida 2012</xref>; <xref ref-type="bibr" rid="B77">Sprouse et al. 2013</xref>), and also more detailed claims such as the high power of T<sc>ARGET PAIRS</sc> (<xref ref-type="bibr" rid="B65">Sch&#252;tze &amp; Sprouse 2014</xref>; <xref ref-type="bibr" rid="B76">Sprouse &amp; Almeida 2017</xref>), the lack of extra information in the extra variability of ME ratings (<xref ref-type="bibr" rid="B88">Weskott &amp; Fanselow 2011</xref>), and the qualitative relationship between decision reliability and sample size (<xref ref-type="bibr" rid="B46">Mahowald et al. 2016</xref>). We further show that these reliability results extend to estimation analyses, with a high correlation in the acceptability scores assigned by different tasks to different sentences.</p>
<p>Overall, our work demonstrates that formal acceptability results are even more informative than previously realized. They agree substantially with each other (as well as informal measures) across the global structure of acceptability, not just individual targeted sentence pairs. Moreover, the best-performing measures (like L<sc>IKERT</sc> and T<sc>HURSTONE</sc>) appear not to impose substantial structure of their own onto the pattern of acceptability responses. This licenses us to use acceptability judgments to address a wider variety of questions that we have previously been able &#8211; from identifying dialectical or language differences (or possibly even individual fluency) using acceptability judgments, to investigating the global structure of grammatical knowledge (e.g., is it all-or-none or multi-dimensional?). Not all of these questions may pan out, but it is exciting to think that the formal tools we have developed for evaluating targeted sentence pairs may have something to say about them as well.</p>
</sec>
<sec sec-type="supplementary-material">
<title>Additional Files</title>
<p>The additional files for this article can be found as follows:</p>
<supplementary-material id="S1" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://doi.org/10.5334/gjgl.396.s1">
<!--[<inline-supplementary-material xlink:title="local_file" xlink:href="gjgl-3-396-s1.pdf">gjgl-3-396-s1.pdf</inline-supplementary-material>]-->
<label>Appendix A</label>
<caption>
<p>Corpus details. DOI: <uri>https://doi.org/10.5334/gjgl.396.s1</uri></p>
</caption>
</supplementary-material>
<supplementary-material id="S2" xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="https://doi.org/10.5334/gjgl.396.s1">
<!--[<inline-supplementary-material xlink:title="local_file" xlink:href="gjgl-3-396-s1.pdf">gjgl-3-396-s1.pdf</inline-supplementary-material>]-->
<label>Appendix B</label>
<caption>
<p>Details of the T<sc>HURSTONE</sc> model. DOI: <uri>https://doi.org/10.5334/gjgl.396.s1</uri></p>
</caption>
</supplementary-material>
</sec>
</body>
<back>
<fn-group>
<fn id="n1"><p>We also ran all of these analyses with raw judgments (no transformations at all), judgments receiving only the <italic>z</italic>-transform (rather than <italic>z</italic> and log), or judgments that were converted to ranks. None had superior reliability than L<sc>IKERT</sc> or T<sc>HURSTONE</sc>, and judgments that did not incorporate some way of taming outliers did not produce meaningful results.</p></fn>
<fn id="n2"><p>The ME specific instructions were:</p>
<p>Some of the questions will ask you to compare the acceptability of each sentence to a standard reference sentence and tell us the result as a number. The standard reference sentence always has an acceptability rating of 100. A sentence that is twice as good should get a rating that is twice as large, a sentence that is half as good should get a rating that is half as large, and so on. Any positive number is a valid rating, please do try to use a wide range of numbers. More detailed responses carry more information about how acceptable you feel the sentences are, and that&#8217;s really what we&#8217;re interested in. Having said that, you don&#8217;t need to spend a lot of time doing a deep analysis of every little detail, we&#8217;re much more interested in your first impressions.</p></fn>
<fn id="n3"><p>Other work (<xref ref-type="bibr" rid="B80">Sprouse &amp; Almeida 2017a</xref>) draws samples with replacement for similar analyses, but unlike our work, their items were organized into lists, preserving an even distribution across people. Because our participants all rated different sets of sentences, sampling people multiple times greatly distorts the distribution of items within the dataset in a way that they would never be distorted had that been the target sample size.</p></fn>
<fn id="n4"><p>This result re-appears in the <sc>WITHIN PARTICIPANTS</sc> replication, but not the <sc>BETWEEN PARTICIPANTS</sc> replication, which is null.</p></fn>
</fn-group>
<ack>
<title>Acknowledgements</title>
<p>All human research reported here complied with Adelaide University&#8217;s ethics requirements. SL and LK were supported through the provision of an Australian Government Research Training Program Scholarship. DN received salary support from ARC grant FT110100431, AP from ARC grants DP110104949 and DP150103280, and ATH from ARC grants DP110104949 and DE120102378. The authors would like to thank Dr Kleanthes Grohmann for his helpful comments on this work, and Jon Sprouse for permission to make quantitative comparisons with previous research data.</p>
</ack>
<sec>
<title>Competing Interests</title>
<p>The authors have no competing interests to declare.</p>
</sec>
<ref-list>
<ref id="B1"><label>1</label><mixed-citation publication-type="journal"><string-name><surname>Arppe</surname>, <given-names>Antti</given-names></string-name> &amp; <string-name><given-names>Juhani</given-names> <surname>J&#228;rvikivi</surname></string-name>. <year>2007</year>. <article-title>Every method counts: Combining corpus-based and experimental evidence in the study of synonymy</article-title>. <source>Corpus Linguistics and Linguistic Theory</source> <volume>3</volume>(<issue>2</issue>). <fpage>131</fpage>&#8211;<lpage>159</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/CLLT.2007.009</pub-id></mixed-citation></ref>
<ref id="B2"><label>2</label><mixed-citation publication-type="journal"><string-name><surname>Bard</surname>, <given-names>Ellen Gurman</given-names></string-name>, <string-name><given-names>Dan</given-names> <surname>Robertson</surname></string-name> &amp; <string-name><given-names>Antonella</given-names> <surname>Sorace</surname></string-name>. <year>1996</year>. <article-title>Magnitude estimation of linguistic acceptability</article-title>. <source>Language</source>, <fpage>32</fpage>&#8211;<lpage>68</lpage>. DOI: <pub-id pub-id-type="doi">10.2307/416793</pub-id></mixed-citation></ref>
<ref id="B3"><label>3</label><mixed-citation publication-type="journal"><string-name><surname>Basilico</surname>, <given-names>David</given-names></string-name>. <year>2003</year>. <article-title>The topic of small clauses</article-title>. <source>Linguistic Inquiry</source> <volume>34</volume>(<issue>1</issue>). <fpage>1</fpage>&#8211;<lpage>35</lpage>. DOI: <pub-id pub-id-type="doi">10.1162/002438903763255913</pub-id></mixed-citation></ref>
<ref id="B4"><label>4</label><mixed-citation publication-type="book"><string-name><surname>Borg</surname>, <given-names>Ingwer</given-names></string-name> &amp; <string-name><given-names>Patrick J. F.</given-names> <surname>Groenen</surname></string-name>. <year>2005</year>. <source>Modern multidimensional scaling: Theory and applications</source>. <publisher-loc>Berlin</publisher-loc>: <publisher-name>Springer Science &amp; Business Media</publisher-name>.</mixed-citation></ref>
<ref id="B5"><label>5</label><mixed-citation publication-type="journal"><string-name><surname>Brandt</surname>, <given-names>Mark J.</given-names></string-name>, <string-name><given-names>Hans</given-names> <surname>IJzerman</surname></string-name>, <string-name><given-names>Ap</given-names> <surname>Dijksterhuis</surname></string-name>, <string-name><given-names>Frank J.</given-names> <surname>Farach</surname></string-name>, <string-name><given-names>Jason</given-names> <surname>Geller</surname></string-name>, <string-name><given-names>Roger</given-names> <surname>Giner-Sorolla</surname></string-name>, <string-name><given-names>James A.</given-names> <surname>Grange</surname></string-name>, <string-name><given-names>Marco</given-names> <surname>Perugini</surname></string-name>, <string-name><given-names>Jeffrey R.</given-names> <surname>Spies</surname></string-name> &amp; <string-name><given-names>Anna Van&#8217;t</given-names> <surname>Veer</surname></string-name>. <year>2014</year>. <article-title>The replication recipe: What makes for a convincing replication?</article-title> <source>Journal of Experimental Social Psychology</source> <volume>50</volume>. <fpage>217</fpage>&#8211;<lpage>224</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.jesp.2013.10.005</pub-id></mixed-citation></ref>
<ref id="B6"><label>6</label><mixed-citation publication-type="journal"><string-name><surname>Carifio</surname>, <given-names>James</given-names></string-name> &amp; <string-name><given-names>Rocco</given-names> <surname>Perla</surname></string-name>. <year>2008</year>. <article-title>Resolving the 50-year debate around using and misusing Likert scales</article-title>. <source>Medical Education</source> <volume>42</volume>(<issue>12</issue>). <fpage>1150</fpage>&#8211;<lpage>1152</lpage>. DOI: <pub-id pub-id-type="doi">10.1111/j.1365-2923.2008.03172.x</pub-id></mixed-citation></ref>
<ref id="B7"><label>7</label><mixed-citation publication-type="book"><string-name><surname>Chomsky</surname>, <given-names>Noam</given-names></string-name>. <year>1965</year>. <source>Aspects of the theory of syntax</source>. <publisher-loc>Cambridge, MA</publisher-loc>: <publisher-name>MIT Press</publisher-name>.</mixed-citation></ref>
<ref id="B8"><label>8</label><mixed-citation publication-type="journal"><string-name><surname>Cornips</surname>, <given-names>Leonie</given-names></string-name> &amp; <string-name><given-names>Cecilia</given-names> <surname>Poletto</surname></string-name>. <year>2005</year>. <article-title>On standardising syntactic elicitation techniques (part 1)</article-title>. <source>Lingua</source> <volume>115</volume>(<issue>7</issue>). <fpage>939</fpage>&#8211;<lpage>957</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2003.11.004</pub-id></mixed-citation></ref>
<ref id="B9"><label>9</label><mixed-citation publication-type="book"><string-name><surname>Cowart</surname>, <given-names>Wayne</given-names></string-name>. <year>1997</year>. <source>Experimental syntax: Applying objective methods to sentence judgments</source>. <publisher-loc>Thousand Oaks, CA</publisher-loc>: <publisher-name>Sage Publications</publisher-name>.</mixed-citation></ref>
<ref id="B10"><label>10</label><mixed-citation publication-type="journal"><string-name><surname>Culbertson</surname>, <given-names>Jennifer</given-names></string-name> &amp; <string-name><given-names>Steven</given-names> <surname>Gross</surname></string-name>. <year>2009</year>. <article-title>Are linguists better subjects?</article-title> <source>The British Journal for the Philosophy of Science</source> <volume>60</volume>(<issue>4</issue>). <fpage>721</fpage>&#8211;<lpage>736</lpage>. DOI: <pub-id pub-id-type="doi">10.1093/bjps/axp032</pub-id></mixed-citation></ref>
<ref id="B11"><label>11</label><mixed-citation publication-type="journal"><string-name><surname>Cumming</surname>, <given-names>Geoff</given-names></string-name> &amp; <string-name><given-names>Robert</given-names> <surname>Maillardet</surname></string-name>. <year>2006</year>. <article-title>Confidence intervals and replication: Where will the next mean fall?</article-title> <source>Psychological Methods</source> <volume>11</volume>(<issue>3</issue>). <fpage>217</fpage>. DOI: <pub-id pub-id-type="doi">10.1037/1082-989X.11.3.217</pub-id></mixed-citation></ref>
<ref id="B12"><label>12</label><mixed-citation publication-type="journal"><string-name><surname>D&#261;browska</surname>, <given-names>Ewa</given-names></string-name>. <year>2010</year>. <article-title>Naive v. expert intuitions: An empirical study of acceptability judgments</article-title>. <source>The Linguistic Review</source> <volume>27</volume>(<issue>1</issue>). <fpage>1</fpage>&#8211;<lpage>23</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/tlir.2010.001</pub-id></mixed-citation></ref>
<ref id="B13"><label>13</label><mixed-citation publication-type="journal"><string-name><surname>Danks</surname>, <given-names>Joseph H.</given-names></string-name> &amp; <string-name><given-names>Sam</given-names> <surname>Glucksberg</surname></string-name>. <year>1970</year>. <article-title>Psychological scaling of linguistic properties</article-title>. <source>Language and Speech</source> <volume>13</volume>(<issue>2</issue>). <fpage>118</fpage>&#8211;<lpage>138</lpage>. DOI: <pub-id pub-id-type="doi">10.1177/002383097001300204</pub-id></mixed-citation></ref>
<ref id="B14"><label>14</label><mixed-citation publication-type="journal"><string-name><surname>Danziger</surname>, <given-names>Kurt</given-names></string-name>. <year>1980</year>. <article-title>The history of introspection reconsidered</article-title>. <source>Journal of the History of the Behavioral Sciences</source> <volume>16</volume>(<issue>3</issue>). <fpage>241</fpage>&#8211;<lpage>262</lpage>. DOI: <pub-id pub-id-type="doi">10.1002/1520-6696(198007)16:3&lt;241::AID-JHBS2300160306&gt;3.0.CO;2-O</pub-id></mixed-citation></ref>
<ref id="B15"><label>15</label><mixed-citation publication-type="book"><string-name><surname>Davison</surname>, <given-names>Anthony C.</given-names></string-name> &amp; <string-name><given-names>David V.</given-names> <surname>Hinkley</surname></string-name>. <year>1997</year>. <source>Bootstrap Methods and Their Applications</source>. <publisher-loc>Cambridge</publisher-loc>: <publisher-name>Cambridge University Press</publisher-name>.</mixed-citation></ref>
<ref id="B16"><label>16</label><mixed-citation publication-type="book"><string-name><surname>DeVellis</surname>, <given-names>Robert F.</given-names></string-name> <year>2016</year>. <source>Scale development: Theory and applications</source> <volume>26</volume>. <publisher-loc>Thousand Oaks, CA</publisher-loc>: <publisher-name>Sage publications</publisher-name>.</mixed-citation></ref>
<ref id="B17"><label>17</label><mixed-citation publication-type="journal"><string-name><surname>Drasgow</surname>, <given-names>Fritz</given-names></string-name>, <string-name><given-names>Oleksandr S.</given-names> <surname>Chernyshenko</surname></string-name> &amp; <string-name><given-names>Stephen</given-names> <surname>Stark</surname></string-name>. <year>2010</year>. <article-title>75 years after Likert: Thurstone was right!</article-title> <source>Industrial and Organizational Psychology</source> <volume>3</volume>(<issue>4</issue>). <fpage>465</fpage>&#8211;<lpage>476</lpage>. DOI: <pub-id pub-id-type="doi">10.1111/j.1754-9434.2010.01273.x</pub-id></mixed-citation></ref>
<ref id="B18"><label>18</label><mixed-citation publication-type="book"><string-name><surname>Ennis</surname>, <given-names>Daniel M.</given-names></string-name> <year>2016</year>. <source>Thurstonian Models: Categorical Decision Making in the Presence of Noise</source>. <publisher-loc>Richmond, VA</publisher-loc>: <publisher-name>The Institute for Perception</publisher-name>.</mixed-citation></ref>
<ref id="B19"><label>19</label><mixed-citation publication-type="journal"><string-name><surname>Erlewine</surname>, <given-names>Michael Yoshitaka</given-names></string-name> &amp; <string-name><given-names>Hadas</given-names> <surname>Kotek</surname></string-name>. <year>2016</year>. <article-title>A streamlined approach to online linguistic surveys</article-title>. <source>Natural Language &amp; Linguistic Theory</source> <volume>34</volume>(<issue>2</issue>). <fpage>481</fpage>&#8211;<lpage>495</lpage>. DOI: <pub-id pub-id-type="doi">10.1007/s11049-015-9305-9</pub-id></mixed-citation></ref>
<ref id="B20"><label>20</label><mixed-citation publication-type="book"><string-name><surname>Fabrigar</surname>, <given-names>Leandre R.</given-names></string-name> &amp; <string-name><given-names>Jung-Eun</given-names> <surname>Shelly Paik</surname></string-name>. <year>2007</year>. <chapter-title>Thurstone Scales</chapter-title>. In <string-name><given-names>Neil</given-names> <surname>Salkind</surname></string-name> (ed.), <source>Encyclopedia of Measurement and Statistics</source>, <fpage>1003</fpage>&#8211;<lpage>1005</lpage>. <publisher-name>SAGE publications</publisher-name>.</mixed-citation></ref>
<ref id="B21"><label>21</label><mixed-citation publication-type="journal"><string-name><surname>Featherston</surname>, <given-names>Sam</given-names></string-name>. <year>2005</year>. <article-title>Magnitude estimation and what it can do for your syntax: Some wh-constraints in German</article-title>. <source>Lingua</source> <volume>115</volume>(<issue>11</issue>). <fpage>1525</fpage>&#8211;<lpage>1550</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2004.07.003</pub-id></mixed-citation></ref>
<ref id="B22"><label>22</label><mixed-citation publication-type="journal"><string-name><surname>Featherston</surname>, <given-names>Sam</given-names></string-name>. <year>2007</year>. <article-title>Data in generative grammar: The stick and the carrot</article-title>. <source>Theoretical Linguistics</source> <volume>33</volume>(<issue>3</issue>). <fpage>269</fpage>&#8211;<lpage>318</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/TL.2007.020</pub-id></mixed-citation></ref>
<ref id="B23"><label>23</label><mixed-citation publication-type="journal"><string-name><surname>Featherston</surname>, <given-names>Sam</given-names></string-name>. <year>2008</year>. <article-title>Thermometer judgments as linguistic evidence</article-title>. <source>Was ist linguistische Evidenz</source>, <fpage>69</fpage>&#8211;<lpage>89</lpage>.</mixed-citation></ref>
<ref id="B24"><label>24</label><mixed-citation publication-type="journal"><string-name><surname>Featherston</surname>, <given-names>Sam</given-names></string-name>. <year>2009</year>. <article-title>Relax, lean back, and be a linguist</article-title>. <source>Zeitschrift f&#252;r Sprachwissenschaft</source> <volume>28</volume>(<issue>1</issue>). <fpage>127</fpage>&#8211;<lpage>132</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/ZFSW.2009.014</pub-id></mixed-citation></ref>
<ref id="B25"><label>25</label><mixed-citation publication-type="confproc"><string-name><surname>Fukuda</surname>, <given-names>Shin</given-names></string-name>, <string-name><given-names>Grant</given-names> <surname>Goodall</surname></string-name>, <string-name><given-names>Dan</given-names> <surname>Michel</surname></string-name> &amp; <string-name><given-names>Henry</given-names> <surname>Beecher</surname></string-name>. <year>2012</year>. <article-title>Is Magnitude Estimation worth the trouble?</article-title> In <string-name><given-names>Jaehoon</given-names> <surname>Choi</surname></string-name>, <string-name><given-names>E. Allan</given-names> <surname>Hogue</surname></string-name>, <string-name><given-names>Jeffrey</given-names> <surname>Punske</surname></string-name>, <string-name><given-names>Deniz</given-names> <surname>Tat</surname></string-name>, <string-name><given-names>Jessamyn</given-names> <surname>Schertz</surname></string-name> &amp; <string-name><given-names>Alex</given-names> <surname>Trueman</surname></string-name> (eds.), <conf-name>Proceedings of the 29th West Coast Conference on Formal Linguistics</conf-name>, <fpage>328</fpage>&#8211;<lpage>336</lpage>.</mixed-citation></ref>
<ref id="B26"><label>26</label><mixed-citation publication-type="journal"><string-name><surname>Gelman</surname>, <given-names>Andrew</given-names></string-name> &amp; <string-name><given-names>Francis</given-names> <surname>Tuerlinckx</surname></string-name>. <year>2000</year>. <article-title>Type S error rates for classical and Bayesian single and multiple comparison procedures</article-title>. <source>Computational Statistics</source> <volume>15</volume>(<issue>3</issue>). <fpage>373</fpage>&#8211;<lpage>390</lpage>. DOI: <pub-id pub-id-type="doi">10.1007/s001800000040</pub-id></mixed-citation></ref>
<ref id="B27"><label>27</label><mixed-citation publication-type="journal"><string-name><surname>Gibson</surname>, <given-names>Edward</given-names></string-name> &amp; <string-name><given-names>Evelina</given-names> <surname>Fedorenko</surname></string-name>. <year>2013</year>. <article-title>The need for quantitative methods in syntax and semantics research</article-title>. <source>Language and Cognitive Processes</source> <volume>28</volume>(<issue>1&#8211;2</issue>). <fpage>88</fpage>&#8211;<lpage>124</lpage>. DOI: <pub-id pub-id-type="doi">10.1080/01690965.2010.515080</pub-id></mixed-citation></ref>
<ref id="B28"><label>28</label><mixed-citation publication-type="journal"><string-name><surname>Gibson</surname>, <given-names>Edward</given-names></string-name>, <string-name><given-names>Steven T.</given-names> <surname>Piantadosi</surname></string-name> &amp; <string-name><given-names>Evelina</given-names> <surname>Fedorenko</surname></string-name>. <year>2013</year>. <article-title>Quantitative methods in syntax/semantics research: A response to sprouse and almeida (2013)</article-title>. <source>Language and Cognitive Processes</source> <volume>28</volume>(<issue>3</issue>). <fpage>229</fpage>&#8211;<lpage>240</lpage>. DOI: <pub-id pub-id-type="doi">10.1080/01690965.2012.704385</pub-id></mixed-citation></ref>
<ref id="B29"><label>29</label><mixed-citation publication-type="journal"><string-name><surname>Hartley</surname>, <given-names>James</given-names></string-name>. <year>2014</year>. <article-title>Some thoughts on Likert-type scales</article-title>. <source>International Journal of Clinical and Health Psychology</source> <volume>14</volume>(<issue>1</issue>). <fpage>83</fpage>&#8211;<lpage>86</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/S1697-2600(14)70040-7</pub-id></mixed-citation></ref>
<ref id="B30"><label>30</label><mixed-citation publication-type="journal"><string-name><surname>H&#228;ussler</surname>, <given-names>Jana</given-names></string-name> &amp; <string-name><given-names>Thomas</given-names> <surname>Juzek</surname></string-name>. <year>2016</year>. <article-title>Detecting and discouraging noncooperative behavior in online experiments using an acceptability judgment task</article-title>. In <string-name><given-names>Hanna</given-names> <surname>Christ</surname></string-name>, <string-name><given-names>Daniel</given-names> <surname>Klenov&#353;ak</surname></string-name>, <string-name><given-names>Lukas</given-names> <surname>S&#246;nning</surname></string-name> &amp; <string-name><given-names>Valentin</given-names> <surname>Werner</surname></string-name> (eds.), <source>A blend of MaLT: Selected contributions from the methods and linguistic theories symposium 2015</source>, <volume>15</volume>. <fpage>73</fpage>&#8211;<lpage>100</lpage>.</mixed-citation></ref>
<ref id="B31"><label>31</label><mixed-citation publication-type="confproc"><string-name><surname>H&#228;ussler</surname>, <given-names>Jana</given-names></string-name>, <string-name><given-names>Thomas</given-names> <surname>Juzek</surname></string-name> &amp; <string-name><given-names>Tom</given-names> <surname>Wasow</surname></string-name>. <year>2016</year>. <article-title>Unsupervised prediction of acceptability judgements</article-title>. In <string-name><given-names>Patrick</given-names> <surname>Farrell</surname></string-name> (ed.), <conf-name>To be grammatical or not to be grammatical &#8211; is that the question</conf-name>. <conf-sponsor>Annual Meeting of the Linguistic Society of America</conf-sponsor>.</mixed-citation></ref>
<ref id="B32"><label>32</label><mixed-citation publication-type="journal"><string-name><surname>Hindle</surname>, <given-names>Donald</given-names></string-name> &amp; <string-name><given-names>Sag</given-names> <surname>Ivan</surname></string-name>. <year>1975</year>. <article-title>Some more on anymore</article-title>. In <string-name><given-names>Ralph</given-names> <surname>Fasold</surname></string-name> &amp; <string-name><given-names>Roger</given-names> <surname>Shuy</surname></string-name> (eds.), <source>Analyzing variation in language: Papers from the second colloquium on new ways of analyzing variation</source>, <fpage>89</fpage>&#8211;<lpage>110</lpage>.</mixed-citation></ref>
<ref id="B33"><label>33</label><mixed-citation publication-type="journal"><string-name><surname>Hofmeister</surname>, <given-names>Philip</given-names></string-name>, <string-name><given-names>T. Florian</given-names> <surname>Jaeger</surname></string-name>, <string-name><given-names>Inbal</given-names> <surname>Arnon</surname></string-name>, <string-name><given-names>Ivan A.</given-names> <surname>Sag</surname></string-name> &amp; <string-name><given-names>Neal</given-names> <surname>Snider</surname></string-name>. <year>2013</year>. <article-title>The source ambiguity problem: Distinguishing the effects of grammar and processing on acceptability judgments</article-title>. <source>Language and Cognitive Processes</source> <volume>28</volume>(<issue>1&#8211;2</issue>). <fpage>48</fpage>&#8211;<lpage>87</lpage>. DOI: <pub-id pub-id-type="doi">10.1080/01690965.2011.572401</pub-id></mixed-citation></ref>
<ref id="B34"><label>34</label><mixed-citation publication-type="book"><string-name><surname>Johnson</surname>, <given-names>Keith</given-names></string-name>. <year>2011</year>. <source>Quantitative methods in linguistics</source>. <publisher-loc>Manchester, MI</publisher-loc>: <publisher-name>John Wiley &amp; Sons</publisher-name>.</mixed-citation></ref>
<ref id="B35"><label>35</label><mixed-citation publication-type="journal"><string-name><surname>Johnson</surname>, <given-names>Timothy R.</given-names></string-name> <year>2003</year>. <article-title>On the use of heterogeneous thresholds ordinal regression models to account for individual differences in response style</article-title>. <source>Psychometrika</source> <volume>68</volume>(<issue>4</issue>). <fpage>563</fpage>&#8211;<lpage>583</lpage>. DOI: <pub-id pub-id-type="doi">10.1007/BF02295612</pub-id></mixed-citation></ref>
<ref id="B36"><label>36</label><mixed-citation publication-type="thesis"><string-name><surname>Juzek</surname>, <given-names>Thomas</given-names></string-name>. <year>2015</year>. <source>Acceptability judgement tasks and grammatical theory</source>. <publisher-loc>Oxford</publisher-loc>: <publisher-name>University of Oxford</publisher-name> Phd thesis.</mixed-citation></ref>
<ref id="B37"><label>37</label><mixed-citation publication-type="confproc"><string-name><surname>Keller</surname>, <given-names>Frank</given-names></string-name>. <year>2003</year>. <article-title>A psychophysical law for linguistic judgments</article-title>. In <string-name><given-names>Richard</given-names> <surname>Alterman</surname></string-name> &amp; <string-name><given-names>David</given-names> <surname>Kirsh</surname></string-name> (eds.), <conf-name>Proceedings of the 25th annual conference of the cognitive science society</conf-name>, <fpage>652</fpage>&#8211;<lpage>657</lpage>.</mixed-citation></ref>
<ref id="B38"><label>38</label><mixed-citation publication-type="confproc"><string-name><surname>Keller</surname>, <given-names>Frank</given-names></string-name> &amp; <string-name><given-names>Ash</given-names> <surname>Asudeh</surname></string-name>. <year>2001</year>. <article-title>Constraints on linguistic co-reference: Structural vs. pragmatic factors</article-title>. In <string-name><given-names>Johanna</given-names> <surname>Moore</surname></string-name> &amp; <string-name><given-names>Keith</given-names> <surname>Stenning</surname></string-name> (eds.), <conf-name>Proceedings of the 23rd annual conference of the cognitive science society</conf-name>, <fpage>483</fpage>&#8211;<lpage>488</lpage>.</mixed-citation></ref>
<ref id="B39"><label>39</label><mixed-citation publication-type="book"><string-name><surname>Kline</surname>, <given-names>Paul</given-names></string-name>. <year>2013</year>. <source>Handbook of psychological testing</source>. <publisher-name>Routledge</publisher-name>.</mixed-citation></ref>
<ref id="B40"><label>40</label><mixed-citation publication-type="journal"><string-name><surname>Lau</surname>, <given-names>Jey Han</given-names></string-name>, <string-name><given-names>Alexander</given-names> <surname>Clark</surname></string-name> &amp; <string-name><given-names>Shalom</given-names> <surname>Lappin</surname></string-name>. <year>2016</year>. <article-title>Grammaticality, acceptability, and probability: A probabilistic view of linguistic knowledge</article-title>. <source>Cognitive Science</source>.</mixed-citation></ref>
<ref id="B41"><label>41</label><mixed-citation publication-type="journal"><string-name><surname>Lee</surname>, <given-names>Jerry</given-names></string-name>, <string-name><given-names>Patricia</given-names> <surname>Jones</surname></string-name>, <string-name><given-names>Yoshimitsu</given-names> <surname>Mineyama</surname></string-name> &amp; <string-name><given-names>Xinwei Esther</given-names> <surname>Zhang</surname></string-name>. <year>2002</year>. <article-title>Cultural differences in responses to a Likert scale</article-title>. <source>Research in Nursing &amp; Health</source> <volume>25</volume>(<issue>4</issue>). <fpage>295</fpage>&#8211;<lpage>306</lpage>. DOI: <pub-id pub-id-type="doi">10.1002/nur.10041</pub-id></mixed-citation></ref>
<ref id="B42"><label>42</label><mixed-citation publication-type="journal"><string-name><surname>Li</surname>, <given-names>Linjie</given-names></string-name>, <string-name><given-names>Vicente</given-names> <surname>Malave</surname></string-name>, <string-name><given-names>Amanda</given-names> <surname>Song</surname></string-name> &amp; <string-name><given-names>Angela</given-names> <surname>Yu</surname></string-name>. <year>2016</year>. <article-title>Extracting human face similarity judgments: Pairs or triplets?</article-title> <source>Journal of Vision</source> <volume>16</volume>(<issue>12</issue>). <fpage>719</fpage>&#8211;<lpage>719</lpage>. DOI: <pub-id pub-id-type="doi">10.1167/16.12.719</pub-id></mixed-citation></ref>
<ref id="B43"><label>43</label><mixed-citation publication-type="journal"><string-name><surname>Likert</surname>, <given-names>Rensis</given-names></string-name>. <year>1932</year>. <article-title>A technique for the measurement of attitudes</article-title>. <source>Archives of Psychology</source> <volume>140</volume>. <fpage>44</fpage>&#8211;<lpage>60</lpage>.</mixed-citation></ref>
<ref id="B44"><label>44</label><mixed-citation publication-type="book"><string-name><surname>Linzen</surname>, <given-names>Tal</given-names></string-name> &amp; <string-name><given-names>Yohei</given-names> <surname>Oseki</surname></string-name>. <year>2015</year>. <source>The reliability of acceptability judgments across languages</source>. <publisher-loc>New York, MS</publisher-loc>: <publisher-name>New York University Press</publisher-name>.</mixed-citation></ref>
<ref id="B45"><label>45</label><mixed-citation publication-type="journal"><string-name><surname>Loken</surname>, <given-names>Eric</given-names></string-name> &amp; <string-name><given-names>Andrew</given-names> <surname>Gelman</surname></string-name>. <year>2017</year>. <article-title>Measurement error and the replication crisis</article-title>. <source>Science</source> <volume>355</volume>(<issue>6325</issue>). <fpage>584</fpage>&#8211;<lpage>585</lpage>. DOI: <pub-id pub-id-type="doi">10.1126/science.aal3618</pub-id></mixed-citation></ref>
<ref id="B46"><label>46</label><mixed-citation publication-type="journal"><string-name><surname>Mahowald</surname>, <given-names>Kyle</given-names></string-name>, <string-name><given-names>Peter</given-names> <surname>Graff</surname></string-name>, <string-name><given-names>Jeremy</given-names> <surname>Hartman</surname></string-name> &amp; <string-name><given-names>Edward</given-names> <surname>Gibson</surname></string-name>. <year>2016</year>. <article-title>SNAP judgments: A small N acceptability paradigm (SNAP) for linguistic acceptability judgments</article-title>. <source>Language</source> <volume>92</volume>(<issue>3</issue>). <fpage>619</fpage>&#8211;<lpage>635</lpage>. DOI: <pub-id pub-id-type="doi">10.1353/lan.2016.0052</pub-id></mixed-citation></ref>
<ref id="B47"><label>47</label><mixed-citation publication-type="confproc"><string-name><surname>Miller</surname>, <given-names>Brent</given-names></string-name>, <string-name><given-names>Pernille</given-names> <surname>Hemmer</surname></string-name>, <string-name><given-names>Mark</given-names> <surname>Steyvers</surname></string-name> &amp; <string-name><given-names>Michael D.</given-names> <surname>Lee</surname></string-name>. <year>2009</year>. <article-title>The wisdom of crowds in rank ordering problems</article-title>. In <string-name><given-names>Andrew</given-names> <surname>Howes</surname></string-name>, <string-name><given-names>David</given-names> <surname>Peebles</surname></string-name> &amp; <string-name><given-names>Richard</given-names> <surname>Cooper</surname></string-name> (eds.), <conf-name>9th International conference on cognitive modeling</conf-name>, <fpage>86</fpage>&#8211;<lpage>91</lpage>. <conf-loc>Manchester</conf-loc>: <conf-sponsor>ICCM</conf-sponsor>.</mixed-citation></ref>
<ref id="B48"><label>48</label><mixed-citation publication-type="confproc"><string-name><surname>Munro</surname>, <given-names>Robert</given-names></string-name>, <string-name><given-names>Steven</given-names> <surname>Bethard</surname></string-name>, <string-name><given-names>Victor</given-names> <surname>Kuperman</surname></string-name>, <string-name><given-names>Vicky Tzuyin</given-names> <surname>Lai</surname></string-name>, <string-name><given-names>Robin</given-names> <surname>Melnick</surname></string-name>, <string-name><given-names>Christopher</given-names> <surname>Potts</surname></string-name>, <string-name><given-names>Tyler</given-names> <surname>Schnoebelen</surname></string-name> &amp; <string-name><given-names>Harry</given-names> <surname>Tily</surname></string-name>. <year>2010</year>. <article-title>Crowdsourcing and language studies: The new generation of linguistic data</article-title>. In <string-name><given-names>Jon</given-names> <surname>Weese</surname></string-name> (ed.), <conf-name>Proceedings of the NAACL HLT 2010 workshop on creating speech and language data with Amazon&#8217;s Mechanical Turk</conf-name>, <fpage>122</fpage>&#8211;<lpage>130</lpage>.</mixed-citation></ref>
<ref id="B49"><label>49</label><mixed-citation publication-type="book"><string-name><surname>Murphy</surname>, <given-names>Brian</given-names></string-name>, <string-name><given-names>Carl</given-names> <surname>Vogel</surname></string-name> &amp; <string-name><given-names>Conny</given-names> <surname>Opitz</surname></string-name>. <year>2006</year>. <chapter-title>Cross-linguistic empirical analysis of constraints on passive</chapter-title>. In <source>Presentation to the symposium on interdisciplinary themes in cognitive language research</source>. <publisher-loc>Helsinki</publisher-loc>: <publisher-name>Finnish Cognitive Linguistics Association</publisher-name>.</mixed-citation></ref>
<ref id="B50"><label>50</label><mixed-citation publication-type="journal"><string-name><surname>Myers</surname>, <given-names>James</given-names></string-name>. <year>2009</year>. <article-title>The design and analysis of small-scale syntactic judgment experiments</article-title>. <source>Lingua</source> <volume>119</volume>(<issue>3</issue>). <fpage>425</fpage>&#8211;<lpage>444</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2008.09.003</pub-id></mixed-citation></ref>
<ref id="B51"><label>51</label><mixed-citation publication-type="journal"><string-name><surname>Myers</surname>, <given-names>James</given-names></string-name>. <year>2012</year>. <article-title>Testing adjunct and conjunct island constraints in Chinese</article-title>. <source>Language and Linguistics</source> <volume>13</volume>(<issue>3</issue>). <fpage>437</fpage>.</mixed-citation></ref>
<ref id="B52"><label>52</label><mixed-citation publication-type="journal"><string-name><surname>Nosofsky</surname>, <given-names>Robert M.</given-names></string-name> <year>1992</year>. <article-title>Similarity scaling and cognitive process models</article-title>. <source>Annual Review of Psychology</source> <volume>43</volume>(<issue>1</issue>). <fpage>25</fpage>&#8211;<lpage>53</lpage>. DOI: <pub-id pub-id-type="doi">10.1146/annurev.ps.43.020192.000325</pub-id></mixed-citation></ref>
<ref id="B53"><label>53</label><mixed-citation publication-type="journal"><string-name><surname>Noveck</surname>, <given-names>Ira</given-names></string-name> &amp; <string-name><given-names>Anne</given-names> <surname>Reboul</surname></string-name>. <year>2008</year>. <article-title>Experimental pragmatics: A Gricean turn in the study of language</article-title>. <source>Trends in Cognitive Sciences</source> <volume>12</volume>(<issue>11</issue>). <fpage>425</fpage>&#8211;<lpage>431</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.tics.2008.07.009</pub-id></mixed-citation></ref>
<ref id="B54"><label>54</label><mixed-citation publication-type="journal"><string-name><surname>O&#8217;Mahony</surname>, <given-names>Michael</given-names></string-name> et al. <year>2003</year>. <article-title>Discrimination testing: A few ideas, old and new</article-title>. <source>Food Quality and Preference</source> <volume>14</volume>(<issue>2</issue>). <fpage>157</fpage>&#8211;<lpage>164</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/S0950-3293(02)00109-X</pub-id></mixed-citation></ref>
<ref id="B55"><label>55</label><mixed-citation publication-type="journal"><string-name><surname>Phillips</surname>, <given-names>Colin</given-names></string-name>. <year>2009</year>. <article-title>Should we impeach armchair linguists?</article-title> <source>Japanese/Korean Linguistics</source> <volume>17</volume>. <fpage>49</fpage>&#8211;<lpage>64</lpage>.</mixed-citation></ref>
<ref id="B56"><label>56</label><mixed-citation publication-type="journal"><string-name><surname>Phillips</surname>, <given-names>Colin</given-names></string-name> &amp; <string-name><given-names>Howard</given-names> <surname>Lasnik</surname></string-name>. <year>2003</year>. <article-title>Linguistics and empirical evidence: Reply to edelman and christiansen</article-title>. <source>Trends in Cognitive Sciences</source> <volume>7</volume>(<issue>2</issue>). <fpage>61</fpage>&#8211;<lpage>62</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/S1364-6613(02)00046-3</pub-id></mixed-citation></ref>
<ref id="B57"><label>57</label><mixed-citation publication-type="book"><string-name><surname>Podesva</surname>, <given-names>Robert</given-names></string-name> &amp; <string-name><given-names>Devyani</given-names> <surname>Sharma</surname></string-name>. <year>2014</year>. <source>Research methods in linguistics</source>. <publisher-loc>Cambridge</publisher-loc>: <publisher-name>Cambridge University Press</publisher-name>.</mixed-citation></ref>
<ref id="B58"><label>58</label><mixed-citation publication-type="journal"><string-name><surname>Porte</surname>, <given-names>Graeme</given-names></string-name>. <year>2013</year>. <article-title>Who needs replication?</article-title> <source>CALICO Journal</source> <volume>30</volume>(<issue>1</issue>). <fpage>10</fpage>&#8211;<lpage>15</lpage>. DOI: <pub-id pub-id-type="doi">10.11139/cj.30.1.10-15</pub-id></mixed-citation></ref>
<ref id="B59"><label>59</label><mixed-citation publication-type="journal"><string-name><surname>Rensink</surname>, <given-names>Ronald</given-names></string-name>, <string-name><given-names>Kevin</given-names> <surname>O&#8217;Regan</surname></string-name> &amp; <string-name><given-names>James</given-names> <surname>Clark</surname></string-name>. <year>1997</year>. <article-title>To see or not to see: The need for attention to perceive changes in scenes</article-title>. <source>Psychological Science</source> <volume>8</volume>(<issue>5</issue>). <fpage>368</fpage>&#8211;<lpage>373</lpage>. DOI: <pub-id pub-id-type="doi">10.1111/j.1467-9280.1997.tb00427.x</pub-id></mixed-citation></ref>
<ref id="B60"><label>60</label><mixed-citation publication-type="journal"><string-name><surname>Roberts</surname>, <given-names>James</given-names></string-name>, <string-name><given-names>James</given-names> <surname>Laughlin</surname></string-name> &amp; <string-name><given-names>Douglas</given-names> <surname>Wedell</surname></string-name>. <year>1999</year>. <article-title>Validity issues in the Likert and Thurstone approaches to attitude measurement</article-title>. <source>Educational and Psychological Measurement</source> <volume>59</volume>(<issue>2</issue>). <fpage>211</fpage>&#8211;<lpage>233</lpage>. DOI: <pub-id pub-id-type="doi">10.1177/00131649921969811</pub-id></mixed-citation></ref>
<ref id="B61"><label>61</label><mixed-citation publication-type="journal"><string-name><surname>Rosenbach</surname>, <given-names>Anette</given-names></string-name>. <year>2003</year>. <article-title>Aspects of iconicity and economy in the choice between the s-genitive and the of-genitive in english</article-title>. <source>Topics in English Linguistics</source> <volume>43</volume>. <fpage>379</fpage>&#8211;<lpage>412</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/9783110900019.379</pub-id></mixed-citation></ref>
<ref id="B62"><label>62</label><mixed-citation publication-type="journal"><string-name><surname>Sampson</surname>, <given-names>Geoffrey</given-names></string-name>. <year>2007</year>. <article-title>Grammar without grammaticality</article-title>. <source>Corpus Linguistics and Linguistic Theory</source> <volume>3</volume>(<issue>1</issue>). <fpage>1</fpage>&#8211;<lpage>32</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/CLLT.2007.001</pub-id></mixed-citation></ref>
<ref id="B63"><label>63</label><mixed-citation publication-type="book"><string-name><surname>Sch&#252;tze</surname>, <given-names>Carson</given-names></string-name>. <year>1996</year>. <source>The empirical base of linguistics: Grammaticality judgments and linguistic methodology</source>. <publisher-loc>Chicago, IL</publisher-loc>: <publisher-name>University of Chicago Press</publisher-name>.</mixed-citation></ref>
<ref id="B64"><label>64</label><mixed-citation publication-type="journal"><string-name><surname>Sch&#252;tze</surname>, <given-names>Carson</given-names></string-name>. <year>2011</year>. <article-title>Linguistic evidence and grammatical theory</article-title>. <source>Wiley Interdisciplinary Reviews: Cognitive Science</source> <volume>2</volume>(<issue>2</issue>). <fpage>206</fpage>&#8211;<lpage>221</lpage>. DOI: <pub-id pub-id-type="doi">10.1002/wcs.102</pub-id></mixed-citation></ref>
<ref id="B65"><label>65</label><mixed-citation publication-type="book"><string-name><surname>Sch&#252;tze</surname>, <given-names>Carson</given-names></string-name> &amp; <string-name><given-names>Jon</given-names> <surname>Sprouse</surname></string-name>. <year>2014</year>. <chapter-title>Judgment data</chapter-title>. In <string-name><given-names>Robert</given-names> <surname>Podesva</surname></string-name> &amp; <string-name><given-names>Devyani</given-names> <surname>Sharma</surname></string-name> (eds.), <source>Research methods in linguistics</source>, chap. 3. <fpage>27</fpage>&#8211;<lpage>51</lpage>. <publisher-loc>Cambridge</publisher-loc>: <publisher-name>Cambridge University Press</publisher-name>.</mixed-citation></ref>
<ref id="B66"><label>66</label><mixed-citation publication-type="journal"><string-name><surname>Selker</surname>, <given-names>Ravi</given-names></string-name>, <string-name><given-names>Michael D.</given-names> <surname>Lee</surname></string-name> &amp; <string-name><given-names>Ravi</given-names> <surname>Iyer</surname></string-name>. <year>2017</year>. <article-title>Thurstonian cognitive models for aggregating top-n lists</article-title>. <source>Decision</source> <volume>4</volume>(<issue>2</issue>). <fpage>87</fpage>. DOI: <pub-id pub-id-type="doi">10.1037/dec0000056</pub-id></mixed-citation></ref>
<ref id="B67"><label>67</label><mixed-citation publication-type="journal"><string-name><surname>Shalom</surname>, <given-names>Dorit Ben</given-names></string-name> &amp; <string-name><given-names>David</given-names> <surname>Poeppel</surname></string-name>. <year>2007</year>. <article-title>Functional anatomic models of language: Assembling the pieces</article-title>. <source>The Neuroscientist</source>.</mixed-citation></ref>
<ref id="B68"><label>68</label><mixed-citation publication-type="journal"><string-name><surname>Simons</surname>, <given-names>Daniel</given-names></string-name> &amp; <string-name><given-names>Ronald</given-names> <surname>Rensink</surname></string-name>. <year>2005</year>. <article-title>Change blindness: Past, present, and future</article-title>. <source>Trends in Cognitive Sciences</source> <volume>9</volume>(<issue>1</issue>). <fpage>16</fpage>&#8211;<lpage>20</lpage>. DOI: <pub-id pub-id-type="doi">10.1111/j.0963-7214.2005.00332.x</pub-id></mixed-citation></ref>
<ref id="B69"><label>69</label><mixed-citation publication-type="book"><string-name><surname>Sorace</surname>, <given-names>Antonella</given-names></string-name>. <year>2010</year>. <chapter-title>Using magnitude estimation in developmental linguistic research</chapter-title>. In <string-name><given-names>Elma</given-names> <surname>Blom</surname></string-name> &amp; <string-name><given-names>Sharon</given-names> <surname>Unsworth</surname></string-name> (eds.), <source>Experimental methods in language acquisition research</source>, <fpage>57</fpage>&#8211;<lpage>72</lpage>. <publisher-loc>Amsterdam</publisher-loc>: <publisher-name>John Benjamins</publisher-name>. DOI: <pub-id pub-id-type="doi">10.1075/lllt.27.05sor</pub-id></mixed-citation></ref>
<ref id="B70"><label>70</label><mixed-citation publication-type="journal"><string-name><surname>Sorace</surname>, <given-names>Antonella</given-names></string-name> &amp; <string-name><given-names>Frank</given-names> <surname>Keller</surname></string-name>. <year>2005</year>. <article-title>Gradience in linguistic data</article-title>. <source>Lingua</source> <volume>115</volume>(<issue>11</issue>). <fpage>1497</fpage>&#8211;<lpage>1524</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2004.07.002</pub-id></mixed-citation></ref>
<ref id="B71"><label>71</label><mixed-citation publication-type="journal"><string-name><surname>Spencer</surname>, <given-names>Nancy Jane</given-names></string-name>. <year>1973</year>. <article-title>Differences between linguists and nonlinguists in intuitions of grammaticality-acceptability</article-title>. <source>Journal of Psycholinguistic Research</source> <volume>2</volume>(<issue>2</issue>). <fpage>83</fpage>&#8211;<lpage>98</lpage>. DOI: <pub-id pub-id-type="doi">10.1007/BF01067203</pub-id></mixed-citation></ref>
<ref id="B72"><label>72</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name>. <year>2007</year>. <article-title>Continuous acceptability, categorical grammaticality, and experimental syntax</article-title>. <source>Biolinguistics</source> <volume>1</volume>. <fpage>123</fpage>&#8211;<lpage>134</lpage>.</mixed-citation></ref>
<ref id="B73"><label>73</label><mixed-citation publication-type="confproc"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name>. <year>2008</year>. <article-title>Magnitude estimation and the non-linearity of acceptability judgments</article-title>. In <string-name><given-names>Natasha</given-names> <surname>Abner</surname></string-name> &amp; <string-name><given-names>Jason</given-names> <surname>Bishop</surname></string-name> (eds.), <conf-name>Proceedings of the 27th West Coast Conference on Formal Linguistics</conf-name>, <fpage>397</fpage>&#8211;<lpage>403</lpage>. <conf-loc>Somerville, MA</conf-loc>: <conf-sponsor>Cascadilla Press</conf-sponsor>.</mixed-citation></ref>
<ref id="B74"><label>74</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name>. <year>2011a</year>. <article-title>A test of the cognitive assumptions of magnitude estimation: Commutativity does not hold for acceptability judgments</article-title>. <source>Language</source> <volume>87</volume>(<issue>2</issue>). <fpage>274</fpage>&#8211;<lpage>288</lpage>. DOI: <pub-id pub-id-type="doi">10.1353/lan.2011.0028</pub-id></mixed-citation></ref>
<ref id="B75"><label>75</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name>. <year>2011b</year>. <article-title>A validation of Amazon Mechanical Turk for the collection of acceptability judgments in linguistic theory</article-title>. <source>Behavior Research Methods</source> <volume>43</volume>(<issue>1</issue>). <fpage>155</fpage>&#8211;<lpage>167</lpage>. DOI: <pub-id pub-id-type="doi">10.3758/s13428-010-0039-7</pub-id></mixed-citation></ref>
<ref id="B76"><label>76</label><mixed-citation publication-type="book"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name> &amp; <string-name><given-names>Carson</given-names> <surname>Sch&#252;tze</surname></string-name>. <year>2017</year>. <chapter-title>Grammar and the use of data</chapter-title>. In <string-name><given-names>Bas</given-names> <surname>Aarts</surname></string-name>, <string-name><given-names>Jill</given-names> <surname>Bowie</surname></string-name> &amp; <string-name><given-names>Gergana</given-names> <surname>Popova</surname></string-name> (eds.), <source>The Oxford handbook of English grammar</source>, chap. 3. <publisher-name>Oxford University Press</publisher-name>.</mixed-citation></ref>
<ref id="B77"><label>77</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name>, <string-name><given-names>Carson</given-names> <surname>Sch&#252;tze</surname></string-name> &amp; <string-name><given-names>Diogo</given-names> <surname>Almeida</surname></string-name>. <year>2013</year>. <article-title>A comparison of informal and formal acceptability judgments using a random sample from Linguistic Inquiry 2001&#8211;2010</article-title>. <source>Lingua</source> <volume>134</volume>. <fpage>219</fpage>&#8211;<lpage>248</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2013.07.002</pub-id></mixed-citation></ref>
<ref id="B78"><label>78</label><mixed-citation publication-type="thesis"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name> &amp; <string-name><given-names>Diogo</given-names> <surname>Almeida</surname></string-name>. <year>2011</year>. <source>Power in acceptability judgment experiments and the reliability of data in syntax</source>. <publisher-loc>Irvine CA &amp; Lansing MI</publisher-loc>: <publisher-name>University of California &amp; Michigan State University</publisher-name> Master&#8217;s thesis.</mixed-citation></ref>
<ref id="B79"><label>79</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name> &amp; <string-name><given-names>Diogo</given-names> <surname>Almeida</surname></string-name>. <year>2012</year>. <article-title>Assessing the reliability of textbook data in syntax: Adger&#8217;s Core Syntax</article-title>. <source>Journal of Linguistics</source> <volume>48</volume>(<issue>03</issue>). <fpage>609</fpage>&#8211;<lpage>652</lpage>. DOI: <pub-id pub-id-type="doi">10.1017/S0022226712000011</pub-id></mixed-citation></ref>
<ref id="B80"><label>80</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name> &amp; <string-name><given-names>Diogo</given-names> <surname>Almeida</surname></string-name>. <year>2017a</year>. <article-title>Design sensitivity and statistical power in acceptability judgment experiments</article-title>. <source>Glossa: a Journal of General Linguistics</source> <volume>2</volume>(<issue>1</issue>). <fpage>14</fpage>.</mixed-citation></ref>
<ref id="B81"><label>81</label><mixed-citation publication-type="journal"><string-name><surname>Sprouse</surname>, <given-names>Jon</given-names></string-name> &amp; <string-name><given-names>Diogo</given-names> <surname>Almeida</surname></string-name>. <year>2017b</year>. <article-title>Setting the empirical record straight: Acceptability judgments appear to be reliable, robust, and replicable</article-title>. <source>Behavioral and Brain Sciences</source> <volume>40</volume>. DOI: <pub-id pub-id-type="doi">10.1017/S0140525X17000590</pub-id></mixed-citation></ref>
<ref id="B82"><label>82</label><mixed-citation publication-type="journal"><string-name><surname>Stevens</surname>, <given-names>Stanley Smith</given-names></string-name>. <year>1956</year>. <article-title>The direct estimation of sensory magnitudes: Loudness</article-title>. <source>The American Journal of Psychology</source>, <fpage>1</fpage>&#8211;<lpage>25</lpage>. DOI: <pub-id pub-id-type="doi">10.2307/1418112</pub-id></mixed-citation></ref>
<ref id="B83"><label>83</label><mixed-citation publication-type="journal"><string-name><surname>Thurstone</surname>, <given-names>Louis</given-names></string-name>. <year>1927</year>. <article-title>A law of comparative judgment</article-title>. <source>Psychological Review</source> <volume>34</volume>(<issue>4</issue>). <fpage>273</fpage>. DOI: <pub-id pub-id-type="doi">10.1037/h0070288</pub-id></mixed-citation></ref>
<ref id="B84"><label>84</label><mixed-citation publication-type="journal"><string-name><surname>Tversky</surname>, <given-names>Amos</given-names></string-name>. <year>1969</year>. <article-title>Intransitivity of preferences</article-title>. <source>Psychological Review</source> <volume>76</volume>(<issue>1</issue>). <fpage>31</fpage>. DOI: <pub-id pub-id-type="doi">10.1037/h0026750</pub-id></mixed-citation></ref>
<ref id="B85"><label>85</label><mixed-citation publication-type="journal"><string-name><surname>Wasow</surname>, <given-names>Thomas</given-names></string-name> &amp; <string-name><given-names>Jennifer</given-names> <surname>Arnold</surname></string-name>. <year>2005</year>. <article-title>Intuitions in linguistic argumentation</article-title>. <source>Lingua</source> <volume>115</volume>(<issue>11</issue>). <fpage>1481</fpage>&#8211;<lpage>1496</lpage>. DOI: <pub-id pub-id-type="doi">10.1016/j.lingua.2004.07.001</pub-id></mixed-citation></ref>
<ref id="B86"><label>86</label><mixed-citation publication-type="confproc"><string-name><surname>Weskott</surname>, <given-names>Thomas</given-names></string-name> &amp; <string-name><given-names>Gisbert</given-names> <surname>Fanselow</surname></string-name>. <year>2008</year>. <article-title>Variance and informativity in different measures of linguistic acceptability</article-title>. In <string-name><given-names>Natasha</given-names> <surname>Abner</surname></string-name> &amp; <string-name><given-names>Jason</given-names> <surname>Bishop</surname></string-name> (eds.), <conf-name>Proceedings of the 27th West Coast Conference on Formal Linguistics</conf-name>, <fpage>431</fpage>&#8211;<lpage>439</lpage>. <conf-loc>Somerville, MA</conf-loc>: <conf-sponsor>Cascadilla Press</conf-sponsor>.</mixed-citation></ref>
<ref id="B87"><label>87</label><mixed-citation publication-type="journal"><string-name><surname>Weskott</surname>, <given-names>Thomas</given-names></string-name> &amp; <string-name><given-names>Gisbert</given-names> <surname>Fanselow</surname></string-name>. <year>2009</year>. <article-title>Scaling issues in the measurement of linguistic acceptability</article-title>. <source>The Fruits of Empirical Linguistics</source> <volume>1</volume>. <fpage>229</fpage>&#8211;<lpage>245</lpage>. DOI: <pub-id pub-id-type="doi">10.1515/9783110216141.229</pub-id></mixed-citation></ref>
<ref id="B88"><label>88</label><mixed-citation publication-type="journal"><string-name><surname>Weskott</surname>, <given-names>Thomas</given-names></string-name> &amp; <string-name><given-names>Gisbert</given-names> <surname>Fanselow</surname></string-name>. <year>2011</year>. <article-title>On the informativity of different measures of linguistic acceptability</article-title>. <source>Language</source> <volume>87</volume>(<issue>2</issue>). <fpage>249</fpage>&#8211;<lpage>273</lpage>. DOI: <pub-id pub-id-type="doi">10.1353/lan.2011.0041</pub-id></mixed-citation></ref>
</ref-list>
</back>
</article>