2.1 Noun complements and relative clauses in Japanese

As the main empirical goal of our study is to compare scrambling out of noun complements and relative clauses in Japanese, some discussion of the syntactic properties of noun complements and relative clauses in Japanese is in order.

Noun complements in this study are complex NPs headed by toyuu ‘that.say’ and nouns such as uwasa ‘rumor’ and shooko ‘evidence’ as in (1a‒b) below.

Following previous studies (e.g., Fukui 1988), we assume that toyuu ‘that.say’ is a complementizer and that noun complements are CPs. There are at least two other types of complex NPs in Japanese: those that are headed by no (2a) and those that are headed by koto (2b).

This study focuses on toyuu complex NPs as in (1) for the following reasons. First, as discussed in the introduction, previous studies on scrambling out of complex NPs in Japanese focused on toyuu complex NPs presumably because they represent the Japanese equivalents of noun complements in English (e.g., Nakau 1973). Second, although all three types of complex NPs in (1) and (2) exhibit the basic syntactic properties of NPs, such as being marked by a case marker and functioning as subjects and direct objects, the nouns like uwasa ‘rumor’ and shooko ‘evidence’ in (1) are different from no and koto in (2) as the former have a clear and identifiable meaning while the latter are semantically bleached.¹ There are also indications that no and koto have become “markers” of certain semantic/pragmatic distinctions beyond the typical functional role of nouns. According to Kuno (1973), the semantic/pragmatic contribution of no and koto is that the complex NPs headed by them represent propositions that the speaker presupposes to be true. Relatedly, nouns like uwasa ‘rumor’ can stand alone as full NPs, whereas no and koto cannot. Under the assumption that noun complements involve the head noun taking a clausal complement, the head noun in a noun complement must be a lexical item with the ability to thematically license its complement. Lexical nouns like uwasa ‘rumor’ and shooko ‘evidence’ fit this description, while no and koto do not.

Relative clauses and noun complements in Japanese differ in the same ways that relative clauses and noun complements differ in many other languages (including English): for example, noun complements stand in a thematic relation with the selecting noun, while relative clauses are modifiers; similarly, in noun complements every argument required by the thematic structure of the embedded verb is present, while relative clauses have a missing argument that is interpreted as coreferential with the head NP (the relative clauses used in this study do not allow resumptive pronouns, but see Kuno 1973 for the claim that certain relative clauses, such as those with multiple levels of embedding may allow resumptive pronouns). However, noun complements and relative clauses also differ in a way that is not, to our knowledge, as common: at least two pieces of evidence suggest that the structure of noun complements in Japanese is more complex along certain dimensions than the structure of the relative clauses. First, Tomioka (2015) observes that noun complements can embed an NP marked by -wa (3a) while relative clauses cannot (3b), potentially suggesting an additional layer of functional structure in noun complements under the assumption that topics appear higher in the left periphery than nominative subjects.

Second, it has also been observed that relative clauses allow a non-episodic interpretation of verbs that normally entails a change of state, as in (4a) (e.g., Teramura 1982; Ogihara 2004). Noun complements do not allow such an interpretation of similar verbs (4b).

(4a) is ambiguous between two interpretations. One interpretation is that what Taro asked for is a dry towel, while the alternative interpretation is that he asked for a dried towel, a towel that was wet at some point in the past, but it had dried. In (4b), in contrast, the embedded sentence can only have the latter interpretation, i.e., a towel underwent a change of state from ‘not dry’ to ‘dry.’ Ogihara (2004) argues that relative clauses such as the one in (4a) do not involve a clausal structure but a TP with a reduced verbal projection.

In our study, noun complements always involve a noun preceded by an embedded clause with a full set of arguments marked by toyuu ‘that.say,’ while relative clauses always involve a gap that is identified with the modified NP and are never marked by toyuu ‘that.say.’²

2.2 The island status of noun complements with NP-scrambling

Haig (1976) was one of the first theoretical studies to investigate complex NPs in Japanese, reporting that NP-scrambling out of a noun complement is acceptable (5a), while NP-scrambling out of a relative clause is not (5b). The judgments in (5) are from Haig (1976).

Saito (1985; 1987) made the more nuanced claim that both noun complements and relative clauses are islands, with noun complements being relatively more acceptable (6a) than relative clauses (6b). The judgments in (6) are from Saito (1985).

To the best of our knowledge, Yano (2019) is the only published study to examine the acceptability of NP-scrambling out of noun complements with formal acceptability judgment experiments. The goal of Yano (2019) was to examine whether D-linked NPs like sono shoosetsu ‘the novel’ undergo syntactic movement when they appear in a fronted position. Yano (2019) uses island effects as a diagnostic of movement. To that end, Yano tested two island types: adjunct islands and noun complement islands. Yano (2019) tested both D-linked NPs (with sono ‘the/its’) as the target of investigation, and non-D-linked NPs (without sono ‘the/its’) as a baseline comparison. Here we focus exclusively on non-D-linked NPs because the effect of D-linked phrases, or lack thereof, is a potentially more complex topic of investigation that takes the island facts for non-D-linked NPs as a starting point (see Szabolcsi & Lohndal 2017 for a review of selective islands). In what follows, we focus on the discussion in Yano (2019) of noun complement islands.

In the first experiment of Yano (2019), the sentences were presented in isolation. In the second experiment, the sentences were presented with a context sentence such as “The novel received the Naoki prize.” to establish the fronted object in the discourse.³ Yano (2019) used the factorial definition of island effects in which the presence of an island effect appears as a superadditive interaction of two (or more) factors that are themselves independent of the island effects (Sprouse 2007; Sprouse et al. 2011; Sprouse et al. 2012, a.o.). For the Yano (2019) experiments these factors were structure, manipulating the structure of the embedded clause (either an island or a non-island), and word.order, manipulating the presence or absence of scrambling out of the embedded clause. An example set of the four conditions is given in (7) for completeness; we review the logic of the factorial design, and provide full examples for our experiments, in Section 3.

The results of the two Yano (2019) experiments are potentially conflicting. In the first experiment (without context), Yano (2019) finds a small superadditive interaction indicative of a noun complement island effect. In the second experiment (with context), Yano (2019) finds no superadditive interaction indicative of a noun complement island effect. There are at least two issues that make the Yano (2019) results difficult to interpret with respect to the question of the islandhood of noun complements (which, to be fair, was not the research question for Yano 2019). First, the Yano (2019) results showed very low acceptability even for putatively grammatical scrambling out of non-island embedded clauses (a declarative CP), in both experiments. Yano notes that this could be due to the long-before-short preference – that is, a preference in Japanese that scrambled NPs be longer than the NPs that they are scrambled over (Dryer 1980; Hawkins 1994; Yamashita & Chang 2001; Yamashita 2002; Omaki et al. 2020). The scrambled NPs in the Yano 2019 experiments are single-word NPs (presumably because of the focus of the study on D-linking), which violates the long-before-short preference, and therefore could have pushed the overall acceptability down. This in turn could have reduced the size of the superadditive interactions (if the long-before-short preference is not additive with island effects, which is itself a potentially interesting observation that might merit future study). Second, as mentioned above, the second experiment manipulated the presence versus absence of a context. The results show that there was no superadditive interaction with the “without context” (“non-D-linked” in Yano’s terms) sentences, while there was a small superadditive interaction with the “with context” (“D-linked” in Yano’s terms) sentences. Here, it is important to reiterate that the experimental sentences for the “without context” condition in the second experiment were identical with respect to island-relevant properties to the non-D-linked experimental sentences in the first experiment (e.g., the scrambled NPs were bare NPs without a demonstrative). Thus, the superadditive interaction observed in the first experiment disappeared in the second experiment despite the fact that virtually identical sentences were judged in these two experiments. Furthermore, the directionality of the result – no island effect without context and an island effect with context – runs contrary to the directionality predicted by theories that predict D-linking to ameliorate island effects. Given the complex pattern of results between the two experiments, Yano (2019) speculates that noun complements may show more between-participant variability than other island types.

2.3 The motivation of the current study

The contradictory results for noun complements between Haig (1976) and Saito (1985; 1987), and between the two experiments in Yano (2019), suggest that additional systematic data collection is needed, with special attention paid to the long-before-short preference as well as to the possibility that noun complements show more between-participant variability than other island types. To that end, here we report the results of two formal acceptability judgment experiments testing whether NP-scrambling out of complex NPs invokes island effects, both with materials that respect the long-before-short preference, and both with large sample sizes (89 and 90 participants, respectively) to allow for high statistical power and the possibility of exploring between-participant variability.⁴

Scrambling dependencies are conventionally used to test island effects in Japanese due to the lack of overt wh-movement (for a discussion of island effects involving wh-in-situ, see Sprouse et al. 2011; Kim & Goodall 2016; Tanaka & Schwartz 2018; Lu et al. 2020). But before we move on to discuss our experiments, a caveat is in order concerning some characteristics of scrambling and the design of our experiments. First, it has been argued that some instances of scrambling exhibit properties of A-bar-movement while others exhibit properties of A-movement. In our study, all instances of scrambling are long distance, i.e., they always cross a clausal boundary. Since the consensus in the literature is that long distance scrambling is A-bar-movement (e.g., Saito 1992; Yoshimura 1992; Nemoto 1993; Tada 1993; see Nemoto 1999 for a comprehensive review of the relevant literature), we assume that all the instances of scrambling examined in this study are instances of A-bar-movement. Second, unlike wh-movement and relativization, scrambling is optional and has been claimed to be “semantically vacuous” (e.g., Saito 1989; but see Miyagawa 2001 for a claim that local scrambling can be triggered by the EPP and, therefore, obligatory). The fact that scrambling is an optional process raises questions about its motivations. Factors such as constituent weight (Yamashita & Chang 2001; Omaki et al. 2020) and information-structure status (Koizumi & Imamura 2017) have been shown to affect the production, acceptability, and processing of scrambling sentences. The optional nature of scrambling also raises the possibility that the effect of scrambling on acceptability judgments in non-island environments might be different from the effects of wh-movement and relativization on acceptability judgments in similar environments. Though the question of whether scrambling differs qualitatively and/or quantitatively from the other types of A-bar dependencies is interesting in its own right, we believe it is beyond the scope of our study, as it requires a broad set of experiments that compares different types of A-bar dependencies, whereas our experiments were narrowly constructed to explore NP-scrambling. Finally, there is one important difference between our experiments with NP-scrambling and previous studies that investigated wh-questions: whereas previous studies of wh-questions manipulated the distance of wh-movement dependencies, with wh-movement originating in either the matrix clause (short) or embedded clause (long), our experiments on NP-scrambling (and those in Yano 2019) manipulated the presence or absence of long-distance scrambling, not the distance of the scrambling. This is because an instance of NP-scrambling is unambiguously A-bar-movement only if it is long distance – thus, one cannot compare instances of short and long scrambling sentences without introducing yet another factor, such as the A-bar vs A distinction, that might affect their acceptability. We therefore note that the presence of scrambling may incur a larger main effect, as the mere presence of a long-distance dependency alone has been shown to cause a significant decrease in acceptability compared to sentences without a long-distance dependency (e.g., Kluender & Kutas 1993). A potentially larger main effect in turn raises the possibility that the superadditive interaction might cause a floor effect; we discuss this possibility as part of the description of the logic of the design in Section 3.1 below, and we note that there is no evidence of floor effects in our results in Sections 3.4 and 4.4.

3.1 The logic of the design

Experiment 1 tested three island types: noun complements, relative clauses, and coordinate structures. By including both types of complex NPs together in the same experiment, we can investigate Saito’s (1985; 1987) claim that noun complements yield smaller island effects than relative clauses. We included coordinate structures because they are uncontroversially considered islands in the literature (Harada 1977), and therefore can serve as a type of baseline comparison for both relative clause islands and noun complements.

We employed the factorial definition of island effects, both because we believe it matches the logic that has historically been used by syntacticians to define island effects, and because it allows us to eventually integrate our results with the growing cross-linguistic experimental literature using the factorial definition (a.o., Christensen et al. 2013;⁵ Almeida 2014; Kim & Goodall 2016; Sprouse et al. 2016; Keshev & Meltzer-Asscher 2018; Kush et al. 2018; 2019; Stepanov et al. 2018; Tanaka & Schwartz 2018; Ko et al. 2019; Lu et al. 2019; Tucker et al. 2019; Omaki et al. 2020). As described below, we implement the factorial design completely within participants, allowing us to quantify to what extent each participant reports an island effect, so that we can investigate the conjecture from Yano (2019) that noun complements may show a higher degree of between-participant variability than other island types. Finally, we use relatively long scrambled NPs to satisfy the long-before-short preference.

The factorial design has two factors: scrambling manipulates the presence or absence of NP-scrambling (no-scrambling/scrambling), and structure manipulates the structure of the embedded clause (non-island/island). Fully crossing these two factors in a 2×2 design leads to four conditions. In (8), we illustrate all four conditions for noun complements. Note that the NPs that are the target of scrambling are outlined with a box.⁶

(9) and (10) below show all four conditions for relative clauses and coordinate structures, respectively. The non-island structure that we chose for relative clauses was a declarative CP. The non-island structure that we chose for coordinate structures was an NP-PP sequence.

The value of the factorial definition is that it isolates the island effect in the interaction between scrambling and structure (while subtracting out the main effects of those factors). If there is no island effect, we expect to see no interaction as illustrated in the left panel of Figure 1, where the two lines that connect the two means for the island condition sentences and the non-island condition sentences are parallel. If there is an island effect, we expect to see a superadditive interaction as illustrated in the center and right panels, where the two lines are not parallel because the mean for the scrambling/island condition sentence is lower than expected if the effects of the two manipulations are all there are. Crucially, we can also look at the size of the interaction as a measure of the size of the island effect (e.g., to test the claim by Saito 1985; 1987); the center panel illustrates a smaller effect, and the right panel illustrates a larger effect.

Figure 1

Possible results using the factorial definition of island effects.

As discussed in Section 2.2, one factor that makes our experiments different from previous studies that examined other types of A-bar dependencies is that our second factor is the absence versus presence of an A-bar dependency, i.e., NP-scrambling, while previous studies manipulated the distance of the dependency (e.g., wh-movement that originated in the matrix versus embedded clause). Because of this difference, our experiments might show a larger main effect of the second factor than is observed in previous studies. Figure 2 demonstrates this. The larger main effect than in Figure 1 appears as a steeper downward slope in the non-island structure line.

Figure 2

Possible results with hypothesized larger main effects of scrambling.

One concern that arises with large main effects is that they make a floor effect likely with superadditive interaction terms. A floor effect arises when the superadditive interaction is so large that it should push the island/scrambling condition beneath the lower bound of the scale, but because the scale does not go any lower, the island/scrambling condition is rated higher than it should be (because it is metaphorically stopped by the floor of the scale). The end result is an underestimation of the island effect size. Though we cannot eliminate this possibility, we can check for the possibility of floor effects by plotting the rating of the least acceptable filler in each plot as a solid gray line as an estimate of the functional floor of the scale. If the island/scrambling condition is lying precisely on this line, then a floor effect is likely (though not certain). If the island/scrambling condition is above this line, then a floor effect is not likely.

3.2 Materials and survey construction

Each participant completed a survey that consisted of 58 items: 6 practice items, 12 experimental items and 40 filler items pseudorandomized to avoid related experimental items appearing in succession. The 12 experimental items consisted of 1 token of each of the 4 conditions for each of the three islands. We chose one judgment per condition per participant to keep the total number of experimental items low to minimize the chance that participants would notice the goal of the experiment. We compensated for the increased risk of noise with one judgment per condition by testing a sample size (n = 89) that is likely to yield very high statistical power for medium and large effect sizes, and moderate statistical power for small effect sizes (Sprouse & Almeida 2017). We created 8 lexically matched sets (of 4 conditions) of items per island. The items were then distributed among 8 experimental lists, each 4 items long (one per condition in the factorial design), using a Latin square procedure so that participants saw a unique lexical item in each condition. We identified 4 errors in the item codes (out of 96 items across lists) after the experiment. We corrected these errors during analysis, but it meant that the total number of observations per condition were mildly uneven (Table 1).

3.3 Participants and presentation

Ninety-one participants from two universities in Tokyo, Japan, participated in the experiment. We excluded two participants from analysis because their answers to our language background questionnaire revealed that they had significant exposure to a language other than Japanese before they were 10-years old. Eighty-nine self-reported native speakers of Japanese remain in the analysis. Participants either received course credit for their participation or 500 yen. The experiment was administered online using IBEX (Drummond 2013). Each sentence was presented one at a time on its own presentation screen with a 1 (mattaku fushizen ‘completely unnatural’) to 7 (mattaku shizen ‘completely natural’) scale. Participants indicated their rating by clicking on the appropriate number. Because complex NP islands may show variability across participants, we did not exclude any participants from analysis because of the distribution of their judgments.

3.4 Results

In this section we describe the results of Experiment 1, with a particular focus on (i) the presence or absence of the superadditive interaction indicative of island effects (and, relatedly, the relative size of the effect) and (ii) the variability of island effects across participants.

3.4.1 The presence or absence of island effects

To determine the presence or absence of island effects, we will look for two properties: (i) a visual pattern indicating a superadditive interaction among the 4 conditions in the factorial design (as illustrated in Figures 1 and 2), and (ii) statistical corroboration of the superadditive interaction. To assess the visual patterns in the results, Figure 3 reports the means and estimated standard errors (±1) for each condition, arranged in an interaction plot. We present the results two ways. The first is as z-score transformed scores (by participant), which reduces the impact of common forms of scale bias among the participants. The second is as raw results from the 7-point scale. Though we believe that the z-score transformed scores are likely the best option for analyzing acceptability judgments, as one anonymous reviewer points out, the raw scores allow us to evaluate the effect of the z-score transformation. The one caveat is that we must be sure not to exercise researcher degrees of freedom by selecting the results that we prefer. For this project, there is no risk – the z-score transformed scores and raw scores yield the same results.

Figure 3

Experiment 1. Interaction plots for NP-scrambling in Japanese. Points are condition means. Error bars represent 1 estimated standard error in either direction. The top row reports the z-score transformed results, and the bottom row reports the raw results. The columns report each island type. The horizontal gray lines indicate the mean rating of the highest and lowest rated filler type to help assess ceiling and floor effects. For space reasons, p-values are rounded to a floor of .0001 and Bayes factors are rounded to a ceiling of 100.

For statistical corroboration, we conducted two types of analyses: one in a null hypothesis testing framework and one in a Bayesian framework. For the null hypothesis test, we constructed linear mixed effects models with scrambling and structure as fixed effects and participant and item as random effects (intercepts only) for each island type using lme4 package in R (Bates et al. 2015). We calculated p-values using the lmerTest package, which uses the Satterthwaite approximation for degrees of freedom (Kuznetsova et al. 2017).⁷ We will interpret p-values below the conventional threshold of .05 as evidence against the null hypothesis, and therefore by implication, corroboration of the presence of an island effect. We will interpret p-values above the conventional threshold of .05 as a failure to reject the null hypothesis. Because the failure to reject the null hypothesis cannot be interpreted as evidence in support of the null hypothesis (because the null hypothesis is assumed to be true in the calculation of p-values), we include a Bayesian analysis to directly evaluate the null hypothesis.

For this Bayesian analysis, we calculated Bayes factors for the interaction term for the fixed effects in the linear models using the BayesFactor package (Morey & Rouder 2018). The Bayes factors reported here are of the BF₁₀ type: they report the ratio of the likelihood of the data under the experimental hypothesis (H1) that an interaction is present to the likelihood of the data under the null hypothesis (H0) that there is no interaction present. Following Jeffreys (1939/1961), we will interpret a BF₁₀ greater than 3 as strong evidence that an interaction is present, as this indicates that the data is at least 3-times more likely under a theory in which the interaction is present than one in which the interaction is absent. Similarly, we will interpret a BF₁₀ less than 0.33 as strong evidence that there is no interaction, as this indicates that the data is 3-times more likely under the null hypothesis that the interaction is absent. We will also interpret Bayes factors between 0.33 and 3 as inconclusive (as the data is equally likely under both theories). In Figure 3, we have added the interaction term p-value and interaction BF₁₀ to each cell of the plot so that the visual patterns and statistical results can be evaluated simultaneously.

As Figure 3 makes clear, in Experiment 1, we see clear evidence of island effects with both relative clauses and coordinate structures – the visual pattern suggests a superadditive interaction, and both statistical analyses corroborate the interaction. However, for noun complements, we see no visual pattern of an interaction. The p-value is substantially above the conventional threshold, suggesting a failure to reject the null hypothesis. The Bayes factor is 0.43, which is close to the conventional threshold of 0.33, and suggests that the data is about 2.5-times more likely under the null hypothesis than under the experimental hypothesis. (The Bayes factor for the raw judgments is at 0.33, but we believe z-scores are the more principled analysis, and therefore focus on the statistical results for z-scores to avoid the appearance of leveraging researcher degrees of freedom to our benefit.) We also note that the mean rating of the island violating condition for noun complements is above the middle of the raw scale (above 4) and right at the middle of the z-score scale, which represents the mean judgment of all the items in the experiment (target conditions and fillers). This is noticeably different from the mean ratings for the island violating conditions for relative clauses and coordinate structures, as both are substantially below the midpoint of both the raw and z-score scales. We thus conclude that Experiment 1 suggests that noun complements are not islands for NP-scrambling, while relative clauses and coordinate structures are. (We also note that there is no evidence of floor or ceiling effects as the mean ratings of all conditions lie below the means of the highest rated fillers and above the means of the lowest rated fillers.)

3.4.2 Variability in island effects between participants

As briefly discussed in Section 2.2, one possibility raised by Yano (2019) to explain the complex pattern of results for noun complements across the two experiments that he reports is that the island status of noun complements may show more between-participant variability than other island types. To investigate this, we calculated the size of the island effect reported by each participant as a differences-in-differences (DD) score (Maxwell & Delaney 2003): (non-island/scrambling – island/scrambling) – (non-island/no-scrambling – island/no-scrambling). These DD scores will be positive when the participant shows a superadditive interaction indicative of an island effect, with the magnitude indicating the size of the effect; these DD scores will be 0 when the participant shows no interaction, and negative if the participant shows a pattern in which the island-violating condition is more acceptable than the main effects of structure and scrambling would predict (this latter case is not predicted by any theory, so may be indicative of noise in that participant’s responses).

Figure 4 reports the distribution of island effect sizes by participant as measured using DD scores for both z-scores and raw scores using histograms overlaid with probability density estimates.

Figure 4

Experiment 1. The distribution of island effect sizes by participant, calculated as differences-in-differences scores for both z-scores and raw scores. The solid line is an estimate of probability density.

One clear sign that noun complements are more variable than the other islands would be for the distribution for noun complements to be wider than the distributions for the other islands. However, this is not what we see in Figure 4. If anything, noun complements show a narrower distribution. What we see instead is that noun complements show a relatively normal distribution centered exceedingly close to 0, as expected if there is no island effect, while the other two islands show distributions that are substantially shifted toward the positive range, as expected if there is an island effect. We therefore conclude that it is unlikely that noun complements show more between-participant variability than the other island types. Instead, we see further corroboration from the relatively normal, and relatively narrow, distribution for noun complements that there is no island effect.

4.1 The logic of the design

For Experiment 2, we modified the design in two ways. First, we increased the number of tokens that participants rated per condition from one to two. This allows us to investigate not only the presence of island effects and the variability between participants as in Experiment 1, but also the variability of each condition within participants across the two ratings. We will therefore report three subsections of results for Experiment 2. Second, we used the same non-island structure for all three island types – specifically, an embedded declarative CP. This is a logically possible non-island structure for all three islands, therefore testing it adds a dimension of generality to our results. (The logic of the factorial design is such that the measurement of the island effect, which is in the interaction term, will not be affected by the choice of the non-island structure, as long as the non-island structure does not itself induce an interaction with scrambling. The only consequence of this change is in the main effect of structure.) Using the same non-island structure across all three islands also reduced the number of conditions tested (by 4), helping to offset the increase in tokens per condition.

4.2 Materials and survey construction

In Experiment 2, each participant completed a survey that consisted of 60 items: 16 experimental items (2 each of 8 target conditions) and 44 filler items pseudorandomized to avoid related experimental items appearing in succession. The 8 target conditions were non-scrambling and scrambling versions of declarative CPs (as in 8a and 8b), noun complements (8c and 8d), relative clauses (as in 9c and 9d), and coordinate structures (10c and 10d). We created 4 lexically matched sets of items per structure. The items were then distributed among 2 experimental lists using a Latin square procedure so that participants saw a unique lexical item in each trial.⁸

4.3 Participants and presentation

Ninety-three participants from two universities in Tokyo, Japan, participated in the experiment. We excluded three participants from analysis because their answers to our language background questionnaire revealed that they had significant exposure to a language other than Japanese before they were 10 years old. Ninety self-reported native speakers of Japanese remain in the analysis. Participants either received course credit for their participation or 500 yen. The presentation was identical to Experiment 1.

4.4 Results

In this section we describe the results of the experiments, with a focus on the three questions licensed by the new design: (i) the presence or absence of the superadditive interaction indicative of island effects (and, relatedly, the relative size of the effect), (ii) the variability of island effects across participants, and (iii) the consistency of participants’ ratings across the two tokens of each condition.

4.4.1 The presence or absence of island effects

As Figure 5 shows, Experiment 2 again revealed clear evidence of island effects with coordinate structures and relative clauses – the visual pattern suggests a large superadditive interaction, and both statistical analyses corroborate the interaction. However, for our critical case, noun complements, the pattern is more complicated. There is a small visual trend toward an effect, albeit with the potential island violating condition (island/scrambling) above the mid-point of the 7-point scale, and the two statistical tests trend in opposite directions: the p-value is well above the conventional threshold of .05, suggesting no evidence of an island effect, while the Bayes factor is trending toward the conventional threshold of 3 for the z-transformed results (although it is still technically below it).⁹

Figure 5

Experiment 2. Interaction plots for NP-scrambling in Japanese. Points are condition means. Error bars represent 1 estimated standard error in either direction. The top row reports the z-score transformed results, and the bottom row reports the raw results. The columns report each island type. The horizontal gray lines indicate the mean rating of the highest and lowest rated filler type to help assess ceiling and floor effects. For space reasons, p-values are rounded to a floor of .0001 and Bayes factors are rounded to a ceiling of 100.

We prefer to be cautious in interpreting this pattern of results. The logically weakest conclusion we can make is that there is no clear evidence of an effect for noun complements, but we also cannot entirely rule it out. A hypothetical future experiment with even more observations per condition could potentially detect a very small effect using this design. We can also conclude that, if noun complements do have this hypothetical effect, it differs from the effects that we see for relative clauses and coordinate structures in important ways. For one, this hypothetical effect would be substantially smaller in size. It must be so small that it did not appear at all in Experiment 1 with 89 participants and one token per participant and does not reliably appear in Experiment 2 with 90 participants and two tokens per participant. In contrast, relative clauses and coordinate structures show relatively large and reliable effects in both experiments. (According to Sprouse & Almeida 2017 only 37 participants and one token per participant are necessary for 80% power to detect medium effect sizes and only 17 participants to detect large effect sizes.) For another, as the raw scores show, the island violating condition (island/scrambling) for noun complement islands is rated above the midpoint of the scale (around 4.5 in both experiments) and does not appear to result in unacceptability, in contrast with relative clause and coordinate structure islands (the island violating condition is in the lower half of the scale). We also note that the island/scrambling condition of the coordinate structure island is rated below the mean of the lowest rated filler. This means that it could be sitting at the floor in a world in which our lowest rated filler failed to be as low as the floor. But since coordinate structures show the largest island effect in the experiment, the theoretical consequence of underestimating its effect size is minimal. Instead, it tells us that the island/scrambling condition for relative clause islands is not sitting at the floor, despite being roughly equal in acceptability to the lowest rated filler. This underscores how much larger these island effects are compared to the hypothetical effect for noun complements, and how much lower the ratings of the island violating conditions are.

To summarize, by conventional statistical criteria, Experiment 2 provides strong evidence for large, classic island effects with relative clauses and coordinate structures, but no evidence either for or against island effects with noun complements. If one wishes to interpret the visual and BF trend as evidence that there may be a small effect of noun complements that we failed to detect, one must also conclude that it differs substantially from relative clause and coordinate structure islands both in size and in location in the scale.

4.4.2 Variability in island effects between participants

Turning next to the variability in island effects between participants in Experiment 2, Figure 6 shows that noun complements once again show a relatively narrow normal distribution. There is no evidence that there is excessive variability in noun complements compared to relative clauses and coordinate structures. We do note, however, that the center of the distribution for noun complements is shifted slightly toward the positive, as expected given the small trend that we saw in the mean ratings in Figure 5. The two island types continue to show the same substantial shift toward the positive that we saw in Experiment 1.

Figure 6

Experiment 2. The distribution of island effect sizes by participant, calculated as differences-in-differences scores for both z-scores and raw scores. The solid line is an estimate of probability density.

4.4.3 Variability in island effects within participants

Though there is no evidence of increased variability for noun complements between participants, it is possible that there is increased variability within participants. Recent work by Kush et al. (2018; 2019) in Norwegian has suggested that some island effects that appear relatively small when viewed through the grand means of the sample may in fact be driven by inconsistent judgments within each participant.¹⁰ Though the source of this inconsistency is still an open area of investigation, here we provide a similar analysis for the ratings in Experiment 2.

Figure 7 plots the two judgments that each participant gave for each structure in a scatterplot, with the first judgment along the x-axis and the second judgment along the y-axis. The columns represent each structure, and the rows separate the no-scrambling (top row) and scrambling (bottom row) conditions. We divide each plot into four quadrants. A point in the top right quadrant (Quadrant 1) represents a participant who rated both tokens in the upper half of the scale. For convenience, we will label such a pattern consistent acceptor. A point in the bottom left quadrant (Quadrant 3) represents a participant who rated both tokens in the lower half of the scale. We can label such a pattern consistent rejector. The other two quadrants (Quadrants 2 and 4) represent participants who rated one token in the upper half of the scale and one token the lower half of the scale. We will label this pattern inconsistent. We have added two features to make the plot a bit easier to read: colors representing the three patterns, and two-dimensional (joint) probability density estimates to draw attention to the density of the points in each location. Similar to a topographic map, in a two-dimensional probability density plot, concentric circles that are closer together represent higher density (because, like topographic maps, these plots are looking down on the peaks in the density space from directly above).

Figure 7

Experiment 2. Scatterplots of the ratings for the two tokens of each condition for each participant, with two-dimensional (joint) probability density estimates overlaid. The points are colored according to the type of judgment pattern defined by the midpoint (0) of the z-score scales.

Figure 8 presents the same plot for the raw scores. We include the raw scores here because we saw in Figure 5 that the midpoint of the z-score scale corresponded with ratings above the midpoint of the raw score scale. This suggests that, on average, the balance of items in the survey was slightly skewed toward higher ratings. This means that the dividing lines in Figure 7 represent consistency relative to the midpoint of the distribution of items in the survey, while the dividing lines for the raw scores in Figure 8 represent consistency relative to the absolute midpoint of the raw rating scale. We have added a small amount of jitter to the points in Figure 8 to make all of the points visible (without jitter, there is quite a bit of overlap among points because the raw scale only allows 7 distinct ratings).

Figure 8

Experiment 2. Scatterplots of the ratings for the two tokens of each condition for each participant, with two-dimensional (joint) probability density estimates overlaid. The points are colored according to the type of judgment pattern defined by the midpoint (4) of the raw score scales. The integer nature of the raw scale would normally mean that many points perfectly overlap. We have added a small amount of jitter to the points to make them all visible.

In both figures we see the same patterns. The no-scrambling conditions (top row) appear to show the upper bounds of consistency in this experiment – the vast majority of participants show the consistent acceptor pattern (Quadrant 1) for each structure, with a small number of inconsistent patterns mixed in. In contrast, the scrambling conditions (bottom row) reveal potentially relevant patterns. In the first column, for the by-hypothesis grammatical scrambling out of declarative CP, we see that the largest mass of participants is in the consistent acceptor quadrant, while there is also a non-negligible number of participants in each of the other three quadrants. This provides more nuance to the mean rating in Figure 5 – we see now that the middle-of-the-scale mean rating (near 0) was actually driven by a mix of consistent acceptors, consistent rejectors, and inconsistent participants. This sets a baseline expectation for NP-scrambling consistency: the rating of NP-scrambling itself, in the absence of islands, is relatively variable in Japanese. We can then apply this baseline as we look at the potential island structures.

For noun complements, we see a shift in the probability mass that moves a bit to the left and down from the declarative CP baseline. The center of mass does not quite fully cross into the consistent rejector quadrant, but instead hovers over the horizontal axis line (indicating a rating near 0 for the second token). This shift from the baseline established by scrambling out of declarative CPs is in line with the equivocal results that we saw for noun complements in the means in Figure 5 and the DD scores in Figure 6 – there is a small trend toward a slightly negative rating, but enough variability that it is still plausibly an effect around 0. For relative clauses, we see a shift further left and down. Though there is still variability in relative clauses, the vast majority of participants are either consistent rejectors or inconsistent raters, consistent with the grand means in Figure 5 and DD scores in Figure 6. Finally, we see an even further shift toward the consistent rejector quadrant for coordinate structures, which is again consistent with the grand means in Figure 5 and the DD scores in Figure 6.

Taken together, what we see is that scrambling itself introduces a fair amount of within-participant variability to judgments when compared to no-scrambling conditions. Noun complements appear (visually) to show roughly the same amount of variability as scrambling from declarative CP structures. This leads to two conclusions. The first is that there is no additional variability in noun complements that could explain past debates about their island status. The second is that noun complements show the same general pattern of variability as unequivocally grammatical sentences. This stands in contrast to the unequivocal islands, which tend to show a mild increase in consistency compared to the grammatical controls. This again points to noun complements being qualitatively distinct from the other islands (and perhaps more similar to grammatical sentences), and thus further corroborates our conclusion that noun complements in Japanese are not islands for NP-scrambling.