1 Introduction

All natural languages change over time. The way each new generation of speakers pronounces their words is subtly different from their parents, new words replace old ones, marginal grammatical paradigms become the norm, and norms dissolve. Many authors have suggested that language change, like other evolutionary processes, involves both directed selection as well as stochastic drift (Sapir 1921; Jespersen 1922; Andersen 1990; McMahon 1994; Croft 2000; Baxter et al. 2006; Van de Velde 2014; Steels & Szathmáry 2018). Systematically quantifying the relative contribution of these two processes — particularly with reference to individual time series — is an open problem.

There are a number of ways in which selective biases may influence language change. For example various cognitive biases have been postulated as important in the evolution of language (Haspelmath 1999; Croft 2000; Kirby, Cornish & Smith 2008; Fay et al. 2010; Smith, Tamariz & Kirby 2013; Enfield 2014; Tamariz et al. 2014) and one might therefore expect to see manifestations of these in instances of language change. Selective advantage stemming from sociolinguistic prestige of (the users of) competing variants has been shown to play a considerable role in change, both via competition between forms within the language community as well as borrowing from other languages (Labov 2011; Hernández-Campoy & Conde-Silvestre 2012). A foreign or novel variant may also be selected for by virtue of filling a lexical or morphosyntactic gap (McMahon 1994; Trask 1996). The form of a variant alone may convey a selective advantage. For example, it has been observed that, all other things being equal, speakers prefer shorter forms that take less effort to utter (Zipf 1949; Kanwal et al. 2017) and limited iconicity can be advantageous (Dingemanse et al. 2015). Various usage and acquisition properties have been shown to be predictors of success (Kershaw, Rowe & Stacey 2016; Calude, Miller & Pagel 2017; Grieve, Nini & Guo 2018; Monaghan & Roberts 2019). There is also evidence that certain phonetic changes are more likely than others, due to the articulatory and acoustic properties of human speech sounds (Ohala 1983; Baxter et al. 2006). In certain circumstances there may be even qualitative evidence of directed selection, such as knowledge of previous activities of some authoritative language planning body, prescriptive grammars, or other exogenous forces (Rubin et al. 1977; Anderwald 2012; Ghanbarnejad et al. 2014; Daoust 2017).

It is a reasonable hypothesis that, given adequately large and representative samples of language use over time (i.e., corpora), signatures of selection should be inferable from the usage data alone. This idea has recently been explored in a number of works (Hahn & Bentley 2003; Bentley 2008; Reali & Griffiths 2010; Blythe 2012; Sindi & Dale 2016; Amato et al. 2018), and has been also applied to domains of cumulative cultural evolution beyond language (Kandler, Wilder & Fortunato 2017; Kandler & Crema 2019). One of the more ambitious attempts is that of Newberry et al. (2017), who employ a standard method borrowed from the field of population genetics, which also deals with the inference of selection in a population and the assessment of drift in evolution. We will henceforth refer to this work as “Newberry et al.” (an earlier version of the paper is Ahern et al. 2016). They use the Frequency Increment Test (Feder, Kryazhimskiy & Plotkin 2014), or FIT for short, and make an explicit connection with the Wright-Fisher model (Wright 1931; Ewens 2004) of neutral stochastic drift (not unlike a previous similar contribution, Sindi & Dale 2016).

Newberry et al. consider three grammatical changes in the English language. Their main focus is the (ir)regularization of past-tense verbs (e.g. the change from irregular snuck to regular sneaked), a topic that has been of some interest (Lieberman et al. 2007; Cuskley et al. 2014; Gray et al. 2018). They also investigate the change in periphrastic do (say not that! becoming don’t say that!), the evolution of verbal negation (from the Old English pre-verbal to the Early Modern English post-verbal), and possible phonological neighborhood effects (which we will not discuss here). They use data from the Corpus of Historical American English (Davies 2010) and the Penn Parsed Corpora of Historical English (Kroch & Taylor 2000). Their method consists of calculating the relative frequencies of alternative forms in a corpus (e.g., the relative frequency of the irregular past tense form snuck against that of the regular sneaked), placing the count data into variable-length temporal bins, and running the FIT on the resulting time series. Ultimately, the test yields a p-value under the null hypothesis of change by drift alone. They also infer the “effective population size” of the verbs and show that the strength of drift (in a subset of verbs with a FIT p > 0.2) correlates inversely with corpus frequencies, echoing the analogous observation about small populations in genetics.

The FIT points towards selection being operative in some cases, while labelling others (in fact, most changes in past-tense forms) as changes stemming from drift. In this work, we replicate this analysis (using Newberry et al.’s original code; see the Data Availability section in the end). We highlight an important methodological issue that arises when applying the FIT to linguistic data and which should be taken into account in future applications of the FIT (and similar tests) to identify cases of selection from linguistic corpora. The key issue lies in the construction of the time series via binning counts (e.g. from a corpus), and the application of the test in question to such time series, but we also draw attention to issues more specific to diachronic language data. While the FIT may be an appropriate test in some cases, we show that an incautious application of the FIT to linguistic data can end up incorrectly identifying cases of drift as cases of selection, and missing subjectively clear cases of selection.

While the approach of applying a test of selection to corpus-based time series shows promise as a method of linguistic analysis, we believe these issues deserve further investigation. We briefly explain the technical aspects of temporal binning and the FIT in the next subsections.

2 The FIT and binning decisions in linguistic corpora: A reanalysis of English past tense verb regularization

We focus here on the main result of Newberry et al. — the application of the FIT for assessing time series of verb form frequencies in order to determine if the observed patterns of change for 36 English verbs results from stochastic drift or selection. Technical data processing details described in this section are based on the Supplementary Information of Newberry et al., their code, and M. Newberry, p.c.

They construct a time series for each of 36 pre-selected verbs using 200 years of data in the Corpus of Historical American English (COHA), by counting how many times the regular past tense form occurs relative to the total number of instances of either the regular or irregular form. The yearly verb count series are then binned (grouped) into a number of variable-width quantile bins n(b) = ┌ln(n(v))┐, where n(v) is the sum of both (regular and irregular) past tense form tokens of the verb counted across the entire corpus. For example, light.PAST occurs n(v) = 8869 times in the corpus, resulting in ┌ln(n(v))┐ = 10 bins to group the years where the verb occurs. The first bin contains years 1810–1863 (and contains 897 tokens), the second 1864–1886 (890 tokens) and so on, up to the tenth (1994–2009, 884 tokens). Since the grouping is by years (years being the time resolution of the corpus), the bin size varies slightly in the exact number of tokens falling into each bin. More frequent verbs thus get more bins (up to 13), whereas less frequent verbs get fewer bins (down to 6). For each verb in each bin, the relative frequency of its regular past tense form in [0, 1] is calculated. Since the FIT assumes relative frequencies in (0, 1), Laplace +1 smoothing is applied to count values in bins where one of the variants has no occurrences at all in this section of the corpus.

As discussed above in the section on corpus binning, some temporal segmentation process is necessary. The binning procedure applied by Newberry et al. is somewhat different from the more common strategy of using fixed length bins such as years or decades. The advantage of their approach is that there is guaranteed to be data in every bin (whereas a low frequency lexeme might be entirely absent in a fixed-width bin), the bins are roughly the same size in terms of tokens, and the resulting increments tend (although are not guaranteed) to be normally distributed with equal variance. These properties are beneficial for the FIT, more likely yielding normally distributed increments with less sampling noise (Feder, Kryazhimskiy & Plotkin 2014). It should be noted though that the resulting bins differ quite widely in their temporal granularity — e.g. in the example above, the longest bin covers the earliest 53 years of the corpus, the shortest covers the most recent 15 years, and different verbs will use different time windows depending on their frequency in the corpus. Since the COHA is smaller on the early end (less tokens per year) and bigger on the more recent end, variable-width bins of the verb data are systematically longer in the early 1800s compared to the 20^th century ones (cf. the Supplementary Appendix for more discussion).

The series of relative frequencies based on the resulting bins are fed into the Frequency Increment Test to assess whether one may reject the null hypothesis of drift and assert that a given trajectory is therefore probably a product of selection. Newberry et al. set the FIT α = 0.05 but also report results for α = 0.2. They conduct the Shapiro-Wilk normality test on the transformed frequency increments, as the FIT assumes the increments to be normally distributed.

We replicate their original results, using their code, and furthermore explore the consequences of manipulating the size of the bins, in two ways. We present results for both binning strategies. That is, variable-width bins, n(b) = cln(n(v)), where c is an additional arbitrary constant, and c = 1 recovers the Newberry et al. procedure; and fixed-width bins, each set to a fixed duration in years.

Importantly, the fixed-width binning approach necessitates the introduction of an additional parameter: since some bins may end up with no or few occurrences of either form of a verb, we set a threshold of minimum 10 total occurrences for a relative value to be calculated in a bin; otherwise the bin is excluded before applying the FIT (hence also reducing the number of bins that make up the time series). As the FIT assumes values in (0, 1), smoothing of boundary values is required. But if there is only a single occurrence of a verb in a bin (meaning the single present form would be at 100%, the other at 0), then the +1 smoothing would force the relative value to be 50–50, which is undesirable. Similar distortions would happen with small frequency values, hence the threshold of 10. See the Supplementary Appendix for more discussion on the differences between these approaches and how different minimal frequency thresholds affect the results. A more conservative threshold (such as 100) would yield more reliable bins (and less noisy time points), but given the size of COHA, most verbs don’t have 100 occurrences per year (or some even in 5 years), which would preclude testing in shorter fixed bins.

Figure 1 shows the results of these various analyses, in terms of how many verbs (out of the 36) allow us to reject the null hypothesis of drift, given the thresholds mentioned in the original work, as well as taking into account the normality assumption of the FIT (see above). We use the Shapiro-Wilk normality test, following Newberry et al. (this test is of course subject to low power in small samples as well). Out of the 466 time series analyses summarised in Figure 1 (36 verbs times 13 binning choices, minus two series with not enough data points), 63% of the FIT p-values are eligible to be interpreted at Shapiro-Wilk α = 0.1.

Figure 1

Results of applying the FIT to time series constructed based on 200 years of COHA frequency data. The verbs are ordered by overall frequency (low on the left). The constant c determines the number of variable length bins via n(b) = cln(n(v)). c = 1 corresponds to Newberry et al.’s original results. 10 years corresponds to fixed bin length of 10 years, etc; “no bin” refers to no additional binning on top of the default yearly bins in the corpus. The colour of each point corresponds to the result of the FIT test of a verb time series in each binning (orange: p < 0.05, gold: 0.05 ≤ p < 0.2, light blue: p ≥ 0.2). The shape corresponds to the Shapiro-Wilk test result (filled circle: p ≥ 0.1, hollow square: p < 0.1, likely not normal), with cases of selection meeting the normality assumption highlighted by a larger circle. The column of numbers on the left displays the (rounded) median of the bins to years ratio in the given binning strategy. Only years where the verb occurs are counted (exclusion of sparse bins also leads the median in the no-binning version to be below 1). The listed variable (panel a) and fixed-width strategies (b) yield comparable binning ratios, e.g. the “c = 1” version is comparable to 20-year fixed-width. In summary, the results presented here demonstrate that the FIT is sensitive to the strategy used for binning.

We find that binning strategy does have an effect on the results, both in variable and fixed binning. Importantly, in broad strokes, the picture presented by Newberry et al. holds. They found that 6 out of 36 verbs undergoing selection; since the majority of verbs do not give a positive signal for selection, they interpret this as indicating that language change is often primarily stochastic. Looking at a wider range of binnings, we find that in most cases, there are indeed 5 ± 2 verbs that get flagged as undergoing selection at FIT α = 0.05, consistent with their conclusion. However, the specific verbs that are flagged as undergoing selection vary depending on the binning strategy. There are 4 verbs for which selection is detected in most binning choices — light, smell, sneak, wake (incidentally the ones with the strongest inferred selection coefficient, given the original binning, cf. EDT1 in Newberry et al.). There are also between 9 and 11 verbs (in variable-width binning; depending on how stringently the normality assumption is observed) which provide a robust absence of significant indications of selection, where the FIT p-value never drops below 0.2 regardless of binning. However, for the remaining verbs the decision as to whether or not they are undergoing selection depends on the binning choices. That being said, Newberry et al. do draw attention to the fact that results of applying the FIT come with a certain margin of error and report their false discovery estimates (30% for verbs with a FIT α = 0.05, 45% at 0.2).

Given that binning leads to different sample sizes of increments for the underlying t-test, those in turn being based on differing distributions of the tokens, some variance in the p-values is to be expected (not unlike in a replication of an experiment). The interpretation of our results and the appropriate conclusion regarding the sensitivity of the FIT test to binning strategy ultimately depends on one’s intention in carrying out a tests of selection in the first place. If the goal is to test a large set of series to determine general tendencies, as is the case for Newberry et al. then this approach may well be good enough — the qualitative result of Newberry et al. does broadly apply in most binning strategies.

However, most individual time series seem rather sensitive to binning, in the sense that the p-values fluctuate across conventional α levels between binnings. No verbs show an unambiguous signal of selection. For example, drift is not rejected in the time series of wed using the Newberry et al. binning, while it is when the number of variable-width bins is multiplied by 2. The verb sneak is significant at α = 0.05 in almost all the variable-width binnings, but in none of the fixed length ones; awake is significant in only a single explored binning strategy (variable-width with c = 0.5) and there are 4 more such verbs particularly sensitive to binning (the 1-year bins notwithstanding).

The no-binning results (i.e., using the default 1-year bins of COHA without further binning) differ visibly from the rest, but the normality assumption is also mostly violated. Given the small and variable bin sizes (tokens per bin), the same is likely true for the homoskedasticity assumption (although how much that matters and how to set a threshold is not clear, cf. the Supplementary Appendix). Most importantly, using “default” 1-year bins leads to testing on series where the increments are often based on very small samples, which is not desirable for any statistical test.

These evaluations obviously depend on the choice of α thresholds for the FIT and the supporting normality test — for example, a more stringent FIT α would lead to more verbs being classified as unambiguous cases of drift. In any case, if the intention is to test a particular example of linguistic change for selection (something a linguist may well be interested in), things become difficult. The issue diminishes if there is sufficient data on the variants, but that does not seem to be the case for many of the verbs tested here, given the size of COHA.

All in all, these findings merit a further investigation into the inner workings of the Frequency Increment Test and its applicability to corpus-based time series, which we will conduct in the following two sections.

3 The behaviour of the Frequency Increment Test in artificial time series

We construct a number of artificial examples (Figure 2) to probe the behaviour of the FIT on time series of length and character similar to those investigated in the original paper (which contained between 6 and 13 time points). The FIT can be shown to yield robust results for a certain range of series (as already shown by the subset of binning-insensitive verbs in the previous section). Yet we also observe a number of scenarios — time series that could be plausibly derived from linguistic corpora — where the results of the FIT are perhaps not what one might expect, from a language science point of view. To put it another way, this is the section where we push the FIT and see if it breaks. The next section demonstrates scenarios where the results of the FIT remain robust.

Figure 2

Artificially constructed time series of fictional variant relative frequencies (thick black lines, in (0, 1)); time on the x-axis. The rescaled increments (after adjusting for absorption) are shown as dotted grey lines with dash points, and their distribution is shown on the left side as a violin plot. Points of interest discussed in this section are highlighted with red on some panels. The FIT and Shapiro-Wilk test p-values are reported in the corners. This figure depicts a number of realistic scenarios where applying the FIT would yield unexpected results, due to either the range of the time series derived from the corpus (a, b), a difference in the number of data points (c), the sensitivity of FIT to near-zero values (d, e), and how stringently the assumption of the normality of the distribution of increments is being observed (e). This figure illustrates reasons to exercise caution when applying a test like the FIT to linguistic time series.

Each series in Figure 2 may be interpreted as the percentage of a variant of some fictional linguistic element over time (after binning). We calculate the FIT p-value of each series, as well as the Shapiro-Wilk test p-values. Figure 2.a draws attention to how the temporal range of the time series (or that of the coverage of the corpus) can lead to quite different conclusions. Both 2.a.1 and 2.a.2 are different ends of the same series (the overlap highlighted with the red circle). The series, if analysed as a whole, would yield a p_FIT = 0.02, but neither end on its own holds sufficient data to reject drift (nor is the FIT technically applicable, if the assumption of normality is observed). This perspective may explain the case of the purportedly drift-driven regularization of the verbs spill and burn, which are brought up in Newberry et al. as examples where drift alone is sufficient to explain the change, but which are problematic because the regular forms were already highly frequent by the early 19th century where the COHA coverage starts. spill starts out with a share of 55% regular forms in the first bin given the variable-width binning strategy; burn is at 86% regular. Under fixed decade binning, burn is 36% regular in the first bin, increasing to 62% and then to 82%, indicating a sharp increase characteristic of strong selection rather than drift (but obscured by the variable binning approach).

This example also points to a case where different evolutionary domains (genetics, language) might have different expectations about what a reasonable time-series characteristic of selection should look like. The FIT assumes the Wright-Fisher as the underlying model (reasonably so in population genetics). The long tail of near-zero values followed by a sudden increase in 2.a.1 is something that is unlikely to be observed in a Wright-Fisher model with constant selection strength parameter. However, from a linguistic point of view, this is a very natural series: a recent innovation or borrowing will be represented in the corpus as an increase preceded by a period of zero frequencies as far back as the corpus goes; this pattern could be explained as a recent change in fitness (e.g. a change in the subjective sociolinguistic prestige of a word).

A similar case is presented in Figure 2.b.1: if the time series chronicles both strong selection for one variant, and subsequent selection for the competing variant, then a blind application of the FIT will invariably indicate drift. Using only (either) half of the series as input to the test would yield a p-value indicating selection. knit is a verb undergoing a somewhat similar process, with usage spiking towards the regular (observable under finer binnings), followed by mostly irregular usage. Figure 2.b.2 is an example of the behaviour of FIT if the corpus coverage is too wide. The S-curve in the middle would yield a FIT p-value of 0.02 — in fact, it is the exact same curve as in Figure 2.c.2 (highlighted by the red dots). Yet the S being surrounded by (near-)absorption values, the FIT would indicate drift (were the test to be used despite the possible non-normality of the distribution).

In the case of real data, the part of the time series depicting the long period of no change could in principle be clipped away. This is straightforward if the “tail” consists of zeroes, but less so given small near-boundary values. Similarly, only the part of the time series far enough from the boundaries could be analysed (keeping in mind the specifics of the FIT, see above). However, any such solutions would introduce yet another researcher degree of freedom (what part of the series to include in the analysis) (cf. Simmons, Nelson & Simonsohn 2011).

Figure 2.c further illustrates how the FIT result is affected by a change in the way the time series is operationalised (e.g., using a different number of bins). 2.c.1 and 2.c.2 are S-curves with identical parameters, differing only in length (by 2 data points). Yet their FIT p-values are notably different (see the next section for more on sensitivity to binning differences). As expected, the FIT is sensitive to small changes if the sample is small (being based on the t-test). This may explain to some extent the changes in FIT p-values of short time series, between similar binnings differing only by a few points in length (cf. Figure 1). However, fewer bins can also lead to a lower p, if it results in a less jagged time series (likely the case for e.g. burn; cf. Section 4 for the effects of binning on drift series).

The examples so far however have had more to do with particularities of pre-test data manipulation. Figure 2.d illustrates a property of the FIT, its sensitivity to changes near the boundaries. 2.d.1 and 2.d.1 differ only by the value of the fourth data point, but the resulting FIT p-value is quite different (and furthermore the Shapiro-Wilk test indicates departure from normality in the increment distribution due to the outlier). The issue of applicability of the FIT to series with increments departing from normality is further illustrated with the last pair of series. 2.e.1 is a typical S-curve often observed in language change, but the non-normal distribution of its increments would disallow the interpretation of the FIT p-value (that would otherwise indicate a clear case of selection).

We observe that in general, for longer series exhibiting monotonic increase (characteristic of strong selection), the distribution of the increments quickly veers into the non-normal (as indicated by the Shapiro-Wilk p-value; other normality tests behave similarly; see also the Supplementary appendix). Time series composed of random values drawn from a uniform or normal distribution (or log-normal with small σ) — i.e., the kind of series that should exhibit no selection — tend to have increments distributed approximately normally, as long as the series is away from the boundary values. However, the increments of S-shaped curves tend towards a bimodal distribution. Increment distributions of are severely skewed when a series is shaped like an S-curve but with a sharp “bend”, a straight line (linear increase or decrease), and when a series include long periods of no change.

The assumption of normality could of course be relaxed. However, we observe that this would lead to at least one additional issue, in the form of false positives stemming from the sensitivity of the FIT to small near-boundary changes, illustrated by 2.e.2. Given a long enough series of random values (here sampled from a normal distribution) with a near-zero mean and small standard deviation, the FIT often yields a small p-value (the same applies to samples from the uniform and log-normal distributions; this effect is not observed when the mean is away from the boundaries). Such series would however usefully get flagged as having non-normal increment distributions.

This is also likely why the otherwise flat-lining series for tell in Newberry et al. ends up being included in the discussion as a possible case of selection (at FIT p = 0.12, with a red flag of Shapiro-Wilk p = 0.001). Among the 12 bins of its series (under the original variable-width quantile binning procedure), it has only a few once-per-bin occurrences of regular telled after the initial three bins — a total of 4 singleton occurrences spread out over the span of a century. The +1 absorption adjustment forces the zeroes for telled in the rest of the bins to be ones as well. The observed fluctuations (and resulting FIT p-value) in the series only reflect the slightly fluctuating token frequency of tell, which ranges between 9189 and 11940 in the variable-width bins. Keeping the relative frequency value constant after the third bin instead (at the value equal to the third bin to avoid bias) would result in a FIT p = 0.21.

These last four usages of the regular past form telled in COHA all occur in the fiction part of the corpus, all appearing to reflect the intention of the author to convey a particular kind of character (not used randomly as per a drift model). This would be an example of how an archaic variant can re-surface — quite possible in a language with a long written record, where speakers need not necessarily even directly “inherit” a variant from the previous generation. In that case, telled could be said to have been selected for, due to having increased fitness in a specific (stylistic) niche, and its usage is not due to random variation in the utterances of the speakers (or drift). However, as shown above, this possible (occasional) selection is not what the FIT is picking up on in this case, but rather simply the fluctuating frequency of tell.

Meaning change can also give rise to apparent re-emergence of variants. The occurrence of a form does not guarantee that it is being used in the same meaning or function that it had in another period or context (an implicit assumption in Newberry et al.). For example, the aforementioned spill in COHA quickly converges to the regular past tense spilled, but occasional usages of the irregular spilt still occur, yielding what appears to be a randomly fluctuating time series. On closer inspection, the latter appear to be mostly adjectival usages, not actual past tense verbs, and often turn up in the lexicalized (or “fossilized”) phrase of cry over spilt milk. Examples like that of the time series of telled and spilt, or the series in Figure 2.a.2 and e.2. may possibly be seen as edge cases from the perspective of population genetics — the original domain of the Frequency Increment Test and related approaches. However, as highlighted here, they are examples of not particularly uncommon processes (lexicalization, stylistic usage of unusual variants) in the domain of language.

Finally, one might argue the examples in Figure 2 are not really counterexamples to the utility of the FIT, being representative of cases where the FIT is, strictly speaking, not designed to apply in the first place, such as series with not-quite-normal increments, long flat segments, and values near the boundaries. Excluding these however would mean excluding a fair share of language change scenarios easily observable in corpora, such as changes starting at zero as in cases of linguistic innovations, ongoing changes stretching beyond the bounds of a corpus, and many S-curves typical of language change (and series in general where the underlying selection coefficient is likely not constant). Yet dismissing these as invalid points of concern would also mean dismissing the FIT as a broadly applicable test of selection for the domain of language change.

In the next section, we turn to simulations to explore the behaviour of the FIT beyond that of a few specific series (Section 4), before finally trying to reconcile these conflicting viewpoints (Section 5).

4 The effect of binning frequency data for time series: A simulated example

Here we attempt to further explore the “parameter space” of applying the FIT to simulated data with known properties of selection strength and binning. (code to replicate these results: see the Data availability section in the end). We use the Wright-Fisher model (Ewens 2004) to simulate a large number of time series using the following parameters: population size N = 1000 (N here does not refer to the “population” of speakers, but is analogous to the sum of parallel variants in a corpus bin, e.g. the sum of the counts of lit and lighted in a given year); selection coefficients s in [0, 5]; 200 generations (the latter emulating COHA, where the minimal time resolution is 1 year, and there is 200 years of data). The update rule for this model is as follows. Given n_t “mutants” (e.g., regular past tense forms) in generation t, each individual in the next generation is a mutant individual with probability

$q=\frac{n_t(1+s)}{n_t(1+s)+(N-n_t)}$ 3

Otherwise, it is the wild type (e.g., irregular past tense forms). Where s = 0 we have random drift; higher values of s given an increasingly strong selective advantage to the mutant variant.

Each series (200 data points) is binned into a decreasing number of bins (i.e., [200, 4], of length [1, 50]), and the FIT is applied to every binned version. The simulation for each combination of selection strength and bin length is replicated 1000 times. In summary, in this section we vary the selection strength s and binning, while keeping N and the number of generations constant.

Importantly, we also apply binning to the series post-simulation the same way one would apply binning to corpus counts, as discussed above. The obvious difference from corpus-based time series is that the latter usually do not come from a population with a stable size (total lexeme frequency usually varies in addition to variation in its variants), and are often not continuous (gaps where a lexeme might be completely absent). Since our artificial series do not suffer from these problems, variable-width and fixed-length binning yield identical results, and we can simply use the latter.

We explore two scenarios, where the competing “mutant” variant starts out at 50% of the population and where it starts out at 5%. The former is useful for exploring the effects of binning at low s and false positive rates, the latter for exploring high s and false negatives. Obviously, any specific s thresholds and ranges discussed in this section apply to this specific experiment and would likely be somewhat different given series of different length and N (cf. the Supplementary Appendix for some further exploration).

4.1 Drift and low selection

Figure 3 depicts how the results of the FIT change depending on binning, given a time series with low selection (s = 0.01, bottom row) and no selection (s = 0, top row; corresponds to the leftmost column of pixels on the panels in Figure 4). At zero selection, the FIT has a reasonable false positive rate of around 5% at α = 0.05. Binning such series into a smaller number of bins causes an increase in the share of p-values below 0.05 (presumably because noise is smoothed out). Binning appears to affect the s = 0.01 range even more (bottom row).

Figure 3

The distribution of FIT p-values given 1000 series from the Wright-Fisher model (200 generations, starting at 50%). The panels are arranged from left to right reflecting increased binning. The small inset panels display how binning affects a single example series. p-values below 0.05 are coloured red (left of the dashed line), above 0.05 in blue. Note the log₁₀ x-axis. This figure illustrates that the false positive rate is susceptible to increasing when the series are binned (top row). At non-zero but low s, differences between binning and no binning can be more pronounced (bottom row). See Figure 4 for the full exploration of the parameter space.

Figure 4

FIT p-values of time series generated using the Wright-Fisher model (with the “mutant” variant starting at 50%), across a range of selection coefficients (x-axis, note the log scale), binned into a decreasing number of bins (y-axis). Left in pink and green (a): % of time series with FIT p < 0.05, in 1000 replicates. Right in red and blue (b): mean FIT p-value. The bottom pair (a.2, b.2): the same data, but series with a Shapiro-Wilk p < 0.1 have been removed before calculating the percentages and means. The white rectangle: the range of s and binning explored in Newberry et al. The vertical black line highlights the s explored in Figure 3. A consistent colour across a column of pixels indicates robustness to binning choices under the corresponding s, while variable colouring indicates sensitivity to binning.

Figure 4 represents the entire parameter space explored in this experiment for the 50% start condition. Each pixel on the heat maps corresponds to a parameter combination of selection strength (horizontal axis) and number of bins (vertical axis). The vertical axis starts with 200 or no binning, corresponding to bin length 1 — and running up to 4 bins, with bin length 50, being the result of 200 data points squeezed into the 4 bins. Minimal binning — compressing 200 generations into 100 bins of length 2 — appears to make the clearest immediate difference: the share of p < 0.05 is consistently about 10% higher between the binned and non-binned series when s is low (observe the bottom two “shifted” looking pixel rows in Figure 4.a.2).

The 50% start is suitable for exploring low selection, as in the case of lower starting values, many such series hit absorption or “run into the ground”, and the resulting mostly-zero series would violate the normality assumption (of its underlying Gaussian approximation of the diffusion process). However, the higher s range in Figure 4.a.2 could be interpreted as a model of the situation where a change is only partially chronicled by a corpus, e.g. Figure 2.a.2 in Section 3. Selection becomes understandably difficult to detect in very short series regardless of the underlying selection coefficient.

4.2 High selection

The 5% start is suitable for exploring high selection, as with higher starting values, many high-selection time series reach absorption fast, yielding series not meeting the increment normality assumption. Figure 5 depicts distributions of FIT p-values under different binnings, given time series with a moderately high s of 0.04, and the incoming variant starting out at 5%. This appears to be the subset of series where the FIT works very well and is most insensitive to binning choices.

Figure 5

The distribution of FIT p-values given 1000 Wright-Fisher series with strong selection (200 generations, starting at 5%). The panels are arranged from left to right reflecting increased binning. The small inset panels display how binning affects a single example series. p-values below 0.05 are coloured red (left of the dashed line), above 0.05 in blue. Note the log₁₀ x-axis. The red value in the bottom left corner shows the percentage of p-values below 0.05. This figure illustrates the s range where the FIT is most robust to binning, retaining a small and stable false negative rate (i.e. the inverse of the percentage value in the corner).

Beyond that, things become more complicated. Our reanalysis of the 36 verb time series in Section 2 indicated that it is series exhibiting the strongest selection that would remain consistent in terms of their FIT result across the different binnings. However, as illustrated in Figure 6, it seems too high selection can have the inverse effect, as this is where false negatives begin to crop up under too much binning (e.g. with 10 bins, >10% at s = 0.07, >90% at s = 0.1). That is, if the increment normality assumption is being be strictly observed — if it is, then the results of the test are not valid any more at this range (cf. white area in Figure 6.a.2). This illustrates that the FIT has a maximum selection strength for which it is effective. At higher selection strengths, i.e. above 0.06.0.1 in our toy model, sensitivity to binning and violations of the normality assumption both become problematic, yielding results with a high false negative rate (if the assumption is relaxed; cf. Figure 6.a.1) or results which are invalid (if it is observed; 6.a.2). Incidentally, this also is the s range where S-curves characteristic of language change begin to form (cf. the Supplementary appendix).

Figure 6

FIT p-values of time series generated using the Wright-Fisher model (with the “mutant” variant starting at 5%), across a range of selection coefficients (x-axis, note the log scale), binned into a decreasing number of bins (y-axis). Left in pink and green (a): % of time series with FIT p < 0.05, in 1000 replicates. Right in red and blue (b): mean FIT p-value. The bottom pair (a.2, b.2): the same data, but series with a Shapiro-Wilk p < 0.1 have been removed before calculating the percentages and means. The white rectangle: the range of s and binning explored in Newberry et al. The vertical black line highlights the s explored in Figure 5. A consistent colour across a column of pixels indicates robustness to binning choices under the corresponding s, while variable colouring indicates sensitivity to binning.

In summary, these results indicate that if one is to take the same ensemble of language changes, with known selection strength, and apply different binning protocols, one could easily end up drawing very different conclusions depending on the bin length and the normality assumption threshold, if the conclusions are based solely on applying a test such as the FIT. However, if awareness of these limits is maintained, then the FIT works well on time series with moderately strong selection, and reasonably well (with the caveat of somewhat increased false positives rate under binning) on time series generated by a zero or low selection coefficient.

5 Discussion

We started out by focussing on the study of the (ir)regularisation of the past tense of 36 English verbs in Newberry et al. specifically their finding that drift cannot be rejected in most cases, leading to the claim of the “an underappreciated role for stochasticity in language evolution” (Newberry et al. 2017: 223). The conclusion of our reanalysis section — that their broad conclusion stands but that the FIT is sensitive in specific instances to the chosen binning strategy — prompted further investigation of the properties and range of potential applicability of the FIT. In the following sections, we demonstrated that the FIT yields reasonable results in a certain subset of possible time series, yet perhaps less expected results in others, when applied to a variety of series with different lengths, shapes and underlying selection coefficients.

The fundamental issue is that corpus data has to be operationalised one way or another if one is to apply a time series analysis that is based on variant frequencies. There is as yet no single best method to do so, and the additional researcher degree of freedom is practically unavoidable. Also, unlike microbial experimental data — for which the FIT was designed originally — the beginning and end of a corpus in terms of temporal coverage may not necessarily overlap with the beginning and end of a language change trajectory. The implications of these scenarios on the FIT approach were explored in Figures 2 and 4. Any test based on increment signatures is likely to miss a significant change, if it is recorded by very few data points. This could be either due to data sparsity or low number of bins, very high underlying selection, or the change happening in the middle of an otherwise long series. This could be remedied to an extent by only considering the bins of a corpus or the segments a time series where a change “looks like” it is taking place — but that introduces yet another parameter or researcher degree of freedom.

In what follows, we attempt to summarize our findings and distil them into actionable guidelines for applying tests of selection to linguistic corpus-derived time series.

6 Conclusions

We find ourselves witnessing an exciting time for linguistic research, where more and more data on actual language usage is becoming available, encompassing different languages, dialects, registers, modalities, but also centuries. At the same time computational means for analysing big data have become readily accessible, hand in hand with the development of methods providing new insight into how languages function, change and evolve over time. Alongside and perhaps interlinked with these developments, language as a domain of scientific investigation has attracted interest in recent decades from fields traditionally not engaged in linguistic research, such as physics and biology.

We evaluated the proposal of Newberry et al. (2017), consisting of the application of the Frequency Increment Test as a method for determining whether any time series constructed from corpus frequencies of competing variants is a case of selection or a case of change stemming from stochastic drift. We found that while some of the original results remain robust to binning choices, other do not. Based on constructed and simulated examples, we find that while the results of the FIT can be robust given a subset of suitable series, there are scenarios where they affected by the way the diachronic corpus data are binned.

We advocate that in the interest of reproducibility, binning, like any other data manipulation and operationalisation procedures, should be explicitly described in a contribution (as it is by Newberry et al.) — but additionally, if the results change given different choices, this should also be reported. Beyond data operationalisation, we drew attention to issues specific to linguistic data that should be taken into account to ensure quality of testing results, as well as to work in cultural evolution where it has been shown that the inference of individual transmission processes from population-level frequency aggregates is susceptible to error and should be handled with care.

To conclude, identifying the role and prevalence of stochastic drift in language change is an important goal, but our results suggest that great care should be exercised when applying such tests to linguistic data, in order for the results to not be biased by issues specific to the domain as well as properties of a particular test.

Data Accessibility Statement

The R code we used to replicate the results of the original paper is available at https://github.com/mnewberry/ldrift, and the corpus at https://corpus.byu.edu/coha. The code to run the simulations described is this paper is available at https://github.com/andreskarjus/wfsim_fit.

Funding Information

The first author of this research was supported by the scholarship program Kristjan Jaak, funded and managed by the Archimedes Foundation in collaboration with the Ministry of Education and Research of Estonia.

Competing Interests

The authors have no competing interests to declare.

Author Contributions

Andres Karjus carried out the research and wrote the paper. Richard A. Blythe, Simon Kirby and Kenny Smith provided revisions, comments and feedback on the design of the research and the paper.