We first summarize the main results of this work, which concern the behavior of the STC phase under serial invasion of new strains. A parent strain in the community occasionally gives rise to a mutant strain whose properties are correlated with those of its parent with correlations parametrized by ρ∈[0,1). Many of the behaviors are similar across this range, from independent invaders with ρ=0 to small-effect mutations with ρ close to 1. In all cases, extinctions are irreversible. Key properties of interest are the number of extant strains in the ecosystem, L, the number of successful invasions Z, and how these evolve with evolutionary time, parametrized by the number of attempted invasions, T.

We find that for a wide range of ρ (and expect for any ρ<1) the STC can enter a steadily diversifying state wherein the number of successful invasions, Z, and the number of coexisting strains, L, both increase linearly with T on average, with only small fluctuations when L is large. Whether diversification occurs, and the rate of the diversification if it does, depends on various parameters, but it is robust over a range of the parameters. If initially the ecosystem has only a modest number of strains, the evolutionary dynamics tend to cause the diversity to crash, after which is it extremely unlikely to transition into the diversifying phase. However, if the initial ecosystem is sufficiently diverse, it is highly likely to diversify further.

We then study the effects of general fitness differences that augment the average growth rate of a strain by an intrinsic amount irrespective of its interactions with the other strains. Focusing on unrelated invaders, we show that a distribution of such general fitness differences (denoted by si for strain i) can either slow down, prevent, or reverse diversification. For distributions of the si whose tails decay faster than exponentially, the diversifying phase still exists, but with the diversification rate gradually slowing down: L increases only as a power of log⁡(T). If the distribution of the si has a broader-than-exponential tail, the diversity decreases and crashes.

The key property of a strain, in terms of which one can understand its behavior, is its bias: defined as the rate at which its population would change when at low-abundance and without migration (Figure 1B). The crucial effect of migration in the STC is to stabilize many strains with negative bias which would have gone extinct without migration. Only if its bias is strongly negative will a strain go globally extinct. The bias of strain i, denoted by ξi, has an intrinsic contribution from its general fitness si, and an extrinsic contribution from the interactions of strain i with all other strains, which includes both an i-dependent part determined by the interactions with the other strains in the community, and an i-independent part that keeps the total population constant. The extrinsic part of the bias of each strain changes as the community evolves, but its intrinsic part says the same.

Building upon the theoretical understanding of the STC phase, developed in PAF, we first analyze evolution in the simplest case of unrelated invaders (ρ=0) with no general fitnesses (si=0). The bias of each strain undergoes a random walk on evolutionary timescales, and we find that for large communities, the number of strains changes at a steady rate. For a range of parameters, this diversification rate is positive, yielding a steadily diversifying phase with the distribution of biases scaling with 1/L, as observed in numerics. We then extend our analysis of the changing bias distribution to include the effects of general fitness differences. This yields predictions of how the rate of diversity increase (or decrease) depends on the distribution of the si, corroborating the behaviors found in simulations.

The distributions of biases and abundances in evolved communities differ subtly from those of the initial communities that were assembled all-at-once from unrelated strains. At early stages of the evolution, most of the close-to-marginal, low-abundance strains are pushed out by the perturbations caused by the invading strains. This extinction process causes the shape of the abundance distributions of assembled and evolved communities to differ at low abundances. Later, in the steadily diversifying state, the numbers of extinctions caused by each invader has a roughly exponential distribution, which is consistent with our theoretical expectations. In contrast to the qualitative (albeit modest) changes in abundance distributions, we find that evolution has only a small effect on the statistics of the interactions between strains.

The structure of this paper is as follows: ‘Models’ introduces the main model and its relation to previous work. ‘Results’ describes the phenomenology of an evolving community in the STC phase, studying the effects of correlated mutants, interaction statistics, and general fitness differences on the ecological diversification. Then ‘Analysis’ develops the theory and analysis that are needed to understand these phenomena. Building upon the dynamical mean field theory developed in PAF, we present an approximate framework, and more general scaling arguments, for understanding the evolutionary dynamics, and compare the predictions with simulations. Finally ‘Discussion’ raises additional questions and discusses possible extensions. Many of the details and further analyses are relegated to appendices.

We first consider an assembled community of K unrelated strains, labelled by i=1,2,…,K, with all possible pairwise interactions between them. A paradigmatic model for the ecological dynamics of the strain populations {ni} is the generalized Lotka-Volterra model (Goel et al., 1971), with each strain i having an intrinsic growth rate which is modulated by its interactions with all the other strains. These interactions are conveniently represented in a matrix W where Wi⁢j describes the effect of strain j on the growth rate of strain i. Since we are interested in closely related strains for which all interactions are similar, the total population will be roughly fixed at some N by the balance between the effects of positive intrinsic growth rate and negative competitive interactions. It is convenient to replace these large terms by a Lagrange multiplier Υ⁢(t) that fixes the total population to ∑ini=N, and work with fractional abundances, νi≡ni/N. This parameterization yields what are known as “replicator equations” (Chawanya and Tokita, 2002; Yoshino et al., 2008; Tokita, 2004).

Variations in intrinsic growth rates and net interactions on a strain can be combined to yield general fitness differences, {si}, between the strains. We parameterize the residual variations in interactions among the strains (after subtracting off Υ and si) by Vi⁢j. Since the Vi⁢j and si are sums and differences of similar magnitude terms, it is natural to approximate them as random variables with the hope that the model will yield behaviors that are robust to specific choices of their statistics: testing this assumption is one of the goals of this paper. For simplicity, we choose E⁡[Vi⁢j]=0, and E⁡[Vi⁢j2]=1 for i≠j — setting the overall ecological timescales — and choose the covariances to be zero except for, importantly, correlations between how i and j affect each other, defining E⁡[Vi⁢j⁢Vj⁢i]=γ. For convenience, we choose E⁡[Vi⁢i2]=1+γ but this choice has negligible effect in large communities.

The parameter γ controls whether the interactions are mainly competitive (γ>0) or host-pathogen-like (γ<0), the latter being the focus of this work. We have shown in PAF that random interaction matrices with such anticorrelations behave very similarly to host-pathogen models with the appropriate block sub-matrix structure, as discussed further in ‘Bacteria-phage interactions and coevolution’.

We study the simplest model with spatial structure: a large number, I, of identical islands (or demes) with interactions only within each island and migration between all pairs of islands. With Greek indices labeling islands, the dynamics of the abundances obey(1)dνi,αdt=νi,α(si+∑j=1KVijνj,α−Υα(t))+∑β=1I(mαβνi,β−mβανi,α).

with mα⁢β the migration rate (per individual) from island β to island α and the local Lagrange multiplier, Υα=∑iνi,α⁢(si+∑jVi⁢j⁢νj,α), keeping the total population on each island fixed at N, (i.e. ∑iνi,α=1 for each island). Here, we focus on the spatial mean field limit in which the migration rate is the same, given by m/I, between every pair of islands. The total migration of strain i into and out of island α is then simply m⁢(ν¯i-νi,α) with ν¯i⁢(t) the average of νi⁢(t) across islands. As the number of islands becomes large, in steady state, each ν¯i⁢(t) becomes constant in time — some being zero corresponding to global extinction. In the STC, the dynamics are asynchronous across islands and ergodicity implies the spatial average, ν¯i, is equal to the time-averaged abundance of strain i on a single island; this is a crucial self-consistency condition. The magnitude, m of the migration rate, is also of fundamental importance. If m is too small the migration is too rare to repopulate islands after local extinctions. If m is too large and the local dynamics is chaotic, the chaos will synchronize across the islands and the total population of each strain will fluctuate wildly, rapidly driving most strains extinct. We will focus on the wide intermediate m regime, which spans several orders of magnitude when K is large [PAF].

With large populations on each island, demographic fluctuations have little effect on the dynamics. Even when the local population of a strain is small, if it has positive growth rate, fluctuations will not matter much, while if it has negative growth rate it will go deterministically extinct which occurs when the fractional abundance drops below the extinction threshold of 1/N indicated in Figure 1A by the horizontal purple line. The value of the extinction threshold does not much affect the behavior as long as it is much below the lower limit of the abundance caused by migration — which we term the migration floor: νfloor∼mν¯. For strains near local extinction (when the fractional abundance is close to 1/N) demographic fluctuations are potentially important. But with N⁢m⁢ν¯ very large, local extinctions for viable strains will be rare: thus we model the population dynamics as fully deterministic. If the fractional abundance on an island drops below 1/N, it is set equal to zero. Global extinction occurs when a strain’s bias becomes too negative, which results in it going below the extinction threshold everywhere. The choice of N does not matter much as long as m⁢N≫1, to which we restrict consideration. Related details of numerical implementation are discussed in ‘Appendix 2’. In ‘Spatial structure and dynamics’, we comment on the effects of local extinctions in the context of real spatial dynamics.

The key properties of the STC phase [PAF] are chaotic coexistence of strains, desynchronized across islands, with the local abundances fluctuating over a range in log⁡ν of M≡log⁡(1/m), which is quite wide for the typical m=10-5 that we use in simulations. Some strains go globally extinct but each persistent strain on each island occasionally has a bloom up to high abundance ν∼M/L. These localized blooms are crucial for stabilizing a strain, as they dominate the migration to other islands needed to recover from local extinctions or near-extinctions. On a single island, at any given moment, O(L/M) strains are at high abundance. A snapshot of the abundances on a single island shows the strains distributed roughly uniformly in log⁡(ν) down to the migration floor, only occasionally fluctuating substantially lower (Figure 1).

The evolutionary process we model is much slower than the ecological and migratory dynamics. Simulations are divided into long epochs, with new strains added only at the end of an epoch. The epochs are chosen long enough that the ecological and migratory dynamics have reached a steady state, with some fraction of the strains having gone permanently extinct globally, leaving L persistent strains. A single new strain is then introduced and the process repeated.

The new strain, generically labeled A, is parameterized by its interactions with all other strains in the community, given by VA⁢j and Vj⁢A, and its general fitness, sA. In the simplest case, a new strain is unrelated to extant (or extinct) strains. More generally, mutant strains, labelled M, are characterized by their degree of correlation, ρ∈[0,1), with a parent strain P chosen from the existing community with probability proportional to its mean abundance ν¯P. These correlations are realized such that Corr⁡[Vj⁢M,Vj⁢P]=Corr⁡[VM⁢j,VP⁢j]=ρ for j≠M,P. The detailed choices for j=M,P are given in ‘Appendix 2’. The general fitness, sM, can also be correlated with sP. Unrelated invaders are equivalent to ρ=0 and hence have no parent.

The actual process of invasion from low abundance on one island is complicated, and often leads to failure. To avoid a proliferation of such failed invasions, we instead assess whether the invader could successfully invade and persist if it were lucky initially. To do this, we set the mutant’s abundance to 1/L on all the islands at the same time (and proportionately decrease the abundances of the other strains to maintain ∑iνi,α=1).

There are multiple timescales involved in the dynamics: these are discussed more fully in ‘Appendix 1’. The basic timescale for differential growth or decay of strains is set by the magnitude of the interactions and the number, L, of extant strains. The extant strains have average abundances of order 1/L, so the average total interaction on a strain, i, is the sum of L random terms, each of order V/L for typical interaction strength V. With the V having variance unity, the average net interactions of other strains on strain i is roughly its mean drive, ζi, which is of order 1/L, implying that the timescale for systematic population growth or decay is of order L. The mean drive is defined more precisely in Results. When there are general fitness differences, si, these also contribute to variations in average growth rates. The variations in the si within the community have substantial effects over a time ∼1/σs with σs roughly the width of the si distribution of the extant strains. Together, the mean drive and general fitness of a strain determine its crucial property: the bias ξi=ζi+si-⟨Υ⟩, with angular brackets denoting a time average. As introduced earlier, the bias of strain i is its average growth rate at low abundance in the absence of migration (Figure 1B). As we shall see, the size of the community is limited by the condition that the inter-strain variation in general fitness is no larger than variation in average drive from interactions. This means that the biases are of order 1/L.

The local population of each strain undergoes wild fluctuations over a logarithmic range M=log⁡(1/m) which is quite large. During blooms, the instantaneous growth and decay rates of local populations are substantially larger than the systematic biases (‘Appendix 1’) and change rapidly from growth to decay as seen in Figure 1. The time for abundances to fluctuate from large to small — the duration of blooms — is of order M⁢L with systematic and fluctuation contributions comparable.

An important timescale for studying slow evolution is the time to reach the STC steady state: the ecological relaxation time. This is determined by the strains that are just barely going extinct and is of order M⁢L for an evolved community, as discussed in ‘Continual assembly and diversification’. We have chosen the evolutionary timescale to be much longer than all the other important timescales. Thus, each epoch between the addition of invaders is chosen to be several times the ecological relaxation time, typically 3⁢M⁢L, and we show in ‘Appendix 2’ that increasing this epoch length by a factor of 10 makes little difference in the diversification dynamics.

The evolutionary process we study starts from an assembled collection of K unrelated strains. After the ecological and migratory dynamics have reached steady state, some of the strains will persist: we call the size of this initial persistent community L0. The K-L0 strains that have gone globally extinct are permanently removed.

When a new strain is introduced into the ecosystem, if it successfully invades it perturbs the biases of the extant strains, and can trigger extinctions of some of them by shifting their bias below ξc (Figure 1A) by an amount of order 1/L. We study the slowly evolving regime in which the ecosystem dynamics reach steady state between each introduction of a new strain — this takes time of order M⁢L. The number of persistent strains and number of successful invasions as a function of the number of attempted invasions, L⁢(T) and Z⁢(T) respectively, are of fundamental interest.

We first describe the evolutionary dynamics when the general fitness differences between the strains can be neglected. For γ=-0.8 and unrelated invaders (ρ=0), multiple simulation runs starting with different sets of K=50 initial strains reveal that around one fifth of the replicates enter a steadily diversifying regime in which L increases roughly linearly with the number of attempted invasions, at a rate of around 0.25 per attempt. The remaining replicates crash down to only a few persistent strains. Subsequent invasions can cause L⁢(T) to increase somewhat, but it quickly crashes back down and the community does not steadily diversify (Figure 2A). The low diversity regime that occurs after a crash (or with a very small initial community) is discussed further in ‘Appendix 3’.

The observations in Figure 2 illustrate one of the crucial findings of this work: spatiotemporally chaotic ecological dynamics can allow — but do not guarantee — gradual strain-level diversification up to arbitrarily high number of strains. The behavior depends on the symmetry parameter γ, which must be substantially negative for the STC to exist. Figure 2B shows that the average rate of diversification, U≡⟨d⁢L/d⁢T⟩, is nonmonotonic in γ, with slow diversification close to γ=-1 (its lower limit). As γ becomes less negative the rate of diversification increases at first. However for γ even less negative, the STC still supports chaotic coexistence of many strains (since L0 is still large), but the diversity decreases under evolutionary dynamics. The community diversifies most rapidly for γ≈-0.8. As we are interested in what can happen with various other additional features, we chose γ=-0.8 and m=10-5 for all further simulations as shorter runs are needed near these values. We expect that the qualitative conclusions will be similar for a range of γ and m around these.

In addition to studying independent invasions, we study evolution via mutations of existing strains. At the start of each epoch, a parent to mutate is chosen with probability proportional to its mean abundance. The interactions of the mutant with other strains are drawn from the same marginal distribution as the original interactions, but with correlation ρ with the interactions of the parent (Evolutionary dynamics). The direct interactions between the parent and mutant have to be chosen separately as specified in ‘Appendix 2’ but, as they only account for a small fraction of the total abundance in diverse communities, the specific choice is not important. As a function of ρ (with γ=-0.8), the rate of diversification is nonmonotonic being fastest for ρ≈0.8, and only weakly varying for smaller ρ (Figure 2C). As ρ nears 1, the mutant and parent are more similar, and it becomes harder for them to coexist, since any difference between them is likely to result in a systematic change in their relative abundance, eventually driving one of them to extinction (see ‘Appendix 9’). Since L can only increase when both the mutant and parent coexist, increasing ρ slows the rate of diversification but L⁢(T) still increases linearly.

This observation implies that, for a large range of ρ, despite not enforcing any precise constraints or perfect tradeoffs, strains that would outcompete all extant members of the community are too rare to emerge and reduce diversity. Therefore, we conjecture that a continually diversifying phase exists even for ρ arbitrarily close to 1.

So far we have observed that when mutants or invaders differ only by their interactions with each other, there is robust and rapid diversification, provided that the initial diversity in the STC phase is high enough. Now we include general fitness differences si between strains and show how these affect the evolutionary dynamics.

The DMFT approximation, which is exact in the limit of a large number of strains with random interactions between them, replaces the full statistical dynamics by the stochastic effects of the others on one chosen strain, with the statistical properties then determined self-consistently from the properties of the distributions over the strains. This approach was first used in the physics of disordered systems such as spin glasses (Sompolinsky and Zippelius, 1982), but has been applied to ecological dynamics in a number of subsequent works (Diederich and Opper, 1989; Opper and Diederich, 1992; Galla, 2006; Roy et al., 2019; Rieger, 1989; Yoshino et al., 2008).

When strain i has very low abundance, its effects on the others are very small and the forces of the others on it, ∑jVi⁢j⁢νj, are comprised of roughly independent random variables and thus act like gaussian noise with correlations C⁢(t,t′)=∑jνj⁢(t)⁢νj⁢(t′). However, when it rises to substantial abundance, it will weakly affect the other strains. Because of the correlation γ between Vi⁢j and Vj⁢i, these feedback terms add coherently, resulting in a contribution to growth rate of i of form γ∫0tR(t,t′)νi(t′)dt′, with R a response function determined by feedback from the total impact on the community of the strain’s own past history (see ‘Appendix 5’). The DMFT allows one to recast the generalized Lotka Volterra equations as an effective single-strain problem, with self consistency conditions on the bias correlations and response function.

With the dynamical mean field understanding of an assembled STC phase in hand, we can proceed to describe the evolutionary process in terms of the distributions of properties of the extant and newly invading strains — in particular their biases and consequent mean abundances. The distribution of biases is perturbed by the introduction of new strains and this can push some of the extant biases below ξc, which itself depends on the bias distribution as modified by prior evolution, and on the number of extant strains, L. It is convenient to define an effective community size L=1/∑iν¯i2 which controls the variances of the mean-drive part of the bias; L scales with the actual L but discounts strains that are close to extinction.

We first analyze invasion of unrelated strains without general fitness differences. The bias of an attempted invader, labelled A, is given by ξA=∑jVA⁢j⁢ν¯j-Υ¯, in terms of its interactions, {VA⁢j} with the extant strains: for unrelated invaders, this is gaussian distributed with mean -Υ¯ and standard deviation of 1/L independent of correlations among the extant strains, though correlations in the existing community will affect the N(ξ) and hence L.

Strain A can successfully invade the community and persist if ξA>ξc. The probability of successful invasion is thus Φ⁢[(ξc+Υ¯)⁢L], with Φ the standard normal cumulative distribution function. In the initial assembled community, ξc⁢L and Υ¯⁢L are independent of L. We make the Ansatz that after a long period of evolution the distribution of extant biases, scaled by L, reaches a steady state — albeit a different state than the initial assembled community. Then the probability of successful invasion will become independent of L for large L. If the mean number of extinctions per successful invasion also reaches a steady state value which is less than unity, this explains the steady linear growth of L⁢(T) seen in Figure 2.

With the one-by-one introduction of new strains, the bias of each extant strain undergoes some kind of random walk, and the strain goes extinct if its bias ventures below ξc. In Figure 4A, we show the evolutionary trajectories of the biases of 5 individual strains that started from similar initial values in a simulation where the community diversified from 50 to 500 strains. Extinctions are caused by ξ being pushed below ξc by an invading strain. For finite L, the sharpness of ξc will be smeared by an amount of order 1/L due to variability in the dynamic noise from strain to strain, which we have not explored.

To numerically investigate any systematic components of the random walk of biases, we average over a large number of strains, binning them according to their initial values normalized by 1/L. We observe a strong tendency of anomalously positive and negative biases to regress toward an intermediate value. In this plot, as evolution proceeds, the asymptotic average bias conditioned on survival is larger than -Υ¯⁢L. (Figure 4B). This is likely due to conditioning on survival of the strains: those that persist for many epochs tend to have larger-than-average (but still negative) bias.

In ‘Appendix 8’, we carry out an analysis of the bias dynamics by approximating these by a Markov process in which the dynamics of the biases depend only on their current values. This analysis shows that when there is a successful invasion, each strain’s drive undergoes both a systematic and random change, consistent with our numerical results.

It is convenient to work with the mean drives, ν¯i (when si=0, this is just ξi+Υ). Both the systematic and the random changes in the drive are proportional to the average abundance of the invading strain, which is of order 1/L. The stochastic change is δ⁢ζi∼±1/L, but the systematic change in the drive is smaller and depends on its current value: E⁡[δζi|ζi]∼ζi/L=O(1/L3/2). In the Markovian approximation, there is a simple Langevin equation for the change of the drive of a strain due to invasions:(3)dζidT=−BζiL(T)+2D1L(T)ηi(T),

with ηi⁢(T) approximately gaussian with mean zero and unit variance — an approximation that should be good if one coarse-grains over a substantial range of T (but with range much smaller than L). From our analysis in ‘Appendix 8’, we see that B and D are order-unity coefficients which respectively characterize the average and mean-squared response of the bias to the invasion of a new strain. Both are proportional to E⁡[νA2] times the fragility, Ξ, of the extant community which is given by Ξ=∑jχj2/(1-∑jχj2) in terms of the individual susceptibilities of strains to changes in their biases, χi=d⁢ν¯i/d⁢ξi. This fragility characterizes the mean-square response of the system to a random perturbation applied simultaneously on all the strains — precisely the effect of a successful invasion. The Langevin equation for the drives must be supplemented by a boundary condition that if ξ=ζ-Υ goes below ξc, the strain disappears.

Analysis of the Langevin equation, which can be converted into a Fokker Planck equation for the distribution of the drives (‘Appendix 8’), shows that there is an eigenvalue-like condition which determines whether the diversification rate of the community is negative or positive, and that the coefficients B and D play a role in determining whether the community diversifies or not. This is consistent with our numerical results, which show that certain parameter regimes allow diversification and other regimes do not — even in the absence of general fitness differerences.

The approximate model of the evolution of the biases makes predictions about the shape of the bias distribution as a function of attempted invasions. Before the onset of evolution, the distribution of biases is a truncated gaussian, with a lower cutoff set by ξc. However as the distribution evolves according to Equation A8.3 with the absorbing boundary condition at the critical bias, it smooths out near this cutoff, going linearly to 0 as ζ→ξc+Υ¯ (or, equivalently, as ξ→ξc).

Simulations confirm the expectation that the typical bias scales as 1/L over one order of magnitude in L (Appendix 3—figure 4). As predicted, one observes a smoothing out of the bias distribution toward ξc when comparing evolved and assembled communities of the same L (Figure 5B), and our analysis allows us to obtain the theory curve for the evolved ecosystem in Figure 5B as the solution of the approximate boundary value problem. However, the critical bias is sufficiently negative that the number of strains affected by the differences between the initial and evolved communities is small and the distinctions hard to see numerically.

However, the density of biases near ξc determines the response of the community to evolutionary perturbations, since these low-bias strains are the ones most susceptible to extinction. In particular, the predicted linearly vanishing density of biases determines the distribution of the number, ℓ, of extinctions per successful invasion (Appendix 3—figure 1B). To estimate this distribution — particular the probability that ℓ is large — we use the fact that an invader will perturb the extant strains' biases by a random amount of order 1/L and proportional to the mean abundance of the invader. The positive tail of the invader’s mean abundance, ν¯A, is gaussian, since for positive ξinv, ν¯inv∼ξinv and the invader’s bias ξinv is itself gaussian distributed. The number of strains whose biases are within ν¯A of ξc is proportional to L2⁢ν¯A2, because the distribution of strains' biases vanishes linearly at ξc. Thus for fixed ν¯A, the number of strains that are driven extinct is Poisson distributed with mean proportional to L2⁢ν¯A2: this is of order one for large L as expected. That the tail of the distribution of ν¯A is gaussian implies ν¯A2 is approximately exponentially distributed in its tail. Integrating the Poisson distribution over this yields, for large ℓ, P(ℓ)∼e−βℓ/ℓ (with β an order-unity coefficient) which is close to exponential as observed in Appendix 3—figure 1B.

A similar analysis for the initial randomly-assembled community shows that for fixed ν¯A, the mean number of extinctions triggered by the first successful invasion is of order L03/2⁢ν¯A∼L0; much larger than after evolution has proceeded for a while. As the Poisson with this mean has a narrow distribution, the probability of an anomalously large number of extinctions will be dominated by the gaussian tail of the distribution of ν¯A and hence itself be roughly gaussian, though unless the initial L0 is huge, the tail is unlikely to still be in the asymptotic regime. The transient caused by a set of early invasions will likely cause a total of order L0 strains — with a relatively small coefficient — to go extinct before L starts steadily increasing, and this will occur over of order L0 invasions. For γ=-0.8 this effect appears to be very small — the critical bias is quite negative — but for smaller or larger γ the effects are noticeable (Figure 2).

The distribution of mean abundances, ν¯i, is related to that of the biases via the function N(ξ): therefore, we expect this also to evolve as the community diversifies. In particular, there should be a reduction in the number of strains at low mean abundance, since these correspond to those with close-to-marginal bias. In Figure 5A, we see that the mean abundances in an evolved community are more narrowly distributed than in an assembled community, with both fewer highly abundant and fewer rare strains. This is consistent with our picture of the bias distribution being smoothed out toward ξc due to invasion-triggered extinctions, resulting in the depletion of low-abundance strains. The depletion of abundant strains is likely due to the kill-the-winner dynamics which rewards invading strains that push the most abundant extant strains down.

Although the mean abundances are not broadly distributed on a log scale, the snapshot abundance distributions are, as seen in Figure 1. Note that most widely-used measures of diversity are not really informative for these kinds of logarithmically broad distributions. For example, the Shannon entropy would weight mostly the highly abundant strains, while the “species richness” would be highly sensitive to the lower cutoff in observable abundance.

Armed with understanding of the scaling of the bias and mean drives with L, we can build upon the analysis of the simple exponential distribution of general fitness (‘Evolutionary dynamics with general fitness differences’) to analyze the evolution when P(s) decays faster or slower than exponentially. A heuristic understanding of how the dynamics of L depend on P(s) follows from the fact that without general fitness differences, the biases are distributed with characteristic scale 1/L. As L increases, the distribution of these biases gets narrower, and the system becomes progressively more “neutral" with overall differences in strain biases becoming smaller. The contribution of the general fitnesses is to add a random extra piece to each bias, broadening the distribution of extant ξi. In the limit of many invasions, the width of the drive distribution becomes comparable to the width of the distribution of the extant si and cannot decrease further. Thereafter, the shape of P(s) determines both the width of the bias distribution, and the number of coexisting strains.

If L≪1/Σ2 initially, the si play little role and the population-weighted mean fitness, s^, only increases gradually. But once s^ is a few times Σ, the tail of P(s) will determine the rate of increase of s^. Henceforth, s^ will grow steadily with subsequent successful invasions, since strains with si≪s^ are very unlikely to persist, and strains with si≫s^ are unlikely to have yet occurred. Thus, the range of extant si will become much narrower than Σ. This implies that the distribution over the currently relevant range can be approximated by an exponential distribution P(s)∼e−s/Σ^ with the effective width of the extant si distribution given by(4)Σ^=−1d[log⁡P(s)]/ds|s=s^=Σψs^ψ−1 ,

where the second equality is for the specific models we study (Equation 2). Provided evolution has proceeded long enough that no strains with si smaller than the original scale Σ survive, Σ^ will vary slowly for a range of evolutionary time and this sets the scale for variations of si of both extant strains and of potentially-successful invaders. This suggests that understanding the general behavior at long evolutionary times can be built on understanding the case of the simple exponential distribution (ψ=1 for which Σ^=Σ). The main difference is that now the community size will change as L∼1/Σ^2, with Σ^ changing as s^ increases: this will govern how L changes as invasions are attempted and occasionally occur.

In the slow evolution regime at long times, successful invaders must have sA comparable to s^. A simple argument gives an upper bound on how fast s^ can increase with T. In order to get a mean of s^ in a community of L strains, at least L attempted invaders must have occurred with sA≳s^. This requires a number of invasion attempts, T, such that T∫0s^(T)P(s)ds≥L(T). But if s^ were substantially smaller than this upper bound, many strains would have already occurred with si-s^≫Σ^ and these would persist for a long time, driving s^ up. We thus make the Ansatz, justified by the analysis and simulation data of ‘Appendix 6’ and Figure 6, that s^⁢(T) grows with log⁡(T) at a rate asymptotically given by this upper bound. For the distributions of interest we then have(5)TP(s^)∼L(T)⟹s^Σ≈[ψlog⁡(T/L)]1/ψandΣ^Σ=(Σs^)ψ−1⟹L∼Σ−2[log⁡(TΣ2)]2−2/ψ,

where the last implication is due to L∼Σ^-2. These scalings become valid once the distribution of the extant si is pushed into the tail of P(s). To crudely take into account the effect of a large initial number of strains when K≳1/Σ2, log⁡(T) can be replaced by log⁡(K+T), as for the plots of L⁢(T) and s^⁢(T) in Figure 6. Figure 6B shows the theoretical prediction for s^⁢(T) in the simplest case of ψ=1, and we see that s^⁢(T)/Σ is reduced from the log⁡[(T+K)⁢Σ2] prediction by only an O(1) constant, as expected.

At long times, the probability of successful invasion decreases very rapidly with s^ and, as we show in ‘Appendix 6’, the cumulative number of successful invasions for ψ≥1 grows very slowly, with Z∼Σ-2⁢(log⁡T)3-2/ψ. Nevertheless, for ψ>1, the number of strains grows without bound albeit as a sub-linear power of the cumulative number of successful invasions. The average number of extinctions per successful invasion gradually decreases towards one as P(s) in the pertinent range, s≈s^, becomes closer and closer to exponential.

For longer-than-exponential tails, ψ<1, the diversity will decrease (possibly after an initial increase if K≫1/Σ2) and eventually — in practice rather soon — crash as seen in Figure 6A.

While for some microbial populations — for example common human gut commensals — a collection of connected “islands” without much spatial structure may be a rough caricature, for most populations there is spatial structure that makes dispersal from one location to another dependent on the distance in one, two or three dimensions. Thus, instead of having all pairs of islands connected by migration, one could model a d-dimensional array of islands with nearest-neighbor migration; a spatial continuum with diffusive dispersal; or a mixture of long and short distance dispersal events as driven by wind, ocean currents, or hitchhiking on migrant animals (Hallatschek and Fisher, 2014). With real spatial structure, local sub-populations are much more prone to extinction and cannot be as readily rescued by migration from another location where the strain is blooming. Thus, in contrast to the regime we have worked in for this paper, recovery from local extinctions must play a crucial role. The dynamics of invasions, extinctions, and repopulation is very different than in the spatial mean field model: if the underlying dynamics is diffusive, invasion and repopulation will occur by propagating Fisher-Kolmogorov-Petrovsky-Piscunov (FKPP) fronts (Fisher, 1937). The properties of FKPP waves are known to be highly sensitive to dynamics at the wavefront, and the effects of demographic fluctuations have been investigated (Korolev et al., 2010). But the approximately multiplicative “noise” from the ecological interactions will surely change this, and even for a single wave understanding the impact of these larger fluctuations is still an open question (Rocco et al., 2002; Rocco et al., 2000).

With long-range dispersal over a multitude of length scales, the dynamics of invasion, extinction and repopulation will be very different, as already occurs for a single successful invader without ecological variations (Hallatschek and Fisher, 2014). Generally, understanding of the STC phase will have to build on better understanding of repopulation dynamics in the presence of large ecological fluctuations, and then understanding the evolution of communities on top of that. We leave investigations of this for future work. But we conjecture that a continually diversifying STC phase can still occur with more realistic spatial dynamics.

An obvious weakness of the Lotka-Volterra models studied here is that the strains do not carry their own phenotypes, but are characterized by their interaction with all possible other strains. Furthermore, the antisymmetric correlations in the interaction matrix (especially without substantial general fitness differences) are rather unnatural for multiple strains of a single species. Thus, the most interesting extension of this work is to much more natural models: multiple strains of a phage species that prey on multiple strains of a bacterial species, with varying effectiveness that is a function of phenotypic properties of the particular phage and bacterial strains. Of particular importance is the interaction between a phage tail and bacterial receptor, as modelled in Weitz et al., 2005. We showed previously [PAF] that the block-antisymmetrically-correlated structure of the interaction matrix with the bacteria having no niche-structure (differing only in the way they interact with the phages) can give rise to an STC phase that is very similar to that of the antisymmetrically correlated Lotka-Volterra model studied here: a similar model was further explored in Martis, 2022. Such a bacteria-phage model can naturally accommodate general fitness advantages through phenotypic changes, eliminating the need to introduce them on separate footing. In ongoing work, we show that much of the basic phenomenology we have found here also occurs in evolving bacteria-phage phenotype models — at this stage only roughly and qualitatively.

For bacteria phage models, studying phylogenies and relatedness questions are natural. Whether more specialist phages tend to evolve, making the interaction matrix sparser and perhaps more hierarchical — and if so under what circumstances — is a particularly interesting question.

We have studied evolution of communities of many closely related strains in the limit that the evolutionary dynamics is slow compared to ecological and spatial dynamics. For a class of models, and in a particular ecological “phase”, evolution drives continual diversification, provided there is sufficient diversity initially. However mutations that change general fitness of strains tend to strongly slow down or even reverse the diversification. Thus we ask: How ubiquitous is diversification in the absence of any niche-like structure? Are there models in which a diversifying phase is easier to nucleate? Will the diversification always tend to be limited or strongly-slowed by general-fitness mutational effects? Or might “entropic” effects associated with difficulty of finding such general fitness mutations — for example from discrete genomes rather than continuous phenotypic parameters, or from soft tradeoffs — counter this slowdown, or perhaps produce evolutionary dynamics that lead to sparse interaction matrices and broader distributions of biases? Conversely, if strains are initially separated in “niche space” but then start to overlap and interact as the number of strains increases, how does the behavior differ? Is continual diversification easier to nucleate? Are the statistical properties of the phylogenies resulting from this evolutionary process — here driven entirely by “selection” in the broad sense, with ecological interactions creating a balance between the many extant strains — similar to a known class of coalescent trees?

What happens when, as in large microbial populations, evolutionary processes are not slow? Faster evolution is likely to make diversification easier, but understanding this even in simple models will require much better understanding of the invasion probabilities of mutants. Other than our scenario in which spatiotemporal chaos is the key to stabilizing coexisting diversity, what other robust continually diversifying scenarios are there? And of course, most crucially, what observable features of the strain, sub-strain, and sub-sub-strain level diversity in a microbial population (or interacting populations) could provide hints to the underlying causes of extensive diversity?