Given a source of odor placed in an unknown position of a two-dimensional space, we consider the problem of learning to reach the source, Figure 1A. We formulate the problem as a discrete Markov Decision Process by discretizing space in tiles, also called ‘gridworld’ in the reinforcement learning literature (Sutton and Barto, 1998).

In this problem, an agent is in a given state s\begin{document}$s$\end{document} which is one of a discrete set of n\begin{document}$n$\end{document} tiles: s∈S:={s1,...,sn}\begin{document}$s\in S:=\{s_{1},...,s_{n}\}$\end{document}. At each time step, it chooses an action a\begin{document}$a$\end{document} which is a step in any of the coordinate directions a∈\begin{document}$a\in$\end{document} {upwind, downwind, crosswind-left, crosswind-right}. Directions are labeled relative to the mean wind, which is assumed known. In our figures, the flow always goes from left to right, hence the actions upwind, downwind, crosswind-right, and crosswind-left correspond in the figures to a step left, right, up, and down, respectively. The goal is to find sequences of actions that lead to the source as fast as possible and is formalized with the notion of policy and reward, which we will introduce later. If agents have perfect knowledge of their own location and of the location of the source in space, the problem reduces to finding the shortest path.

In our problem, however, the agent does not know where the source is; hence, its position s\begin{document}$ s$\end{document} relative to the source is unknown or ‘partially observed’. Instead, it can sense odor released by the target. In the language of RL, odor is an ‘observation’ – but does it hold information about the position s\begin{document}$s$\end{document}? The answer is yes: several properties of odor stimuli depend on the distance from the source (Boie et al., 2018; Ackels et al., 2021; Nag and van Breugel, 2024). However, in the presence of turbulence, information lies in the statistics of the odor stimulus. Indeed, when odor is carried by a turbulent flow, it develops into a dramatically stochastic plume, that is a complex and convoluted region of space where the fluid is rich in odor molecules. Turbulent plumes break into structures that distort and expand while they travel away from their source and become more and more diluted (Falkovich et al., 2001; Shraiman and Siggia, 2000; Celani et al., 2014; Reddy et al., 2022), see Figure 1A. As a consequence, an agent within the plume experiences intermittent odor traces that endlessly switch on (whiff) and off (blank) Figure 1B. The intensity of odor whiffs and how they are interleaved with blanks depends on distance from release, as dictated by physics (Celani et al., 2014). Thus, the upshot of turbulent transport is that the statistical properties of odor traces depend intricately on the position of the agent relative to the source. In other words, information about the state s\begin{document}$s$\end{document} is hidden within the observed odor traces.

This positional information can be leveraged with a Bayesian approach that relies on guessing s\begin{document}$s$\end{document}, that is defining the probability distribution of the position, also called belief. This is the approach that has been more commonly adopted in the literature until now (Rigolli et al., 2022b; Heinonen et al., 2023; Loisy and Heinonen, 2023). Note that because of the complexity of these algorithms, only relatively simple measures of the odor are computationally feasible, for example instantaneous presence/absence. Here, we take a different model-free and map-free approach. Instead of guessing the current state s\begin{document}$ s$\end{document}, we ignore the spatial position and respond directly to the temporal traces of the odor cues. Two other algorithms have been proposed to solve partial observability by responding solely to odor traces with recurrent neural networks (Singh et al., 2023) and finite state controllers (Verano et al., 2023) that manipulate implicitly the odor traces. Here, instead, we manipulate odor traces explicitly, by defining memory as a moving window and by crafting a small number of features of odor traces.

To learn a response to odor traces, we set out to craft a finite set of olfactory states, o∈O\begin{document}$o\in O$\end{document}, so that they bear information about the location s\begin{document}$s$\end{document}. Defining the olfactory states is a challenge due to the dramatic fluctuations and irregularity of turbulent odor traces. To construct a fully interpretable low-dimensional state space, we aim at a small number of olfactory states that bear robust information about s\begin{document}$s$\end{document}, that is for all values of s\begin{document}$ s$\end{document}. We previously found that pairing features of sparsity as well as intensity of turbulent odor traces predicts robustly the location of the source for all s\begin{document}$s$\end{document} (Rigolli et al., 2022a). Guided by these results, we use these two features extracted from the temporal history of odor detections to define a small set of olfactory states.

We proceed to define a function that takes as input the history of odor detections along an agent’s path and returns its current olfactory state. We indicate with s(t)\begin{document}$ s(t)$\end{document} the (unknown) path of an agent, and with z(s(t),t)\begin{document}$z(s(t),t)$\end{document} the observations that is odor concentration along its path. First, we define a sensing memory T\begin{document}$T$\end{document} and we consider a short excerpt of the history of odor detections zT\begin{document}$z_{T}$\end{document} of duration T\begin{document}$T$\end{document} prior to the current time t\begin{document}$t$\end{document}. Formally, zT(t):={z(s(t′),t′)|t−T≤t′≤t}\begin{document}$ z_{T}(t):=\{z(s(t^{\prime}),t^{\prime})\,|\,t-T\leq t^{\prime}\leq t\}$\end{document}. Second, we measure the average intensity c\begin{document}$c$\end{document} (moving average of odor intensity over the time window T\begin{document}$T$\end{document}, conditioned to times when odor is above threshold), and intermittency i\begin{document}$ i$\end{document} (the fraction of time the odor is above threshold during the sensing window T\begin{document}$T$\end{document}). Both features c\begin{document}$c$\end{document} and i\begin{document}$i$\end{document} are described by continuous, positive real numbers. Third, we define 15 olfactory states by discretizing i\begin{document}$i$\end{document} and c\begin{document}$c$\end{document} in 3 and 5 bins, respectively. Intermittency i\begin{document}$i$\end{document} is bounded between 0 and 1, and we discretize it in 3 bins by defining two thresholds (33% and 66%). The average concentration, c\begin{document}$c$\end{document}, is bounded between 0 and the odor concentration at the source, hence prior information on the source is needed to discretize c\begin{document}$c$\end{document} using set thresholds. To avoid relying on prior information, we define thresholds of intensity as percentiles, based on a histogram that is populated online, along each agent’s path (see Materials and methods). The special case where no odor is detected over T\begin{document}$ T$\end{document} deserves attention, hence we include it as an additional state named ‘void state’ and indicate it with o≡∅\begin{document}$o\equiv\emptyset$\end{document}. When T\begin{document}$T$\end{document} is sufficiently long, the resulting olfactory states map to different spatial locations (Figure 1C, with T\begin{document}$ T$\end{document} equal to the simulation time). Hence, this definition of olfactory states can potentially mitigate the problem of partial observability using temporal traces, rather than spatial maps. But will these olfactory states with finite memory T\begin{document}$T$\end{document} guide agents to the source?

To answer this question, we trained tabular episodic Q learning (Sutton and Barto, 1998). In a nutshell, we use a simulator to place an agent at a random location in space at the beginning of each episode. The agent is not aware of its location in space, but it senses odor provided by the fluid dynamics simulator and thus can compute its olfactory state o\begin{document}$ o$\end{document}, based on odor detected along its path in the previous T\begin{document}$T$\end{document} sensing window. It then makes a move according to a set policy of actions a∼π0(a|o)\begin{document}$ a\sim\pi_{0}(a|o)$\end{document}. After the move, the simulator displaces the agent to its new location and relays the agent a penalty R=−σ\begin{document}$ R=-\sigma$\end{document} with σ=0.001\begin{document}$\sigma=0.001$\end{document} if it is not at the source and a reward R=1\begin{document}$ R=1$\end{document} if it reaches the source. The goal of RL is to find a policy of actions that maximizes the expected cumulative future reward G=Eπ(∑t=0∞γtRt+1)\begin{document}$G=E_{\pi}(\sum_{t=0}^{\infty}\gamma^{t}R_{t+1})$\end{document} where the expectation is over the ensemble of trajectories and rewards generated by the policy from any initial condition. Because reward is only positive at the source, the optimal policy is the one that reaches the source as fast as possible. To further encourage the agent to reach the source quickly, we introduce a discount factor γ<1\begin{document}$\gamma \lt 1$\end{document}.

Episodes where the agent does not reach the source are ended after Hmax=5000\begin{document}$H_{\text{max}}=5000$\end{document} with no positive reward. As it tries actions and receives rewards, the agent learns how good the actions are. This is accomplished by estimating the quality matrix Q(o,a)\begin{document}$Q(o,a)$\end{document}, that is the maximum expected cumulative reward conditioned to being in o\begin{document}$o$\end{document} and choosing action a\begin{document}$a$\end{document} at the present time: Q(o,a)=maxπEπ(∑t=0∞γtRt+1|ot=o,at=a)\begin{document}$Q(o,a)=\max_{\pi}E_{\pi}(\sum_{t=0}^{\infty}\gamma^{t}R_{t+1}|o_{t}=o,a_{t}=a)$\end{document}. At each step, the agent improves its policy by choosing more frequently putatively good actions. Once the agent has a good approximation of the quality matrix, the optimal policy corresponds to the simple readout: π∗(a|o)=δ(a−a∗(o))\begin{document}$\pi^{*}(a|o)=\delta(a-a^{*}(o))$\end{document} where a∗(o)=arg⁡maxaQ(o,a)\begin{document}$a^{*}(o)=\arg\max_{a}Q(o,a)$\end{document}, for non-void states o≠∅\begin{document}$ o\neq\emptyset$\end{document}.

By repeating training using different values of T\begin{document}$T$\end{document}, we find that performance depends on memory and an optimal memory T∗\begin{document}$T^{*}$\end{document} exists (Figure 2A). Why is there an optimal memory? The shortest memory T=1\begin{document}$ T=1$\end{document} corresponds to instantaneous olfactory states: the instantaneous contour maps of the olfactory states are convoluted, and the void state is pervasive (Figure 2C, top). As a consequence, agents often activate recovery even when they are within the plume. The policy almost always leads to the source (f+=79%±13%\begin{document}$f^{+}=79\%\pm 13\%$\end{document}) but follows lengthy convoluted paths (τmin/τ=0.14±0.05\begin{document}$\tau_{\text{min}}/\tau=0.14\pm 0.05$\end{document}, Figure 2C, bottom). As memory increases, the olfactory states become smoother and agents encounter fewer voids (Figure 2C, center), perform straighter trajectories (τmin/τ=0.5±0.3\begin{document}$\tau_{\text{min}}/\tau=0.5\pm 0.3$\end{document}), and reach the source reliably (f+=95%±8%\begin{document}$ f^{+}=95\%\pm 8\%$\end{document}), Figure 2A, bottom. Further increasing memory leads to even less voids within the plume and even smoother olfactory states. However – perhaps surprisingly – performance does not further improve but slightly decreases (at T=50\begin{document}$ T=50$\end{document}, f+=94%±8%\begin{document}$ f^{+}=94\%\pm 8\%$\end{document} and τmin/τ=0.38±0.36\begin{document}$\tau_{\text{min}}/\tau=0.38\pm 0.36$\end{document}). A long memory is deleterious because it delays recovery from accidentally exiting the plume, thus increasing the number of voids outside of the plume (Figure 2C, bottom). Indeed, agents often leave the plume accidentally as they measure their olfactory state while they move. They receive no warning, but realize their mistake after T\begin{document}$ T$\end{document} steps, when they enter the void state and activate recovery to re-enter the plume. The delay is linear with memory when agents recover by backtracking, but it depends on the recovery strategy (see Materials and methods and Figure 2—figure supplement 1).

Thus, short memories increase time in void within the plume, whereas long memories increase time in void outside the plume: the optimal memory minimizes the overall chances to experience the void (Figure 2B). Intuitively, T∗\begin{document}$T^{*}$\end{document} should match the typical duration τb\begin{document}$\tau_{b}$\end{document} of blanks encountered within the plume, so that voids within the plume are effectively ignored without delaying recovery unnecessarily. Consistently, ⟨τb⟩\begin{document}$\langle\tau_{b}\rangle$\end{document} averaged across all locations and times within the plume is ⟨τb⟩=9.97±41.16\begin{document}$\langle\tau_{b}\rangle=9.97\pm 41.16$\end{document}, comparable with the optimal memory T∗\begin{document}$T^{*}$\end{document} (Figure 2A).

There is no way to select the optimal memory T∗\begin{document}$T^{*}$\end{document} without comparing several agents or relying on prior information on the blank durations. In order to avoid prior information, we venture to define memory adaptively along each agent’s path, using the intuition outlined above. We define a buffer memory Tb\begin{document}$ T_{b}$\end{document}, and let the agent respond to a sensing window T<Tb\begin{document}$T \lt T_{b}$\end{document}. Ideally, we would like to set T∼⟨τb⟩\begin{document}$T\sim\langle\tau_{b}\rangle$\end{document}. With this choice, blanks shorter than the average blank are ignored, as they are expected within the plume, whereas blanks longer than average initiate recovery, as they signal that the agent exited the plume. However, agents do not have access to ⟨τb⟩\begin{document}$\langle\tau_{b}\rangle$\end{document} hence we set T=τb−\begin{document}$T=\tau_{b}^{-}$\end{document}, where τb−\begin{document}$\tau_{b}^{-}$\end{document} is the most recent blank experienced by the agent. With this choice, the sensing memory T\begin{document}$ T$\end{document} fluctuates considerably along an agent’s path, due to turbulence (Celani et al., 2014 and Figure 3A–B). Note that blanks are estimated along paths, thus the statistics of T\begin{document}$T$\end{document} only qualitatively matches the Eulerian statistics of τb\begin{document}$ \tau_{b}$\end{document}. Despite the fluctuations, performance using the adaptive memory nears performance with the optimal memory (Figure 3C). This result confirms our intuition that memory should match the blank time. The advantage of adaptive memory is that it relies solely on experience, with no prior information whatsoever. This is unlike T∗\begin{document}$T^{*}$\end{document} which can only be selected using prior information, with no guarantee of generalization to other plumes.

So far, our agents combine a learned policy from non-void states to a heuristic from the void state, which we called the recovery strategy. We have considered biologically inspired heuristics where searchers make it back to locations within the plume by retracing their path backward. To further strip the algorithm of heuristics, we ask whether the recovery strategy may be learned, rather than fixed a priori. To this end, we split the void state in many states, labeled with the time elapsed since first entering the void. We pick 50 void states, as less than 50 void states results in no convergence, and states above 50 are useless because they are rarely visited. The counter is reset to 0 whenever the searcher detects the odor. The definition of the 15 non-void states o1,…,o15\begin{document}$ o_{1},\dots,o_{15}$\end{document} remains unaltered. Interestingly, with this added degree of freedom, the agent learns an even better recovery strategy as reflected by all our measures of performance (Figure 3D). Note that the learned recovery strategy resembles the casting behavior observed in flying insects (David et al., 1983), as discussed below. In fact, insects deploy a range of recovery strategies depending on locomotor mode and environment. To corroborate these results, we compare performance using two additional biologically-inspired recovery strategies, i.e. circling (observed in windless environments Stupski and van Breugel, 2024a), and cast & surge (David et al., 1983) as well as a Brownian recovery which does not have a direct biological relevance but represents a simple computational benchmark. The learned recovery outperforms all heuristic recoveries, as seen by the cumulative reward G\begin{document}$G$\end{document} (Figure 3D). Circling is the second-best recovery and shortly follows the learned recovery. Circling achieves nearly optimal performance by further decreasing failures (metrics f+\begin{document}$f^{+}$\end{document}), but slowing down (metrics g+\begin{document}$g^{+}$\end{document} and τmin/τ\begin{document}$\tau_{min}/\tau$\end{document}).

Finally, we test the performance of the trained agents on six environments, characterized by distinct fluid flows and odor plumes (Figure 5 and Materials and methods). Environment 1 (‘bulk native’) is the environment where the agents were originally trained; Environment 2 (‘bulk sparser’) is obtained by increasing the threshold of detection, which makes the signals considerably more sparse with longer blanks. Environments 3 (‘surface’) and 4 (‘surface sparser’) are closer to the lower surface of the simulated domain, where the plume is smaller and fluctuates less. Environment 5 (‘bulk lower Re’) is a similar geometry, but obtained for a smaller Reynolds number and a different way to generate turbulence. Finally, Environment 6 (‘bulk higher Re’) has an even larger Reynolds number, a longer domain, and a smaller source, which creates an even more dramatically sparse signal. All bulk environments (1, 2, 5, and 6) are representative of conditions encountered far from a substrate, for example by flying or swimming organisms. The two surface environments (3 and 4) represent odor near surfaces relevant to terrestrial or benthic navigation (but not directly applicable to trail tracking, where odor traces on the substrate are tracked). Note that we consider a Schmidt number Sc = 1 appropriate for odors in air but not in water. However, we expect a weak dependence on the Schmidt number as the Batchelor and Kolmogorov scales are below the size of the source and we are interested in the large scale statistics (Falkovich et al., 2001; Celani et al., 2014; Duplat et al., 2010).

We consider agents with adaptive memory and compare the five recovery strategies discussed above – backtracking, learned, circling, cast and surge, and Brownian, see Figure 5B. Comparing performance across environments, we find that: (i) although performance is degraded when testing in non-native environments, backtracking, learned and circling recoveries with adaptive memory are still extremely likely to find the source. The upshot of generalization is that agents may navigate distinct turbulent plumes using a baseline strategy learned in a specific plume. Importantly, as most of these agents still do reach the source, fine-tuning may enable efficient adaptation to different environments. Further work is needed to establish how much fine-tuning is needed to fully adapt to different environments. (ii) Brownian and cast and surge recoveries have the lowest performance and generalization across all environments. Cast and surge is often used as a comparison (Verano et al., 2023) and can be extremely effective: our results do not contradict the literature, but simply showcase that cast width and surge length need to be carefully defined. (iii) The cumulative reward G\begin{document}$G$\end{document} shows that the learned recovery is the best at generalizing to environments 2, 5, and 6. Particularly, in the most intermittent Environment 6 a striking 91% of agents succeed in finding the source, with trajectories less than twice as long as the shortest path to the source. (iv) Circling is the best at generalizing to environments 3 and 4, representative of less intermittent regions near the substrate. As observed for the native environment, circling favors success rate (metrics f+\begin{document}$ f^{+}$\end{document}) against speed (metrics g+\begin{document}$g^{+}$\end{document} and τmin/τ\begin{document}$\tau_{\text{min}}/\tau$\end{document}).

The data we used to train the agents (Environment 1) is a set of 2598 matrices {Dt}t=12598\begin{document}$ \{D_{t}\}_{t=1}^{2598}$\end{document}. Every matrix Dt∈R1225×280\begin{document}$D_{t}\in\mathbb{R}^{1225\times 280}$\end{document} contains the odor intensity in every position (i,j)\begin{document}$ (i,j)$\end{document} i.e. (Dt)i,j\begin{document}$ (D_{t})_{i,j}$\end{document} represents the odor intensity in position (i,j)\begin{document}$ (i,j)$\end{document} at time t\begin{document}$ t$\end{document}. The source of odor is in position (20, 142), and in order to simplify the training, we considered as terminal states every position in a circle centered in the source position and with radius 10 called the source region. Data are obtained from a direct numerical simulation of the Navier-Stokes equations and the equations of transport of the odor. Environments 1–4 are derived from Simulation 1, a direct numerical simulation of a channel flow described in Rigolli et al., 2022a and used to develop a POMDP algorithm in Rigolli et al., 2022b, dataset available from Rigolli et al., 2022c. The simulation represents a boundary layer whose dimensions match the lowest ∼1 m of the atmosphere and with horizontal dimensions ∼1.9 × 9.5 m and Reynolds number Reλ∼1400\begin{document}$ Re_{\lambda}\sim 1400$\end{document} (on the low side of atmospheric Reynolds typically ranging from 1000–10,000 Gulitski et al., 2007). Odor snapshots are extracted at a height ∼.5m\begin{document}$\sim.5\,\rm m$\end{document} from the ground for Environments 1 and 2 and ∼.01m\begin{document}$ \sim.01\,\rm m$\end{document} for Environments 3 and 4 respectively. We preprocess the data to zero every entry of these matrices when they are smaller than a noise level nlvl:=3×10−6\begin{document}$ n_{\text{lvl}}:=3\times 10^{-6}$\end{document} (or 0.13% relative to concentration at the source). The noise level is increased to 0.22% in Environments 2 and 4. Data information are summarized in Tables 2 and 3. Levels of intermittency in Figure 1 show that only a thin core region has intermittency larger than 66%, whereas the most challenging regions at the edge of the plume have intermittency under 33%. For reference, experimental values of 25% to 20% were reported for a surrogate odor in the atmospheric boundary layer, along the centerline at 2–15 m from the source (Murlis and Jones, 1981).

Environments 5 and 6 correspond to horizontal slices at mid height extracted from two additional simulations we performed to corroborate the results (Simulation 2 and 3). In Simulation 2, the odor is advected by a turbulent open channel flow, with three hemispherical obstacles placed on the ground close to the inlet to generate turbulence. The Navier-Stokes equations (1) and advection-diffusion equation for odor transport (3) are solved using a central second-order finite difference scheme. The convective terms are discretized in time using an explicit Adams–Bashforth method, and the viscous and diffusion terms using an implicit Crank-Nicolson method (Viola et al., 2023; Viola et al., 2020; Viola et al., 2022). The code is written in Fortran and is GPU parallelized. The channel is divided into 1024 × 256 × 128 grid points along streamwise, spanwise, and wall-normal directions respectively. The corresponding average spatial resolutions are Δx=5η,Δy=5η,Δz=4η\begin{document}$\Delta x=5\eta,\Delta y=5\eta,\Delta z=4\eta$\end{document}, where η\begin{document}$ \eta$\end{document} is the Kolmogorov length scale. Three hemispheres of radius 100η\begin{document}$ 100\eta$\end{document} are placed at a distance of 250η\begin{document}$ 250\eta$\end{document} from the inlet on the ground, equally spaced along the spanwise direction. The obstacles are implemented using the immersed boundary method (Verzicco et al., 2025). The channel is forced using a constant pressure gradient. For the velocity field, we impose a no-slip boundary condition at the ground and on the obstacles (u=0\begin{document}$ {\boldsymbol{u}}=0$\end{document}) and a free-slip boundary on top (uz=0,∂zux=∂zuy=0\begin{document}$ u_{z}=0,\partial_{z}u_{x}=\partial_{z}u_{y}=0$\end{document}). The velocity field is periodic along the streamwise and spanwise directions. The bulk Reynolds number is 7800. For the odor field, we impose Dirichlet condition (c=0\begin{document}$ c=0$\end{document}) at the ground, on the obstacles and inlet, no-flux (∂zc=0\begin{document}$\partial_{z}c=0$\end{document}) on top, and outflow along other directions. Similar to the native environment, we choose the Schmidt number to be 1. The odor source is located downstream of the obstacle and centered at [640η\begin{document}$640\eta$\end{document}, 640η\begin{document}$ 640\eta$\end{document}, 256η\begin{document}$ 256\eta$\end{document}] along streamwise, spanwise, and wall-normal directions. respectively. The odor source has a Gaussian profile with a standard deviation of 8η\begin{document}$ 8\eta$\end{document}.

Simulation 3 is similar to Simulation 2, albeit with a higher bulk Reynolds number of 17,500. Here, the channel is divided into 2000 × 500 × 200 grid points and has an average spatial resolution of Δx=Δy=Δz=5.5η\begin{document}$ \Delta x=\Delta y=\Delta z=5.5\eta$\end{document}. The odor source has a Gaussian profile centered at [825η\begin{document}$ 825\eta$\end{document}, 1375η\begin{document}$ 1375\eta$\end{document}, 550η\begin{document}$550\eta$\end{document}] with a standard deviation of 3η\begin{document}$ 3\eta$\end{document}. For Environments 5 and 6, the noise level is 0.01% relative to concentration at the source. See Table for a summary of parameters and how they match the physical dimensions of the domain.(1)ρ(∂u∂t+u.∇u)=−∇P+μ∇2u+f;∇u=0.\begin{document}$$\displaystyle \begin{array}{ll}\rho \left(\frac{\partial u}{\partial t} + \boldsymbol{u}.\boldsymbol{\nabla u} \right) =- {\nabla \boldsymbol P}+\mu \boldsymbol\nabla^2 u + \boldsymbol f;\\\quad \quad \quad \quad \quad \boldsymbol{\nabla u}=0. \end{array}$$\end{document}(2)∂z∂t+u.∇z=D∇2z+s.\begin{document}$$\displaystyle \frac{\partial z}{\partial t} +\mathbf{u} .\nabla z=D\nabla^2 z+s.$$\end{document}

Each agent stores the odor concentrations detected in the previous T\begin{document}$T$\end{document} time steps in a vector M=(z(s(t−T),t−T),...,z(s(t),t))\begin{document}$ \mathbf{M}=(z(s(t-T),t-T),...,z(s(t),t))$\end{document}. We introduce an adaptive sensitivity threshold function sthr(⋅)\begin{document}$s_{\text{thr}}(\cdot)$\end{document} defined as(3)sthr(T):=max{CthrT∑i=1TMi,nthr},\begin{document}$$\displaystyle s_\text{thr}(T) := \max \Big\{ \frac{C_\text{thr}}{T} \sum\limits_{i = 1}^{T} M_i , n_\text{thr} \Big\}, $$\end{document}

where Mi\begin{document}$ M_{i}$\end{document} denotes the i-th element of M\begin{document}$ M$\end{document} and Cthr>0\begin{document}$C_{\text{thr}} \gt 0$\end{document} is a scaling constant (in our experiments we set it as 0.5). T\begin{document}$ T$\end{document} denotes the cardinality of M\begin{document}$ M$\end{document}. Given a memory M\begin{document}$ M$\end{document}, we can define the filtered memory ΔM\begin{document}$\Delta^{M}$\end{document} as the set which contains every element of the memory M\begin{document}$ M$\end{document} that is higher than the sensitivity threshold sthr(M)\begin{document}$ s_{\text{thr}}(M)$\end{document}, that is(4)ΔM:={z∈M|z>sthr(M)}.\begin{document}$$\displaystyle \Delta^M := \{z \in M \, | \, z \gt s_\text{thr}(M)\}. $$\end{document}

Then at time step t\begin{document}$ t$\end{document}, given the agent memory Mt\begin{document}$ M_{t}$\end{document}, we define the average intensity c(Mt)\begin{document}$ c(M_{t})$\end{document} and the intermittency i(Mt)\begin{document}$ i(M_{t})$\end{document} as:(5)c(Mt):={1|ΔMt|∑i=1|ΔMt|(ΔMt)i,|ΔMt|>00,i(Mt):=|ΔMt||Mt|.\begin{document}$$\displaystyle c(M_t) := \left\{\begin{array}{ll} \frac{1}{|\Delta^{M_t}|} \sum\limits_{i = 1}^{|\Delta^{M_t}|} \Big(\Delta^{M_t} \Big)_i, & |\Delta^{M_t}| \gt 0\\ & \\ 0 \end{array} \right.,\\ i(M_t) := \frac{| \Delta^{M_t} |}{|M_t|}. $$\end{document}

Note that the average intensity is defined on the filtered memory ΔM\begin{document}$ \Delta^{M}$\end{document}, that is conditioned to detecting odors above threshold. Since the features defined in Equation 6 return real numbers, in order to use (tabular) q-learning, we need to discretize them. We denote with i¯(Mt)\begin{document}$ \bar{i}(M_{t})$\end{document} the discretized intermittency. This is defined as follow(6)i¯(Mt):={0,ifi(Mt)≤0.331,if0.33<i(Mt)≤0.662,ifi(Mt)>0.66\begin{document}$$\displaystyle \bar{i} (M_t) := \left\{ \begin{array}{ll} 0 ,& \text{if} \quad i(M_t) \leq 0.33\\ 1 ,& \text{if} \quad 0.33 \lt i(M_t) \leq 0.66\\ 2 ,& \text{if} \quad i(M_t) \gt 0.66 \end{array} \right.$$\end{document}

The average intensity is bounded between zero and the maximum concentration of odor at the source. To avoid prior information on the source, we use a more structured procedure to discretize the average intensity online, based on the agent’s experience only. At every time step t\begin{document}$t$\end{document}, the average intensity c(Mt)\begin{document}$ c(M_{t})$\end{document} is computed and collected in a dataset Xt\begin{document}$ X_{t}$\end{document}, that isXt:={c(M0),⋯,c(Mt)}.\begin{document}$$\displaystyle X_t := \{c(M_0), \cdots, c(M_t)\}. $$\end{document}

Then, its discretized value is obtained by the following rule:(7)c¯(Mt,Xt):={0,c(Mt)≤p(Xt,25)1,p(Xt,25)<c(Mt)≤p(Xt,50)2,p(Xt,50)<c(Mt)≤p(Xt,80)3,p(Xt,80)<c(Mt)≤p(Xt,99)4,c(Mt)>p(Xt,99),\begin{document}$$\displaystyle \bar{c}(M_t, X_t) := \left\{ \begin{array}{ll} 0 ,& c(M_t) \leq \text{p}(X_t, 25)\\ 1 ,& \text{p}(X_t, 25) \lt c(M_t) \leq \text{p}(X_t, 50)\\ 2 ,& \text{p}(X_t, 50) \lt c(M_t) \leq \text{p}(X_t, 80)\\ 3 ,& \text{p}(X_t, 80) \lt c(M_t) \leq \text{p}(X_t, 99)\\ 4 ,& c(M_t) \gt \text{p}(X_t, 99) \end{array} \right.,$$\end{document}

where p(Xt,n)\begin{document}$p(X_{t},n)$\end{document} denotes the n-th percentile of Xt\begin{document}$ X_{t}$\end{document}. Finally, we can define the feature map ϕt\begin{document}$ \phi_{t}$\end{document} as a function of the memory Mt\begin{document}$ M_{t}$\end{document} and the dataset of average intensities Xt\begin{document}$ X_{t}$\end{document} at time step t\begin{document}$ t$\end{document}ϕt(Mt,Xt):=[i¯(Mt),c¯(Mt,Xt)].\begin{document}$$\displaystyle \phi_t(M_t, X_t) := [\bar{i}(M_t), \bar{c}(M_t, X_t)]. $$\end{document}

This defines the current olfactory state st\begin{document}$ s_{t}$\end{document} that is at time step t\begin{document}$ t$\end{document}, the agent is in the olfactory state ot:=ϕt(Mt,Xt)\begin{document}$o_{t}:=\phi_{t}(M_{t},X_{t})$\end{document}. The case where the agent has no odor detections above threshold in its current memory, that is |Δ(Mt)|=0\begin{document}$|\Delta(M_{t})|=0$\end{document} corresponds to an additional state called void state (∅\begin{document}$ \emptyset$\end{document}) in the main text.

Now, we describe how the agent interacts with the environment to solve the navigation problem. At every time step t∈N\begin{document}$ t\in\mathbb{N}$\end{document}, the agent observes an odor point zt\begin{document}$ z_{t}$\end{document} and updates its memory, including the new observation and removing the oldest that is it defines a memory Mt\begin{document}$ M_{t}$\end{document} with the following rule(8)Mt:=[(Mt−1)2,⋯,(Mt−1)|Mt−1|,ot].\begin{document}$$\displaystyle M_t := \Bigg[\Big(M_{t - 1} \Big)_{2}, \cdots, \Big(M_{t - 1} \Big)_{|M_{t - 1}|}, o_t \Bigg]. $$\end{document}

Then, it updates the dataset of average intensities that is Xt:=Xt−1∪{c(Mt)}\begin{document}$ X_{t}:=X_{t-1}\cup\{c(M_{t})\}$\end{document} and it computes the olfactory state ot\begin{document}$o_{t}$\end{document}. According to ot\begin{document}$ o_{t}$\end{document}, the agent chooses an action at\begin{document}$ a_{t}$\end{document} using a policy. As indicated in the main text, actions are the coordinate directions that is we define an action set A\begin{document}$ \mathcal{A}$\end{document} as followA:={e1,e2,−e1,−e2},\begin{document}$$\displaystyle \mathcal{A} := \{e_1, e_2, -e_1, -e_2\}, $$\end{document}

where ei\begin{document}$ e_{i}$\end{document} denotes the i-th canonical base. Actions are steps in any of the four directions, labeled relative to the mean flow which is assumed fixed and known. The gridworld is infinite, in that agents can leave indefinitely. If they exit the simulation box, they continue to receive zero signal and negative reward –0.001. As explained in the main text, actions are selected using one of two policies according to the current olfactory state ot\begin{document}$o_{t}$\end{document}. More precisely, if the olfactory state ot\begin{document}$ o_{t}$\end{document} is not the void state, then the (ϵ\begin{document}$ \epsilon$\end{document}-greedy) Q-learning policy is used. Formally, let Q\begin{document}$ Q$\end{document} be the Q matrix of the agent and let ot≠∅\begin{document}$ o_{t}\neq\emptyset$\end{document}, then the agent plays the action at\begin{document}$ a_{t}$\end{document} such that(9)at={a∈arg⁡maxa∈AQ(ot,a)with probability 1−ϵa∼U(A)with probability ϵ,\begin{document}$$\displaystyle a_t = \left\{ \begin{array}{ll} a \in \arg\max\limits_{a \in \mathcal{A}} Q(o_t, a) & \text{with probability } 1 - \epsilon\\ a \sim \mathcal{U}(\mathcal{A}) & \text{with probability } \epsilon \end{array} \right., $$\end{document}

where, with a∼U(A)\begin{document}$ a\sim\mathcal{U}(\mathcal{A})$\end{document}, we indicate an action a uniformly sampled from A\begin{document}$ \mathcal{A}$\end{document}. At the test phase, the exploration-exploitation parameter ϵ\begin{document}$ \epsilon$\end{document} is set to 0, and thus, in an olfactory state ot≠∅\begin{document}$ o_{t}\neq\emptyset$\end{document} the policy is deterministic. While training phase behavior is described in the next paragraphs. In the void state ot=∅\begin{document}$o_{t}=\emptyset$\end{document}, the agent chooses the action at∈A\begin{document}$ a_{t}\in\mathcal{A}$\end{document} according to a separated policy called recovery strategy. In our experiments, we defined and compared three different recovery strategies: Brownian, Backtracking, and Learned.

It is the simplest strategy we consider, consisting of playing random actions in the void state. Suppose that at time step t\begin{document}$ t$\end{document}, the agent is in the void olfactory state, that is ot=∅\begin{document}$ o_{t}=\emptyset$\end{document}, then at\begin{document}$ a_{t}$\end{document} is sampled uniformly from the action set A\begin{document}$ \mathcal{A}$\end{document}. However, it is important to note that long-memory agents start to recover when they are already far from the plume, and hitting the plume by random walk is prohibitively long. To avoid wandering away from the plume, the memory is constrained to be shorter, consistent with the observation that the optimal memory is T∗=3\begin{document}$T^{*}=3$\end{document} to 5, much shorter than for backtracking. At this memory, several blanks within the plume will cause the agent to recover, hence the lower performance of the Brownian recovery.

In order to accelerate recovery from accidentally exiting the plume, we let the agents backtrack to the position where they last detected the odor. To this end, we first enumerate the actions with numbers from one to four. Then, we introduce a new memory called action memory A\begin{document}$ A$\end{document}. For simplicity, we consider the setting in which |A|=|M|\begin{document}$|A|=|M|$\end{document}. At time step t=0\begin{document}$ t=0$\end{document}, this memory is initialized as a vector of zeros indicating that the action memory is empty that is we define A0∈N|M|\begin{document}$ A_{0}\in\mathbb{N}^{|M|}$\end{document} such that for every i=1,⋯,|A|\begin{document}$ i=1,\cdots,|A|$\end{document}Ai=0.\begin{document}$$\displaystyle A_i = 0. $$\end{document}

For every time step t>0\begin{document}$ t \gt 0$\end{document}, the agent observes an odor point zt\begin{document}$ z_{t}$\end{document} and updates the memory through (Equation 8). Moreover, the action memory is updated according to the status of the memory. If the last observation is smaller than the sensitivity threshold, that is zt<sthr(Mt)\begin{document}$ z_{t} \lt s_{\text{thr}}(M_{t})$\end{document}, the action previously played at−1\begin{document}$ a_{t-1}$\end{document} (represented by a natural number in [1,4]\begin{document}$ [1,4]$\end{document}) is stored in the action memory, that is for some Δ>0\begin{document}$ \Delta \gt 0$\end{document}, letAt−1=[at−Δ,⋯,at−2,0,⋯,0].\begin{document}$$\displaystyle A_{t - 1} = [a_{t - \Delta}, \cdots, a_{t - 2}, 0, \cdots, 0]. $$\end{document}

ThenAt=[at−Δ,⋯,at−2,at−1,⋯,0].\begin{document}$$\displaystyle A_{t} = [a_{t - \Delta}, \cdots, a_{t - 2}, a_{t - 1}, \cdots, 0]. $$\end{document}

If at time step t\begin{document}$ t$\end{document}, the observation zt\begin{document}$ z_{t}$\end{document} is larger than the sensitivity threshold then the action memory is reset, that is At∈N|M|\begin{document}$ A_{t}\in\mathbb{N}^{|M|}$\end{document} with (At)i=0\begin{document}$ (A_{t})_{i}=0$\end{document} for every i. If at time step t\begin{document}$ t$\end{document}, the memory is empty, that is c(Mt)=0\begin{document}$ c(M_{t})=0$\end{document}, then the backtracking procedure is executed: the last non-zero element of the action memory is extracted, and the inverse action is played that is for some Δ>0\begin{document}$ \Delta \gt 0$\end{document}, letAt−1=[at−Δ,⋯,at−2].\begin{document}$$\displaystyle A_{t-1} = [a_{t - \Delta}, \cdots, a_{t - 2}]. $$\end{document}

Then, it plays the action at−2\begin{document}$ a_{t-2}$\end{document} and updates the action memory as followAt=[at−Δ,⋯,at−3,0].\begin{document}$$\displaystyle A_{t} = [a_{t - \Delta}, \cdots, a_{t - 3}, 0]. $$\end{document}

This procedure is repeated until either an observation larger than the sensitivity threshold is obtained or the action memory becomes empty. In the former case, the action memory is cleared, and the action is chosen according to the Q-learning policy (Equation 9). In the latter case, a random action is played.

Note that this strategy only provides exploration after the backtracking fails to recover detections. Also, if agents start with no detection at time 0, the procedure is equivalent to Brownian motion.

In this case, the recovery strategy consists of adopting a circling behavior (Stupski and van Breugel, 2024a) when the agent is in a void state. The agent keeps in memory two counters, tvoid\begin{document}$ t_{\text{void}}$\end{document} and Tchange\begin{document}$ T_{\text{change}}$\end{document}, as well as an action avoid∈A\begin{document}$ a_{\text{void}}\in\mathcal{A}$\end{document}, initialized to 0, 1, and e1\begin{document}$ e_{1}$\end{document}, respectively. The first counter represents the consecutive number of void observations, avoid\begin{document}$a_{\text{void}}$\end{document} is the action to play when the void state is reached, and Tchange\begin{document}$ T_{\text{change}}$\end{document} is a time threshold that indicates when to switch the action avoid\begin{document}$ a_{\text{void}}$\end{document}. When the agent reaches a void state, it plays the action avoid\begin{document}$ a_{\text{void}}$\end{document} and increments the counter tvoid\begin{document}$ t_{\text{void}}$\end{document} by one. If tvoid=Tchange\begin{document}$t_{\text{void}}=T_{\text{change}}$\end{document} then the agent resets tvoid\begin{document}$ t_{\text{void}}$\end{document} to zero, increases Tchange\begin{document}$ T_{\text{change}}$\end{document} by one, and updates the action avoid\begin{document}$ a_{\text{void}}$\end{document} as followsavoid←{e2if avoid=e1−e1if avoid=e2−e2if avoid=−e1e1if avoid=−e2\begin{document}$$\displaystyle a_\text{void} \leftarrow \left\{\begin{array}{cc} e_2 & \text{if } a_\text{void} = e_1 \\ -e_1 & \text{if } a_\text{void} = e_2 \\ -e_2 & \text{if } a_\text{void} = -e_1 \\ e_1 & \text{if } a_\text{void} = -e_2 \\ \end{array} \right. $$\end{document}

When the agent receives a non-void observation, it resets tvoid\begin{document}$ t_{\text{void}}$\end{document} to 0, Tchange\begin{document}$ T_{\text{change}}$\end{document} to 1, and avoid\begin{document}$ a_{\text{void}}$\end{document} to e1\begin{document}$ e_{1}$\end{document}.

In this case, the agent plays a cast-surge behavior (Baker, 1990) when reaching the void state. As in the circling recovery strategy, the agent keeps two counters, tvoid\begin{document}$ t_{\text{void}}$\end{document} and Tchange\begin{document}$ T_{\text{change}}$\end{document}, as well as an action avoid∈A\begin{document}$ a_{\text{void}}\in\mathcal{A}$\end{document}, initialized to 0, 1, and e2\begin{document}$e_{2}$\end{document}, respectively. At every step in the void state, the agent plays the action avoid\begin{document}$ a_{\text{void}}$\end{document} and increments the void counter tvoid\begin{document}$t_{\text{void}}$\end{document} by one. If tvoid=Tchange\begin{document}$ t_{\text{void}}=T_{\text{change}}$\end{document}, the agent takes an upwind step, doubles the value of Tchange\begin{document}$ T_{\text{change}}$\end{document}, updates avoid\begin{document}$ a_{\text{void}}$\end{document} by setting it to −avoid\begin{document}$ -a_{\text{void}}$\end{document}, and resets tvoid\begin{document}$ t_{\text{void}}$\end{document} to zero. When the agent receives a non-void observation, it resets tvoid\begin{document}$t_{\text{void}}$\end{document} to 0, Tchange\begin{document}$ T_{\text{change}}$\end{document} to 1, and avoid\begin{document}$ a_{\text{void}}$\end{document} to e2\begin{document}$ e_{2}$\end{document}.

In this case, the recovery policy is learned by splitting the void state into several states labeled by the time since entry in the void state. In our experiments, we split the void state into 50 states. Actions are then learned as in all other non-void states, and the optimal action is always chosen with (Equation 9).

An agent starts at a random location within the odor plume at time 0. Its memory is initialized with the prior |M0|\begin{document}$ |M_{0}|$\end{document} odor detections at its initial location M0=[z−|M0|,⋯,z0]\begin{document}$M_{0}=[z_{-|M_{0}|},\cdots,z_{0}]$\end{document}, obtained from the fluid dynamics simulation. The Q-function Q0\begin{document}$ Q_{0}$\end{document} is initialized with 0.6 for all actions and olfactory states. The first dataset of average intensities contains the first value X0={c(M0)}\begin{document}$ X_{0}=\{c(M_{0})\}$\end{document}. At every time step t>0\begin{document}$ t \gt 0$\end{document}, the agent gets an odor observation zt\begin{document}$z_{t}$\end{document} from its new position and updates its memory, including the new observation and removing the oldest; the olfactory state ot\begin{document}$o_{t}$\end{document} is computed (as described in previous paragraphs). The dataset of average intensities is updated: Xt=Xt−1∪{c(Mt)}\begin{document}$ X_{t}=X_{t-1}\cup\{c(M_{t})\}$\end{document}. Exploration-exploitation parameter ϵk\begin{document}$ \epsilon_{k}$\end{document} is scheduled as followϵk=ηinitexp⁡(−ηdecayk),\begin{document}$$\displaystyle \epsilon_k = \eta_\text{init} \exp(- \eta_\text{decay} k), $$\end{document}

where, in our experiments, ηinit=0.99\begin{document}$ \eta_{\text{init}}=0.99$\end{document} and ηdecay=0.0001\begin{document}$ \eta_{\text{decay}}=0.0001$\end{document}. At every episode k\begin{document}$ k$\end{document}, the Q-function is updated at every time step t\begin{document}$ t$\end{document} asQk+1(st,at):=(1−αk)Qk(st,at)+αk(rt+γmaxa′Qk(st+1,a′)),\begin{document}$$\displaystyle Q_{k + 1}(s_t, a_t) := (1 - \alpha_k) Q_k(s_t, a_t) + \alpha_k (r_t + \gamma \max\limits_{a'} Q_k(s_{t + 1}, a')), $$\end{document}

where Rt\begin{document}$ R_{t}$\end{document} is the immediate reward received playing the action at\begin{document}$ a_{t}$\end{document} and ot+1\begin{document}$ o_{t+1}$\end{document} are the current and the next olfactory states and αk\begin{document}$\alpha_{k}$\end{document} is the learning rate at episode k\begin{document}$ k$\end{document}. This is scheduled asαk=αinitexp⁡(−αdecayk),\begin{document}$$\displaystyle \alpha_k = \alpha_\text{init} \exp(- \alpha_\text{decay} k), $$\end{document}

where, in our experiments, αinit=0.25\begin{document}$ \alpha_{\text{init}}=0.25$\end{document} and αdecay=0.001\begin{document}$ \alpha_{\text{decay}}=0.001$\end{document}. For the experiments, agents are trained in 100,000 episodes and a horizon of 5000 steps. The agent velocity is set to 10, and the discount factor is γ=0.9999\begin{document}$ \gamma=0.9999$\end{document}.

To evaluate the performance of the different agents, we consider four metrics: the cumulative reward G\begin{document}$ G$\end{document} (which is the actual quantity that the algorithm optimizes for); normalized time (defined below); the fraction of success f+\begin{document}$f^{+}$\end{document} and the value conditioned on success g+\begin{document}$ g^{+}$\end{document}. For a fixed position (i,j)\begin{document}$ (i,j)$\end{document}, we denote with τmin(i,j)\begin{document}$ \tau_{\text{min}}(i,j)$\end{document} the minimum number of steps required to reach the source region from (i,j)\begin{document}$(i,j)$\end{document} that is the length of the shortest path.

We define Dinit\begin{document}$D_{\text{init}}$\end{document} the set of points in which the first observation is above the sensitivity threshold (valid points). For each initial position (i,j)∈Dinit\begin{document}$ (i,j)\in D_{\text{init}}$\end{document}, let τ(i,j)\begin{document}$\tau(i,j)$\end{document} be the duration of the path obtained by an agent to reach the source. Note that τ(i,j)\begin{document}$\tau(i,j)$\end{document} is a random variable for the stochastic backtracking and Brownian recoveries, but it is deterministic for the learned strategy that has no random components. For each admissible location (i,j)\begin{document}$ (i,j)$\end{document}, we define four performance metrics:G(i,j)=⟨e−λτ(i,j)−σ1−γ(1−e−λτ(i,j))⟩\begin{document}$$\displaystyle \begin{array}{ll}G(i,j) =&\langle e^{-\lambda \tau(i,j)}-\frac{\sigma}{1-\gamma}(1-e^{-\lambda \tau(i,j)})\rangle\end{array}$$\end{document}f+(i,j)=nsuccess(i,j)nreps\begin{document}$$\displaystyle f^+(i,j) = \frac{n_{\text{success}}(i,j)}{n_{\text{reps}}}$$\end{document}g+(i,j)=⟨e−λτ(i,j)|success⟩\begin{document}$$\displaystyle g^+(i,j)=\langle e^{-\lambda \tau(i,j)}\,|\, \text{success}\rangle$$\end{document}τminτ(i,j)=⟨τmin(i,j)τ(i,j)⟩\begin{document}$$\displaystyle \frac{\tau_{\text{min}}}{\tau}(i,j)= \langle \frac{\tau_{\text{min}}(i, j)}{\tau(i, j)}\rangle$$\end{document}

where nreps\begin{document}$ n_{\text{reps}}$\end{document} is the number of test trajectories from each admissible location, and we use nreps=10\begin{document}$ n_{\text{reps}}=10$\end{document}. We then compute statistics of the performance metrics over the Dinit\begin{document}$ D_{\text{init}}$\end{document} initial positions and report the average (⟨⋅⟩\begin{document}$ \langle\cdot\rangle$\end{document}) and standard deviation (std). Note that both the backtracking and Brownian strategies have stochastic steps; for these strategies, f+(i,j)\begin{document}$f^{+}(i,j)$\end{document} denotes the average success fraction computed at each position over 10 repetitions.