We compared the search performance in response to different types of stimuli by presenting wind and visual stimuli in addition to olfactory stimuli in all conditions. We first measured the behavior when wind or visual stimuli were provided in addition to an odor stimulus (Group 1). The blue and black lines in Figure 2B—F are the trajectories of successful and unsuccessful searches, respectively. The amplitude of the heading angle histogram (Figure 2B—F; displayed in polar coordinates) indicates the frequency. Judging from the trajectory and heading angle histogram, the trajectories in cond. 1 (only odor) show that although the silkmoth moved toward the upwind direction with a high frequency, it deviated from the range where the odor had a high probability of reaching (high odor area), and the search failed (Figure 2B). By presenting wind (cond. 2) and visual (cond. 3) information in addition to odor, the silkmoth reduced the deviations from the high odor area, which qualitatively indicates that the odor-source search behavior was modulated by other sensory information (Figure 2CD). In the case of wind input, even if there was a deviation, modulation was elicited to change the heading angle to the upwind direction. In the case of visual input, although there was no trajectory deviating away from the high-odor area, because the frequency of the heading angle histogram was also high at angles other than 0∘, the posture control makes the search behavior more careful. According to the heading angle histogram of cond. 4, which was presented from the direction opposite to the direction in which the wind actually blew, there was also a peak in the downwind direction; This indicates that the behavioral modulation was affected by the wind information (Figure 2E). Even when the visual stimulus direction was reversed, the peak in the upwind direction was not extreme, and a peak was generated in the crosswind direction (Figure 2F). By presenting the visual information in the opposite direction (the visual information was input as if it were rotating in the opposite direction), the illusion is that it was not rotating correctly, and more rotational behavior was induced. In order to quantitatively evaluate how similar these search trajectories were for each condition, we performed a two-dimensional histogramization of the trajectories and a calculation of similarity.

We created a migration probability map in order to visualize the effects of differences in environmental conditions on the behavioral trajectory (Figure 3A—E). To quantitatively evaluate the similarity between the migration probability maps, we calculated the earth mover’s distance (EMD) (Rubner et al., 1997). EMD is an index that calculates whether the distributions of two histograms are similar. The smaller the EMD value, the more similar the two histograms; the larger the value, the less similar they are. Figure 4F shows the result of calculating EMD based on cond. 1, in which only the odor stimulus was presented. The silkmoths’ trajectories in conds. 3 and 5 (odor + vision) were very similar to the trajectory in cond. 1 (only odor), suggesting that vision did not significantly affect search behavior. In addition, the EMD value when wind stimulus was presented in addition to odor was 10, suggesting that the addition of wind stimulus significantly affected search behavior. We illustrated the navigation success rate and search time (yellow background in Figure 4AB). Additionally, we calculated the search success rate per unit time (SPT) as a measure of the relationship between the search success rate and search time (yellow background in Figure 4C). Higher SPT values represent better search performance. The success rate was most improved when the wind was presented correctly (cond. 2) compared to when only the odor stimulus was provided (cond. 1). The success rate was lower under the condition where the wind was presented from the direction opposite to the direction detected in the virtual environment (cond. 4), compared with the condition where only the odor stimulus was presented (cond. 1). Moreover, SPT in conds 3 and 5 (in which the direction of the visual stimulus was changed without presenting the wind stimulus) was almost the same (0.411 vs 0.409), suggesting that the visual stimulus had little effect on search performance. Therefore, the success rate of navigation tends to improve by correctly presenting a wind stimulus in addition to the odor.

We found that when three types of sensory stimuli were presented (Group 2), the success rate changed significantly compared to when two types of sensory stimuli were presented (red background in Figure 4). Focusing on cond. i, the SPT values were also significantly different, suggesting that olfactory navigation can be performed more accurately and efficiently by presenting all sensory stimuli. Additionally, under the condition that the wind stimulus is presented from the direction opposite to the direction received in the virtual environment, the search success rate is significantly reduced, and it is clear that information on wind direction is an important factor. We therefore hypothesized that the silkmoth performs efficient female searches using all sensory inputs: odor, wind, and vision.

In the previous section, it was found that information on wind direction, in addition to odor, contributed to improving the success rate in searching for the odor source. Here, we analyze in detail how behavior is modulated by visual and wind information using three experimental conditions: a forward condition (cond. i), an inverse condition for wind direction information (cond. ii), and an inverse condition for visual stimuli (cond. iii).

We first analyzed the effect of information on wind direction on odor-source search behavior. Specifically, we analyzed whether the movement speed changed between cond. i and cond. ii because the wind causes behavioral modulation. We calculated the movement speed change with respect to the odor detection frequency (Hz) because it has been reported that the odor detection frequency is related to the distance from the odor source (Kikas et al., 2001; Figure 5AB). Here, odor detection frequency is defined as the frequency at which the silkmoth receives odors per unit of time. The odor detection frequency at each point of the odor field in this study is listed in the Supplementary Materials (Figure 1—figure supplement 1). The velocity distribution at each odor detection frequency is represented by a box plot. The red box plots in each figure represent the maximum values, and the circled dots in each box plot are the average values. The dotted line in Figure 5A/B is the result of performing a least-square approximation on the average value. The result of this least-square approximation is utilized in the mathematical modeling. According to the results of the speed change of cond. i, the translational speed increases monotonically up to the odor detection frequency of 0.7 Hz and decreases after 0.7 Hz. Moreover, the angular velocity increases monotonically up to 0.4 Hz, and then decreases. Considering that the pheromone release frequency of the silk moth female is about 0.8 Hz, when the wind is received from the same direction as the odor, the male silkmoth may actively move until it approaches the pheromone release frequency. At higher frequencies ( > 0.8 Hz), the male silkmoth reduces its movement speed to facilitate a mate search because there is a possibility that the female is nearby.

According to the results of the translational speed of cond. ii, the moth always moves at a constant speed regardless of the odor detection frequency. We also found that the frequency at which the maximum value occurs for the angular velocity was shifted to 0.2 Hz. This may be because the peak of the angular velocity shifts toward a lower odor detection frequency, and if the odor and wind direction do not match, the silkmoth rotates slowly to carefully search for the odor. This is probably because if the odor and wind direction do not match, the odor is carefully searched for by slowing down the overall movement speed.

Next, we investigated the effects of visual stimuli on odor-source search behavior. Section 2.1 suggests that visual stimuli do not directly contribute to search performance. However, many flying insects with compound eyes use visual information for their own postural control (Dyhr and Higgins, 2010; Dyhr et al., 2013). Because the silkmoth, which may retain some vestiges of flying insects (Kanzaki, 1998; Shigaki et al., 2016), also changes its rotational behavior in response to a visual stimulus presented as an optical flow (Pansopha et al., 2014), it is possible that the silkmoth utilizes the visual information as the input amount for postural control. To test this hypothesis, we compared the angular velocities of silkmoths in cond. i (all stimuli in the forward direction) to those in cond. iii (visual stimuli presented in the inverse direction). Figure 5C shows a histogram of the left and right angular velocities under each experimental condition. The red color in Figure 5C indicates the angular velocity when rotating counterclockwise, and the blue color shows the angular velocity when rotating clockwise. The position of the vertical bar above the histogram represents the average angular velocity, and the length of the horizontal bar represents the standard deviation. Although the average angular velocities of the left and right rotations are the same in cond. i (visual stimuli presented correctly), the angular velocities of the left and right rotations differed when the visual stimuli were presented in the opposite direction to reality (Wilcoxon rank sum statistical test, p < 0.05). These results indicate that silkmoths, like flying insects, use visual stimuli for postural control.

We found that the silkmoth modulated its behavior based on whether or not the direction of odor and wind detection coincided.

Here, we investigated how behavioral modulation informed by behavioral experiments using a VR system contributes to the odor-source search. A silkmoth moves by walking on a two-dimensional plane with six legs, but it does not move in a lateral direction. Therefore, we assumed that it has non-holonomic constraints and constructed our model to output straight-ahead and rotational movements. Previous studies have proposed a silkmoth search model called the surge-zigzagging algorithm (Ryohei et al., 1992; Shigaki et al., 2020), which makes action decisions using only olfactory information. In this study, we updated the surge-zigzagging algorithm to incorporate wind direction information. For convenience, we called this algorithm MiM2 (the multisensory input-based motor modulation) algorithm. A block diagram of the constructed MiM2 algorithm is presented in Figure 6A. The model has a straight-ahead speed (v) controller (Equation (1)) and an angular velocity () controller (Equation 2), and each controller changes output depending on the amount of sensory input. The detailed equations for each controller are shown below:(1)v(t)=Kv(f)×exp⁡(−(t−td))1+exp⁡(γ((t−td)−β))(2)ω(t)=ω0×Rd(0)1+exp⁡(γ((t−td)−β))+Kω(f)×Rd(N)1+exp⁡(−γ((t−td)−β))(3)Kv/ω(f)=av/ω×f+bv/ω

The free parameters were set to γ=1000, β=0.50, and ω0=0.57. Here, f denotes the odor detection frequency. Moreover, K⁢(f) is a linear function that varies with the direction of odor and wind detection. The parameters a and b of K⁢(f) are shown in Table 1. These were obtained by approximating the results of the behavioral experiment using the least-squares method (dotted line in Figure 5A/B). In addition, t_d, Rd, and N represent the odor detection timing, the odor detection direction, and the number of turn motions, respectively; Rd⁢(0) represents a straight-ahead motion state and takes a value of –1 when the left antenna detects and one when the right antenna detects; if both antennae detect, a value of –1 or one is randomly selected; N increases according to Equation (4) up to four, but does not take a value of four or more because the number of zigzagging motions is approximately three (Ryohei et al., 1992). When it receives an odor stimulus according to Equations (1–3), it generates a straight motion for 0.5 s and then makes a turn motion.(4)N(t)=[0.0116(t−td)3−0.199(t−td)2+1.1971(t−td)+0.4482]

By passing through the directional discriminator and frequency counter, the odor is converted into information such as whether it was detected by the left or right antennae and how much odor it was exposed to. The wind is converted into wind direction information. A comparator is used to determine whether the direction of odor detection and the direction of the wind are the same, and the results are input to the straight-ahead speed and angular velocity controllers, which also receive odor detection frequency. Visual information controls the angular velocity of the left and right rotational movements, following which olfactory behavior takes place. From this information, the movement speed output is calculated, and the behavior is generated. We compared the MiM2 algorithm to the previous surge-zigzagging algorithm and the casting algorithm, which uses wind information for searching in a moth-inspired algorithm (Li et al., 2016).

The simulation environment employed a virtual environment similar to that used in the behavioral experiments in the VR system (see Figure 6—video 1). For each algorithm, we performed 1000 odor-source search experiments to evaluate search success rate and trajectory. In the simulation experiment, we set two scenarios to verify each algorithm. One was the same initial position (x, y) = (300, 0) [mm] as in the biological experiment, and the other was the initial position moved ± 100 mm in the crosswind direction (x, y) = (300, ± 100) [mm]. Because the latter was near the edge of the area where the odor reaches, deviation behavior possibly occur frequently. Figure 6B and C show the search success rate and the search time, respectively. The MiM2 algorithm showed a higher search success rate than the other algorithms, regardless of the initial position (Fisher’s exact test, p < 0.01). The casting algorithm showed the shortest search time (Steel-Dwass test, p < 0.01) because it is an algorithm that actively moves upwind using odor detection. However, if the heading angle when moving upwind was incorrect, the search success rate of this algorithm was the lowest of the three, because there was a high possibility that the agent would have moved in the wrong direction. Next, we focused on the trajectory and heading angle change shown in Figure 6D—L. The blue and black lines in Figure 6D—I are the trajectories of successful and unsuccessful searches, respectively. Moreover, the amplitude of the heading angle histogram (Figure 6J—L; displayed in polar coordinates) indicates the frequency. For the readability of the trajectory and heading angle histogram, we randomly selected 100 trials of the 1000 replicates and plotted them. Regardless of the initial position, the search trajectory indicates that the MiM2 algorithm always places the searching agent in the middle of the odor distribution, whereas the conventional surge-zigzagging and surge-casting algorithms tend to place the searching agents near the edges of the odor. According to the heading angle histogram, the MiM2 algorithm resembles the actual silkmoth (Figure 2), but the other algorithms produced completely different histograms.

In order to quantitatively evaluate the trajectory of 1,000 simulations, we visualized the trajectory data (Figure 7A—F). The black line in Figure 7A—F shows the range that the odor has a high probability of reaching. Moreover, we evaluated the differences in trajectory between algorithms quantitatively by calculating their EDMs (Figure 7H). The comparison of these EDM values suggests that the MiM2 algorithm most closely resembles the actual silkmoth movements (Figure 7G) and was the closest to the results of behavioral experiments of the silkmoths while free walking. Thus, the MiM2 algorithm can simulate search behavior similar to that of real silkmoths by modulating the movement speed based on wind information in addition to odor, and equalizes the left-right angular velocities based on visual information.

All animals, including insects, need to navigate for survival and reproduction. It is especially important to navigate efficiently in harsh environments and in situations in which there are many competitors. Previous studies have investigated the homing behavior of a desert ant (Cataglyphis) (Wehner, 2003), the pheromone source localization behavior of a male silkmoth (Bombyx mori) (Obara, 1979), and the sound source localization behavior of female crickets (Gryllus campestris L.) (Schmitz et al., 1982). These studies have contributed to our understanding of the sensory-motor integration mechanism that converts sensory input into motor output – an important function of the neural system. However, the insects in these studies may not have displayed navigation behavior as they would have under natural conditions, because stimuli were controlled and presented in a fixed amount and at regular intervals in the behavioral experiments. In addition, the nature of behavioral change in a large-scale space and over a long period is relatively unknown because the behavioral experiments were carried out in a relatively small space and over a relatively short time. Because of this, multimodal virtual reality (VR) systems have attracted a great deal of attention (Kaushik et al., 2020; Naik et al., 2020). The advantage of using a multimodal VR system is that not only can behavior be measured on a large spatiotemporal scale, but the transmission of stimuli can be precisely controlled, and the behavioral output can be precisely measured. Biological data describing the relationship between sensory input and motor output are useful not only for clarifying biological functions, but also for the field of robotics, because these data play a very important role in modeling insect systems. Because the multimodal VR system in previous studies was developed for visually-dominated navigation, it was difficult to measure olfactory-dominated navigation, which was the focus of our research. Therefore, we developed a novel multimodal VR system that allowed us to measure changes in navigational behavior when the olfactory, visual, and wind directions of the silkmoth were modified. Conventionally, previous studies on the silkmoth have shown that (1) a ‘mating dance’ was elicited in response to sexual pheromones (Obara, 1979), (2) the condition in which visual stimuli was presented immediately after the reception of sex pheromones influenced the subsequent rotational behavior (Pansopha et al., 2014), and (3) behavioral inhibition occurred in response to frontal winds (Shigaki et al., 2019). However, the integration of the above phenomena during navigation has not been investigated. By presenting three types of sensory information simultaneously as well as continuously, we were able to clarify the roles of each type of sensory information in navigation. In the behavioral analysis of a flying moth, it was found that when the moth detected the odor, it moved in an upwind direction (Willis et al., 2011; Willis and Arbas, 1991); and wind information is important for behavior determination. Even in the case of the silkmoth, which is a walking insect, the success rate is higher in the condition that the wind information is correctly provided (cond. 2) than in a non-correct condition (cond. 4) or in a no-wind condition (cond. 1) (Figure 4A). However, it does not mean that the silkmoth cannot reach the odor source when wind information is not correctly provided, and the search success rate was about the same as in the no-wind condition (approximately 60%). The wind information is therefore important, but if the wind information is unreliable, the priority of the odor information may be increased and the silkmoth performs the navigation. When the phenomena obtained by the VR experiment were mathematically modeled and the performance was confirmed by the simulation experiment, not only the original search trajectory of the silkmoth could be reproduced, but the search performance was improved, compared to the conventional algorithms.

Organisms of all size scales rely on the ability to locate an odor source in space. The chemotaxis of bacteria and a nematode have been determined by measuring and analyzing the relationship between the chemical stimulus input and behavioral output under a controlled environment (Berg, 2008; Lockery, 2011). These studies found that as the concentration of chemical stimuli increased, the probability of rotating behavior decreased linearly. In other words, bacteria or nematode chemotaxis followed an odor gradient, which allowed them to locate the odor source. In the space where bacteria and the nematode exist, chemotaxis is effective in part because the odor field of the environment does not change significantly due to wind. In the case of larger size animals, the odor field of the environment is quite complex and it is difficult to reach the odor source by following the gradient alone. Odor fields can be complex because odor molecules are transported by airflow and mixed with other molecules at their destinations, forming complex structures (Crimaldi and Koseff, 2001; Murlis et al., 1992). In addition, the odor molecules themselves are discrete in space, and do not carry information about the source of the odor. However, a study using a sensor array to measure the arrival of odors carried by the wind revealed that the odors emitted from the source arrive periodically (Kikas et al., 2001; Murlis et al., 2000). Moreover, the periodicity is correlated to some extent with the distance from the source; the closer the source, the shorter the cycle, and the farther the source, the longer the cycle of arrival. Neither the concentration of the odor nor the number of exposures played an important role in the search process in Drosophila, but the ‘tempo’ at which flies encountered an odor was an important factor in the decision-making process (Celani, 2020; Demir et al., 2020). If we assume that the ‘tempo’ is the rate of odor detection per unit time, it is related to the cycle in which the odor arrives. We hypothesized that the silkmoth, like flying insects such as Drosophila, modulates its behavior based on the ‘tempo’ of the odor, therefore we included odor frequency in our analyses.

When the direction of the wind and the odor coincided, movement speed peaked when the odor frequency was 0.7—0.8, which is similar to the frequency at which a female silkmoth releases sex pheromones (0.79 ± 0.05 Hz) (Fujiwara et al., 2014). Based on these findings, we hypothesized that if the male silkmoth detects wind and odor from the same direction, it correctly moves in the direction of the female and actively searches the field until it reaches a location where the sex pheromone release frequency approximates that of the female. When the frequency exceeds 0.8 Hz, the male seems to be in the vicinity of the female and therefore increases the spatiotemporal resolution of the search in order to locate the female and prepare for the transition to mating behavior. This might correspond to the silkmoth switching to the odor source declaration algorithm in olfactory navigation. The in vivo data obtained in the current study are consistent with those of earlier research demonstrating that an algorithm that shortens the travel distance of the surge toward the end of the search improves the search performance of the robot (Shigaki et al., 2018).

In an earlier study investigating the direction of wind and odor in the environment, the direction of wind and odor were the same in the open-field experiment (no obstacles) (Murlis et al., 2000). However, the direction of wind and odor is not always the same in a complex environment, such as a forest with many trees (Murlis et al., 2000). Our data showed that in situations where the wind and odor direction do not match, the silkmoth always moves at the same rate, given any odor detection frequency. This may be a chemical tracking strategy to avoid leaving the odor range by moving at a speed lower than normal speed and suggests that the silkmoth is estimating the degree of environmental turbulence and modulating its behavior based on the odor and wind direction information. This strategy may be applicable to an engineering search system that switches the search strategy according to the environmental conditions.

Odor sensing plays an important role in situations in which a visual search is difficult (e.g. a space filled with darkness and thick smoke). However, because the technology for the development of an odor sensor is lagging behind that of other sensory sensors (e.g. cameras, microphones), robot olfactory research is still a developing field. Currently, dogs play the role of odor sensors, but due to the high cost of training dogs and the deterioration of their odor sense with physical condition and age, engineering solutions using autonomous mobile robots for searching are needed. Conventional robot olfactory research has emphasized the development of motion planning (a search algorithm). The search algorithm is roughly divided into two fields: one is a bio-inspired algorithm that imitates the search behavior of living organisms, and the other is a search that estimates the position of the odor source using a statistical method (statistical algorithm). The bio-inspired algorithm, which mimics the behavior of organisms that can search in real time, is superior to the statistical algorithm in terms of robot implementation (Russell et al., 2003; Lochmatter et al., 2008). However, behavior patterns and movement speeds in these bio-inspired algorithms are always constant, regardless of the environmental conditions. Accordingly, there is a problem that the original search performance of living things cannot be replicated. For this reason, different kinds of research have been carried out more recently and applied to the bio-inspired algorithm, such as information-theoretic analysis of the trajectory of an insect (Hernandez-Reyes et al., 2021), extraction of adaptability from neuroethology data by fuzzy inference (Shigaki et al., 2020), and the acquisition of behavioral switching indices in response to environmental changes by measuring insect behavior while visualizing airflow (Demir et al., 2020). In the current study, we found that behavioral modulation occurred based on the relationship between the direction of the odor arrival and the wind, and we used our data to reproduce a behavioral trajectory that was not only better than that of the conventional bio-inspired algorithm but was also more similar to the search behavior of the actual silkmoth. Therefore, it is clear that a probabilistic and time-varying behavior modulation mechanism has a better search performance than a time-invariant search algorithm.

Silkmoths (Bombyx mori; Lepidoptera: Bombycidae) were purchased from Ehine Sansyu Co., Japan. Adult male moths were cooled to 16°C one day after eclosion to reduce their activity and were tested within 2-7 d of eclosion. Before the experiments, the moths were kept at room temperature (25-28°C) for at least 10 min.

To generate a virtual odor field closely mimicking reality, smoke was emitted into the actual environment. The diffusion of the smoke in a two-dimensional plane was recorded and implemented into the simulator using image processing. The smoke visualization experiment was conducted in an approximately 2.5 × 0.8 m area in a darkroom. First, the smoke was visualized using a smoke generator (particle diameter: < 10 μm) and a laser sheet (1 mm in thickness) (Figure 1—figure supplement 1A). The smoke emitted into the room gleamed when it hit the irradiated laser sheet, and the distribution of smoke was recorded using a high-sensitivity camera. Note that the behavior of smoke reflected by the laser could be observed in two dimensions because the laser was in the form of a sheet. In this experiment, we carried out a visualization experiment in a darkroom because it is important to accurately record the light reflected by the smoke. This method was based on particle image velocimetry (PIV), which measures the velocity of fluid flow in space by visualizing particles and analyzing their movements. PIV accuracy is improved by scattering a large amount of smoke (particles) in space because it focuses on accurately tracking each particle; however, our method did not require following each particle. Instead, our method used a mass of smoke as an odor plume, and we observed how smoke floated and was distributed in space. In the visualization experiment, the flow meter (1.0 L/min) and solenoid valve (1 Hz) were controlled under the same release conditions as in the behavioral experiment. Our method required less smoke than PIV, and the smoke was thinner and more difficult to visualize. For this reason, it was necessary to increase the sensitivity of the camera; however, noise increased accordingly. In PIV, it is not necessary to distinguish between smoke particles and dust because only the movement of particles in space is important, but our method required removing other particles in order to measure only the smoke position as the plume. Hence, we applied image processing to the smoke image. Luminance values can have a large range because the intensity of smoke varies with airflow and time. In addition, it is difficult to extract only smoke with simple thresholding because the noise caused by dust has the same luminance as does smoke. Therefore, we adopted a method that focuses on connected components to remove noise while maintaining the shape of the smoke. In groupings based on connected components, two objects were considered to be connected if adjacent pixels in a binary image take a value of 1. Because smoke exists as a mass, it can be inferred that it has an area above a certain level when divided into connected components. For this reason, we removed the pixels whose connected components were not in adjacent pixels and whose area did not exceed a certain level. These processes were performed with source code using OpenCV.

The behavioral experiment was carried out using a homemade virtual reality system (a photograph of the actual device is shown in Figure 1A). The traditional tethered measurement system was used for behavior measurement, and a stimulator that presented odor, visual, and wind direction information was installed around the tethered measurement system (Figure 1—figure supplement 1A). The details of each stimulator were as follows:

A migration pathway ratio map was used to statistically process the trajectories of all trials and denoted the points where the quadcopter frequently passed through the experimental field. We created the migration pathway ratio map Cn⁢(x,y) based on the rule given by Equation (5). Here, n represents the trial number, and the size of the grid cell was 0.1 × 0.1 cm.(5)Cn(x,y)={1:The silkmoth passed through the coordinates (x,y).0:The silkmoth did not pass through the coordinates (x,y).

We applied the equation to all trial trajectories and summarized them, and then calculated the trial average based on Equation (6) to create the migration pathway ratio map P⁢(x,y). The migration pathway ratio map was created using MATLAB (2020a, MathWorks, MA, USA).(6)P(x,y)=∑n=1NCn(x,y)N

For all data analyses, R version 4.0.3 was used (R Core Team). All earth mover’s distance calculations were performed using the Python 3.7 language.

The details of the three algorithms for which simulation experiments were performed are described here: