Flies collect information 4 times faster than we do, our vain fly-swatting efforts appearing in slow motion to that fly on the salad, albeit at lower resolution.
“Although they operate faster, and at lower resolution, they’re still able to do a huge variety of tasks much better than anything that our current systems can,” says Professor Russell Brinkworth of Flinders University, Adelaide.
“Understanding how they do it can help us to build better machines, that use insect features to sense and more efficiently navigate natural and built environments,” he says.
We’ve all seen insect eyes portrayed in movies (The Fly, Monsters Vs Aliens). Bizarrely beautiful, almost other-worldly, such compound eyes are the oldest and most dominant vision system on Earth, used by 75% of all animals, including 10 million species of insects. And Australian robotics researchers are tapping this well of visual expertise to make smarter machines. Researchers such as Prof Brinkworth, and collaborators Prof. Matt Garratt, Dr Sridhar Ravi of UNSW Canberra and Prof. Mandyam Srinivasan of UQ.
Biological eyes adapt where cameras can’t
We can read black writing on white paper in the sun and in the shade because our eyes adapt — something we take for granted. As can insects — not read, as far as we know — but certainly perceive contrast under different lighting conditions. “All biological eyes can do this” says Brinkworth, but to a camera, writing disappears in the glare or into the dark.”
Compound eyes are the oldest and most dominant vision system on Earth.
Until now, even small insects outshine most cameras in these tasks, which is why Brinkworth and his students are using insect-eye models to make camera systems that recognise subtle differences and small contrast changes, allowing us to decipher our complex environments. That’s the key, not trying to capture the perfect image, says Brinkworth.
Hoverfly eyes, and what they do with them, focus Brinkworth’s attention. Adult hoverflies are nectar and pollen feeders, and pollinators. Small and often colourful, you’ll see them floating through gardens around the world. Their eyes may be more than 20% of their body’s mass — imagine having eyes the size of watermelons!
Consummate hoverers and acrobats, these little flies understand their world through optic flow — the picture of the hoverfly’s world moving across its eye as it passes through it. This is not detection per se, but the flow of information past the eye, says Brinkworth. Relative velocity and position — and time to impact — are critical.
“Optic flow is effectively the ratio of speed to distance,” says Garratt. The closer the object, the faster it appears to move, relative to you. Imagine you travelling to Byron Bay in your car. Look out the window to the side — that kangaroo grazing on the side of the road is ‘approaching’ faster than the semi-trailer raising dust on the side-road in the distance. Hoverflies estimate where objects are through optic flow — relatively fast or getting faster means less time to impact.
We’re using the animals to inform robotics and using the robotics to better understand animals.
Dr Sridhar Ravi
Brinkworth studies hoverfly vision to make better sensors for detecting fine-scale changes in the environment, such as unauthorised drones at airports and military sites. Trials at Woomera in South Australia, showed prototypes could “spot incoming objects on a direct collision course coming immediately over the horizon directly towards the camera, when they’re smaller than a single pixel,” says Brinkworth. “From the ground or from a drone,” he adds.
Achieved by reverse-engineering hoverfly abilities, they built cameras able to detect objects camouflaged against messy backgrounds — “slight, subtle lighting and contrast differences against different backgrounds and combinations.”
Small contrast differences were amplified, and movement and lighting changes rapidly detected. Essentially, they were able to separate the signal they wanted, from the noise they didn’t.
Out of the corner of your eye
Your peripheral vision is unfocussed, like an insect’s, but was enough to save your life when you started to cross that road and suddenly sensed a car coming, out of the ‘corner’ of your eye — you stepped back without thinking or focussing. Brinkworth’s technology switches from this low resolution, but really fast, hoverfly-like vision, to focussed (‘foveal’ in biology) mode, to get as clear a picture of the object as possible — using acoustic or optical (i.e. camera) sensors (including infra-red) or both. Camera frame rates are 50-100 frames per second.
Honeybees guiding drone development
Similarly, collaborators Professor Matt Garratt, Dr Sridhar Ravi of UNSW Canberra and Professor Mandyam Srinivasan of UQ are using the honey bee’s ability to visually navigate complex environments to develop autonomous miniature drones for use in precision agriculture, search and rescue, wildlife monitoring and war zones.
“Honey bees are excellent long-distance flyers and can be trained for experiments”, says Ravi. “Studying their responses to environmental manipulations allows us to better understand their vision systems.”
Applying honey bee skills to miniature drones involves working out “how bees solve the problem of navigating in completely new environments, said Ravi, then: “What does that look like from a sensor standpoint and how do the algorithms work”. “This could be applied to a whole suite of other platforms, not just miniature drones,” he said.
Flies collect information 4 times faster than we do.
The project follows more than 20 years of research, says Garratt. From getting drones to take off, hover and land using visual sensors alone — no lasers or GPS — to now moving forward through an obstacle course. Current hypotheses are tested using a 2kg eight-rotor drone, paired with honey bee experiments. Each step builds on the last and gets the team closer to its goal. “It’s two-way communication — we’re using the animals to inform robotics and using the robotics to better understand animals. So, we’re hoping for that symbiotic transfer of knowledge,” says Ravi
The miniature drone is fitted with a panoramic (360o) vision system to provide the broad field of view so important to insects’ flight control. Pan-tilt capability provides stability and a second source of optic flow, enabling the drone to move in a straight line, up and down and left and right — providing the versatility and stability of movement needed to take on the results of the honey bee trials.
GPS-free
The combination of miniaturisation and desired applications provides many challenges, including navigation. GPS is ubiquitous but the interest is in environments where GPS doesn’t work well — indoors or in a forest — or in war zones, where it can be jammed, says Garratt. Lasers can also be detected, are heavy and emit radiation. So, the miniature drone’s navigation must rely on a passive, non-GPS, radiation-free system which leaves optic flow. Interestingly, NASA’s Mars helicopter ‘Ingenuity’ uses such vision sensors for stabilisation, says Ravi.
Once the test drone flies as it should — relying entirely on vision sensors — the miniaturisation challenge will include the panoramic imaging and pan-tilt systems, with electronics possibly taking the place of the latter physical system.