Bio-inspired network structures for next-generation AI

Increasing AI’s ability to tackle complex challenges with greater accuracy and energy efficiency is not simply a matter of adding more computing power. Subtle details in the networking of the computing elements can have a significant impact on AI performance.

The architecture, or wiring, of the computing elements in an AI is inspired by how neurons form circuits to process information and learn. However, a key aspect of neural network structure has so far been overlooked in AI design, as a KAUST-led team has shown^[1][2].

Jesper Tegnér and his team at KAUST –  in collaboration with an international team including the AI technology company NVIDIA –  made the discovery while examining the network architecture of an AI solving a balance task. “We focused on ‘network motifs’, which often form the fundamental building blocks of large, complex networks,” explains Haoling Zhang, a Ph.D. student in Tegnér’s lab.

Network motifs combine to form complete, complex networks, just as words come together to create language. “Network motifs have been widely studied in biology and social science, but surprisingly little in the AI literature,” Zhang says.

The concept had been so overlooked in the AI community that no methods existed to study it. “To make progress, we had to build from scratch our own analytical methods, combining neuroevolution with machine learning,” Zhang adds.

The team focused on some of the simplest possible network motifs, consisting of three nodes – two inputs and one output – connected to form a triangle. Depending on how they are wired and how information flows through them, different types of three-node network motifs can form. Two of the most important are known as ‘coherent’ and ‘incoherent’ loops. “The abundance of incoherent motifs in natural systems, where both activation and repression occur on the same node in small circuits, has remained intriguing,” Tegnér explains.

“Initially, we didn’t suspect that these two loop types would have a significant impact on AI performance,” Zhang says. But, as the team progressed from testing individual network motifs to assessing more complex networks in various tasks, a consistent and notable difference emerged between the two loop types.

Neural networks dominated by coherent loops tended to quickly focus on feature-rich ‘high-gradient’ patterns encoded in the learning dataset. In contrast, networks dominated by incoherent loops explored patterns without any early preference for specific features.

In real-world situations, random or irrelevant noise in datasets is almost impossible to eliminate. During the early training phase of machine learning, noise can easily mislead a neural network because it cannot yet distinguish meaningful from non-meaningful signals. By rapidly narrowing in on high-gradient regions of the dataset, coherent loops were more susceptible to being distracted by noise. “This could slow learning and distort what is learned,” explains Zhang.

Networks built from incoherent loops, in contrast, learned in a richer, more balanced way. “In noisy datasets, networks with more incoherent loops stayed noticeably more stable and got less confused,” Zhang says.

“We found that these biological connectivity patterns had functional significance in an AI system,” Tegnér says. The team is now exploring various nature-inspired ways to develop high-performing, creative, energy-efficient next-generation AI systems.