Scientists from the Swiss Federal Institute of Technology Lausanne (EPFL) have released an open-source AI model that can read animal behavior. Based on deep learning, the AI, called SuperAnimal, studies the motion of an animal’s different body (such as the eyes, mouth, hands, and feet) and performs accurate behavioral analysis.  

SuperAnimal is a foundational model, meaning it can be used to build more specialized automated animal behavior analysis tools. For instance, a conservationist can make his own version of SuperAnimal to track endangered animals. A veterinary doctor can train and develop a SuperAnimal model to monitor the health of animals in their hospital. 

In 2018, the EPFL team launched DeepLabCut, a powerful pose-estimation software capable of achieving human-level precision in identifying and tracking animal movements. SuperAnimal is the upgraded version of this earlier work.

“DeepLabCut was the first animal pose estimation tool that allowed users to build customized neural networks. It automated and improved the accuracy of behavioral studies. Our new SuperAnimal models, which are integrated within DeepLabCut, can track many, many species, and is the largest collective animal dataset plus trained models to be released to date,” Mackenzie Mathis, one of the researchers and a neuroscientist at EPFL, told ZME Science.

How does SuperAnimal work?

SuperAnimal is trained using 85,000 images and can be used to study the behavior of over 45 species including many rodents and four-legged animals. What’s more surprising is that this tool can also analyze the behavior of a fictional creature from a photo or video. 

However, the fictitious creature, such as a unicorn, should have some of its body features similar to that of real animals. For instance, Mathis and her team produced an AI-generated image of a wolpertinger (an animal mentioned in German folklore supposedly born from a hare and a deer) and used SuperAnimal to analyze its behavior.    

The AI model treats the body of the animal as a collection of numerous “keypoints” and uses these points to map movements. While other AIs used for behavior analysis require human intervention to label key points on animals, SuperAnimal does it automatically.

“The current pipeline requires human effort to identify keypoints on each animal, creating a training set. This results in duplicated labeling efforts and inconsistent semantic labels, complicating the training of large foundation models. Our new method standardizes this process and makes labeling 10 to 100 times more efficient than current tools,” Mathis said.

Once all the keypoints are identified and labeled, SuperAnimal can track the slightest change in an animal’s posture, facial expressions, body movements, and motion. This eventually allows the AI model to learn which particular motion or action will lead to specific behaviors.

“Behavior – simply defined – is considered the changes in the animal’s motion over time. Namely, how they move is deeply connected to their internal goals and motivations, and in neuroscience, it’s critical to measure this behavior to link to neural data,” Mathis told ZME Science.

SuperAnimal can benefit both humans and animals

DeepLabCut has been downloaded and installed 700,000 times. Anyone can use it and this open-source software has already helped many researchers, organizations, and individuals in studying animal behavior in different environments. 

“SuperAnimal can further revolutionize animal behavior studies, enhance conservation efforts, improve animal welfare, and advance fields like neuroscience and ethology,” Mathis said.

For instance, this AI tool can allow farmers to monitor and improve the health of their livestock, enable scientists to effectively study the behavior of lab animals during experiments, and help people better understand how their pets feel in a particular situation. It is also likely to make it easy for biologists to decode the interaction between different animals.

“We will also leverage these models in natural language interfaces to build even more accessible and next-generation tools,” Mathis added.

The researchers plan plan to further expand the capabilities of SuperAnimal so that it can be used to study the behavior of fish, insects, and birds in the future.

The study is published in the journal Nature Communications.