Anime is a style of animation that originated in Japan and is still heavily centered around Japan. The term “anime” is derived from the English word “animation,” and in Japan it is used to refer to any kind of animation. However, outside of Japan, it is typically used to refer to a specific kind of animation.

The art style in anime can vary greatly, depending on the artist, genre, and intended audience. However, there are usually some common stylistic elements. Characters are vibrant and colorful and generally drawn in great detail. Anime characters often have exaggerated features. Large, expressive eyes are a staple. Flamboyant hair colors and hairstyles are also common, and clothing is often unique, ranging from traditional Japanese to futuristic or extravagant.

Drawing anime characters can also be very challenging, and animators are notoriously underpaid. Now, a new tool is about to enter the stage — one that can draw characters much more easily — though it’s not clear what its effects will be on the industry.

AI, meet anime

For the past year or so, generative AIs have roared through the world. There’s the unavoidable ChatGPT that’s impressively good at writing text and about half a dozen image-generating AIs. It was only natural that anime would get some attention too.

But the new tool doesn’t intend to create new images. It aims to augment and complement human ability.

“This paper focuses on how artificial intelligence (AI) can be used to assist general users in the creation of professional portraits, that
is, consistently converting rough sketches into high-quality anime portraits during their sketching process,” the researchers write in the study.

The research team features scientists from Japan Advanced Institute of Science and Technology (JAIST) and Waseda University in Japan, led by Zhengyu Huang. The researchers looked at one specific task: how to transform rough sketches into concrete anime portraits.

The training arc

Many image-generative AIs are good at producing abstract imagery. Most commonly, they use generative adversarial networks (or GAN). GANs are a type of machine learning model designed to create new data samples that are similar to a given set of input data. In the context of images, GANs can generate new images that look like they were drawn from the same distribution as a provided set of training images. The ‘adversarial’ part comes from the two main parts of the model: the generator and the discriminator.

1. Generator: This network takes a random noise vector as input and transforms it into an image. At the start of training, the images it produces are random, but over time the generator learns to create images that look like the ones in the training dataset.
2. Discriminator: This network takes an image (a real one from the training dataset or produced by the generator) as input and outputs a probability that the input image is real (i.e., from the training dataset). Its role is essentially to judge the quality of the images produced by the generator.

In this case, researchers used a pre-trained Style Generative Adversarial Network (StyleGAN)—a state-of-the-art generative model that uses adversarial networks to generate new images.

They employed an unsupervised training strategy. This essentially means that the different features were not labeled and were matched directly by the AI.

 “We first trained an image encoder using a pre-trained StyleGAN model as a teacher encoder. In the second stage, we simulated the drawing process of generated images without additional data to train the sketch encoder for incomplete progressive sketches. This helped us generate high-quality portrait images that align with the disentangled representations of teacher encoder.”

This approach allows users more freedom to tweak the parameters of drawing, thus having greater autonomy over the properties of generated images. 

Creating anime with ease

The researchers then put the system (which they called AniFaceDrawing) to the test.

They asked 15 graduate students to draw digital freehand anime-style portraits using the AniFaceDrawing tool. The users could switch between rough and detailed guidance for the art and could refine their input sketch. Ultimately, the researchers also measured how satisfied the graduate students were with the results.

It was a roaring success.

“Our generative AI framework enables users, regardless of their skill level and experience, to create professional anime portraits even from incomplete drawings. Our approach consistently produces high-quality image generation results throughout the creation process, regardless of the drawing order or how poor the initial sketches are,” says Kazunori Miyata, study author.

“Our system could successfully transform the user’s rough sketches into high-quality anime portraits. The user study indicated that even novices could make reasonable sketches with the help of the system and end up with high-quality color art drawings.”

Researchers highlight potential areas for further development, in particular expanding the models with more data that can produce more stylistically varied results, and also deploy the approach to other art styles.

For the anime industry, this poses a new conundrum: on one hand, it can make life easier for animators who are already under a lot of strain. But on the other hand, it could squeeze out even more jobs in an industry that’s already hyper-competitive. As it seems to always be the case, AI can be an exciting and useful tool — how we put it to use is another problem.

Journal Reference: Zhengyu Huang et al, AniFaceDrawing: Anime Portrait Exploration during Your Sketching, Special Interest Group on Computer Graphics and Interactive Techniques Conference Conference Proceedings (2023). DOI: 10.1145/3588432.3591548