Engineers at MIT are working on instilling robots with “socially aware” navigation, allowing them to observe humans’ code of pedestrian conduct.

Image credits Chen et al., 2017.

Generally speaking, most pedestrians follow a set of unspoken social conventions when they go out and about our day. Things like don’t walk into other people; try to walk past instead of through groups; keep to the right of the sidewalk and pass on the left; avoid incoming obstacles and walk at a steady pace so other pedestrians can avoid a collision in turn.

Robots aren’t at that level of social finesse just yet. So a team of researchers from MIT is trying to program it into the bots so they can better integrate and function alongside humans in society — while keeping annoying bumps in foot traffic to a minimum. Led by Yu Fan Chen, a former MIT graduate student, they designed and build a small robot which relies on “socially aware navigation”. They tested the robot inside the sprawling hallways of MIT’s Stata Center in Cambridge, Mass., where it managed to successfully avoid collisions while keeping up with the average speed of pedestrians.

A walk in the campus

“Socially aware navigation is a central capability for mobile robots operating in environments that require frequent interactions with pedestrians,” says Yu Fan Chen.

“For instance, small robots could operate on sidewalks for package and food delivery. Similarly, personal mobility devices could transport people in large, crowded spaces, such as shopping malls, airports, and hospitals.”

Successfully going about in traffic, if you’re a robot, mostly revolves around four processes:

• localization (knowing where you are),
• perception (having a way to observe your surroundings),
• motion planning (deciding on the best path to a given destination), and
• control (the ability to actually move along that path.)

As far as localization, perception, and control are concerned, Chen’s team went with standard approaches: off-the-shelf sensors such as webcams, a depth sensor, and a high-resolution LIDAR sensor for perception, and open-source algorithms to map the bot’s environment and determine its position for localization. Control was handled by a standard ground vehicle drive — an electrical motor and wheels.

What the team wanted to innovate on was the planning step. Roboticists usually tackle this issue by using one of two approaches: having the bots calculate everybody’s trajectories and then deciding on which one to take (a trajectory-based model), or selecting a general route then adapting on the fly to avoid possible collisions (reactive-based model). However, they both have limitations.

“But this [trajectory-based approach] takes forever to compute. Your robot is just going to be parked, figuring out what to do next, and meanwhile the person’s already moved way past it before it decides ‘I should probably go to the right,'” says MIT graduate student and paper co-author Michael Everett. “So that approach is not very realistic, especially if you want to drive faster.”

On the other hand, reactive-based approaches don’t always deliver since people don’t walk exclusively in straight, predictable lines — they tend to weave and wander around. Because of their unpredictable environment, these bots end up either colliding with people or being pushed around so much they take way too long to reach their destination — although to be fair, the first one might make them more likable in a goofy sort of way.

“The knock on robots in real situations is that they might be too cautious or aggressive,” Everett adds. “People don’t find them to fit into the socially accepted rules, like giving people enough space or driving at acceptable speeds, and they get more in the way than they help.”

Virtual lessons

The team’s solution was to use reinforcement learning (a type of machine learning) to drill pedestrian etiquette into the robot. The advantage of this method was that the team can train the robot in a simulation, by-passing the cost in time and computing power (which is limited based on the robot’s frame) of training in real life. Once there, it can use what it learned to handle similar scenarios.

A probably-unintended side effect was how cute this thing was when going about its day:

They guided the robot to prefer certain paths based on the trajectories of other objects in their environment. They also encouraged behaviors that mirror how human pedestrians interact in this phase, such as having the robot pass people or groups on the right by penalizing it when it passed on the left. To enable it to make the most out of what it’s learned, the team enabled the robot to assess the state of its environment and adjust its trajectory every one-tenth of a second.

“We’re not planning an entire path to the goal — it doesn’t make sense to do that anymore, especially if you’re assuming the world is changing,” Everett says. “We just look at what we see, choose a velocity, do that for a tenth of a second, then look at the world again, choose another velocity, and go again. This way, we think our robot looks more natural, and is anticipating what people are doing.”

To test the outcome, the team instructed the robot to make its way through the halls of MIT’s Stata Building. This location was chosen since it’s quite busy with people doing everyday things, such as going/rushing to class, getting food, hanging out — in other words, it offers a wide range of motion patterns. It was able to autonomously drive for 20 minutes at a time, rolling along with pedestrian flow, usually keeping to the right of hallways (although it did pass people on the left on occasion), and successfully avoiding any collision.

Moving forward, the team plans to refine their social navigation algorithms by teaching robots how to better interact with crowds in a pedestrian setting — things such as “don’t move through people, don’t split people up, treat them as one mass.”

The team will be presenting their paper “Socially Aware Motion Planning with Deep Reinforcement Learning” at the IEEE Conference on Intelligent Robots and Systems in September. Until then, it can be read on the preprint server ArXiv.