RoboCup 2018 - S&T Test Methods Used to Evaluate Rescue Robots

Since 1997, several continents have played host to an international soccer tournament. No, not the World Cup -- the RoboCup. Robots of all shapes and sizes test their “metal” in the world’s favorite sport. Engineers and fans from across the globe have gathered to watch hunks of autonomous steel try to nudge a ball into a miniature net. 

Those robots have improved their footwork over the decades, and the RoboCup competition itself has evolved as well. Today, the event is more than a game. It is a conference that showcases different families of robots with applications for home, industry and public safety, all of which were a part of this year’s conference in Montreal, Quebec, Canada. Robots in the Public Safety category, RoboCup Rescue, were once again judged according to standards developed by the Department of Homeland Security (DHS) Science and Technology Directorate (S&T) and the National Institute of Standards and Technology (NIST).

RoboCup Rescue Standards

RoboCup Rescue was added to the bill in 2001. The S&T- and NIST-developed test methods include requirements and apparatuses that allow robots to demonstrate and measure their abilities, everything from climbing stairs to scaling walls and turning knobs. Test methods are divided into three levels of difficulty in order to distinguish the more powerful systems.

To date, S&T has developed about 30 different methods with NIST to evaluate the performance of robots in search and rescue, decision support, information systems and many other functions that could help in a crisis. Twenty are for ground robots and 10 are for drones. The apparatuses used at this year’s RoboCup Rescue tested maneuvering, mobility, dexterity and mapping. Sensors were also tested for visual acuity, color acuity, thermal acuity, audio acuity, hazmat label identification, motion acuity and latency.

“This year’s challenge tasks used the same standard test methods as previous years, but had more difficult apparatus settings toward the final missions,” said Adam Jacoff of NIST, who has chaired RoboCup Rescue since 2001, “That’s how we use the test methods to push the state-of-the-science while maintaining the ability to measure and compare system capabilities in different locations at different times.”

What does this robot do?

The original test methods were developed to provide the Federal Emergency Management Agency (FEMA) and first responders a reference point for selecting robots to carry out different missions. The methods helped FEMA make better purchasing decisions, as they could validate effectiveness and distinguish robots that did their jobs well.

“We wanted to develop standards the whole community agreed upon, especially first responders who will be using them to disarm bombs, go into fires, etc.,” said Kai-Dee Chu, S&T Program Manager. “If the claim is that a robot can carry a victim out of a fire or inspect luggage aboard a plane, we design an environment to test it.”

Without standards and benchmarks for developing and testing machines, there are no clearly-defined rules by which the robots can be compared.

Before S&T and NIST standards were developed for public safety robots, it was more difficult to distinguish one robot from another. The design of a soccer-playing robot, for instance, can be guided by the rules of the game, but there were no preexisting ways to define excellence for rescue robots. Today, instead of each robot being its own unique set of wires, wheels, belts and bolts, the robots can be categorized and given specific requirements for tasks that support activities such as fire rescue or handling explosives.

Not long after the test methods were developed, FEMA realized some robots could only be used by well-trained operators, so different apparatuses were adopted as training grounds for first responders. Since then, various public and private partners have found them useful—the North Atlantic Treaty Organization has also adopted these methods for training.

The standard test methods have been received by different public and private representatives at the RoboCup and beyond. Their adoption continues to expand.

“If you’re in a major city, you can probably find someone employing our test methods,” said Chu.

The Big Leagues

Now that the 2018 RoboCup has passed, the S&T- and NIST-developed apparatuses created by local Canadian sponsors and organizers will head to Canada’s Police College in Ontario and the Canadian Explosive Technicians Association. While many organizations have adopted some of the standards or housed the apparatuses after RoboCup events in past years, this is the first nationwide adoption of robot standards.

“They will set minimum acceptable levels of proficiency for bomb technicians across Canada,” said Jacoff, “like getting a driver’s license for emergency response robots. The test methods focus training and measure basic maneuvering and dexterity along with more operationally relevant tasks such as accessing buildings.”  

Last year’s apparatuses, from RoboCup Rescue 2017 in Nagoya, Japan, were donated to the Fukushima Test Field, set to open in 2020 for first responder training.

The widespread adoption of robot standards demonstrates how innovation is not necessarily always in the hardware. The hardware, in this case, needed purpose. S&T and NIST provided it, and as a result, various law enforcement agencies are given unprecedented capabilities to help them in their missions.