Model AV testing

On a darkened street in Tempe, Arizona, in March of 2018, Elaine Herzberg was crossing a street while walking her bicycle. An Uber SUV in autonomous driving mode struck and killed her. The Guardian reported that the Uber vehicle detected Hertzberg more than five seconds before the crash, but it could not identify her as pedestrian, and the backup driver was not paying attention to the road. It was the first reported instance of a pedestrian being killed by a self-driving vehicle.

Autonomous vehicles (AVs), cars that can drive themselves without human control, are taking to streets and highways across the country with increasing frequency. The McKinsey & Company consulting firm estimates that cars with self-driving technology could constitute 20% of auto sales by 2030, and 57% by 2035, but the record of the new technology is questionable.

Since Herzberg’s death, accidents caused by autonomously driven vehicles have continued to make headlines, and AAA reports that fear of self-driving cars rose markedly last year. Definitive numbers are hard to come by, due to the method by which the National Highway Traffic Safety Administration handles statistics on AV accidents, but in June the Washington Post reported that Tesla’s Autopilot self-driving system has been involved in 736 crashes and 17 deaths since 2019.

Proponents argue that their data show that AVs are safer than human drivers overall, but a recent detailed review by Ars Technica, an online publication that covers technology, concluded it is too early to say. It is clear, though, that the artificial intelligence driving AVs can be prone to error in novel situations. AI algorithms may be able to beat grandmasters at chess, but they are devoid of, for lack of a better word, common sense.

As scrutiny of autonomous vehicles increases, researchers at Washington University in St. Louis are puzzling out the best ways to improve their safety. What if, asks computer engineer Yevgeniy (Eugene) Vorobeychik and architect Constance Vale, you could build a scale model of an urban neighborhood to test self-driving cars, in an environment that you could customize to recreate the kinds of situations that cause problems for autonomous vehicles, all without endangering people or property? The two and their research teams are doing just that in the Architectural Design of an Experimental Platform for Autonomous Driving project — the Washington University Miniature City (WashU Mini-City), constructed in McKelvey Hall.

The goal of WashU Mini-City is to provide a physical autonomous driving platform that combines computer science and architecture to allow novel experimentation in a safe environment — at a low cost. The mini-city aims to be a laboratory for broader stress-testing and monitoring of AVs, ultimately increasing their safety and reliability. The project is funded by seed grants from WashU’s Office of the Vice Chancellor for Research and Provost’s Office, as well as the National Science Foundation.

Vorobeychik, professor of computer science and engineering at the McKelvey School of Engineering; and Vale, chair of undergraduate architecture and associate professor in the Sam Fox School of Design & Visual Arts, have designed and created a marvel of a charming city center, at 1:8 scale. WashU Mini-City — with a current footprint of about 3000 square feet — features realistic building façades and streets complete with precise traffic markings, streetlamps, stop lights, traffic cones and billboards. Most important, it has 3D pedestrians at scale, some accompanied by their faithful 3D pups. The model has a variety of spaces that mimic the kinds of neighborhoods AVs need to navigate in cities.

Almost all elements of WashU Mini-City are modular, so they can be easily moved, replaced or modified depending on the experiment. The care in the design and construction, and the verisimilitude Vale and Vorobeychik achieve, are noteworthy. To create trees, for example, Vale and her students devised a sophisticated fabrication technique using both digital and handcrafted methods. The trees are made of vellum painted with translucent acrylic painted strings that were then wrapped around laser-cut tree branches and slotted into 3D-printed tree trunks. “They are truly incredible,” Vorobeychik says.

The life-like appearance is for more than show. One of the most difficult aspects of improving the safety of AVs is the long-tail problem — situations that are rare, but that challenge the capacities of the algorithms that drive the vehicles. For instance, it’s routine to train an AV to stop at a traffic light, but what if an AV confronted something utterly unexpected in the middle of a road, like a giraffe that had escaped from the zoo? A human driver would pull over, stop the car and probably post a video to social media. AI is less predictable when faced with unusual circumstances.

With WashU Mini-City, Vorobeychik and Vale are focusing specifically on exactly these kinds of perceptual vulnerabilities, where the long-tail issue arises. “When architects make physical models, we think about what kind of register we have to hit relative to the real — how real they need to look,” Vale says. “In a miniature city used to train AVs, one has to take account for color, sheen and reflection, qualities that are important in machine sensing and challenging in the case of reflection.”

There are also known malicious hacks. One is to trick AVs by putting bits of tape onto speed limit or stop signs to fool an AV’s sensors. “What if you’d like to test if a malicious actor could cause the car to crash or hit pedestrians by changing something small in the environment?” Vorobeychik asks. “This is precisely the type of situation you could test in a controlled environment with this scale model and 3D-printed pedestrians.”

In addition, WashU Mini-City enables what Vorobeychik calls stress testing, that is, specifically engineering malfunctions in AVs to study outcomes and find ways to patch problems. All this could be done at costs orders of magnitude lower than similar experiments at full scale. “Full-scale experimentation with AVs is extremely expensive, and the cycles are slow,” Vorobeychik says. “In WashU Mini-City, we are in this in-between space where we can run experiments quickly with fewer precautions, as the worst thing that can happen is not that horrible.”

Another of WashU Mini-City’s prime research benefits is that it is a physical space — a real autonomous vehicle, albeit a small one, driving in a model city. As Vorobeychik explains, data from live testing is a key research shortcoming in AV research particularly. “There’s a well-known problem called the real-to-sim gap. Simulation is the most efficient way to test things, but it’s digital, not physical. It’s just a different kind of animal, and it’s always going to be missing something, as it’s not honest to reality,” Vorobeychik says.

One physical feature that is hard to model in simulations is shading. The team is now in the last phases of wiring the platform to simulate driving under different light settings and with working traffic signals. In addition, WashU Mini-City will give researchers a chance to easily experiment with different architectural, building and infrastructure features to determine how they are “seen” by AV sensing devices. By swapping out windows, façades and street features, and altering atmospheres, Vale and Vorobeychik can recreate a wide palette of driving conditions and environments. Vale can imagine adding snow to the mix, calling on techniques used in the performing arts. “We could easily simulate snow using salt, which is the standard strategy on theater and film sets,” she says.

While WashU Mini-City is still in its relative infancy, Vale and Vorobeychik are looking over the horizon to experiments that involve delivery drones, which may soon be a feature of city streets that will add complexity to AV navigation. Vale is also thinking of new architectural features of smart cities, where mass transit, energy and other infrastructure features are more connected in ways that could address sustainability and equality issues.

“How could we interconnect autonomous vehicles with existing transit systems such that we’re reducing the amount of individual travel in cities?” Vale asks. “Autonomous vehicles have this incredible potential to be able to improve the way that our cities operate, but at the same time, they’re just as rife with possibilities to make things much, much worse.

“It could mean more cars on the road and actually make people commute further to work because they won’t have to be engaged when they’re driving,” Vale says. “I’m a big proponent of interdisciplinary thinking. It’s my hope that architects and urban designers, in collaboration with engineers like Eugene, can craft a way forward for autonomous vehicles that can help address equality, climate change and other challenges.”

As WashU Mini-City scales up, the duo hopes to work with academic and industry partners to expand the initiative and research capacities of the project. “We have established the Center for Trustworthy AI in Cyber-physical Systems,” Vorobeychik says. “The vision is to expand our scope and extend beyond WashU, but at the moment, we’re just getting started.”