Testing self-driving cars

[andrewducker]

There's an interesting discussion here of self-driving car safety, and how to measure how effective it is.

And while it seemed reasonable enough, I realised that what I'd really like to see for self-driving cars is...a driving test. In fact, multiple driving tests of different kinds.

Whenever I write code it has unit tests to go with it. Basically _extra_ code which calls into my code and says "When I ask you to do something, do I get the expected response?" For example, "If I call the 'multiply' function with '6' and '9'" do I get back "54"?

Whenever I finish writing some new functionality (or fixing a bug, or anything else that's got me making code changes) all of these tests get run, and if that unit test comes back with "42" then a big checkbox goes red and I don't get to give my updated program to users until everything is green again.

Now, there are two problems with applying this to self-driving cars - one is that the inputs to a self driving car aren't as simple as a pair of numbers, and the other is that different cars are written in completely different ways, with different hardware hooked up to them.

However - I don't think this should make it impossible to write suites of tests for them. If the regulators came together and provided a standard virtual environment, and a set of test cases ("Empty road", "Road with 1 cyclist coming the other way", "Downtown Delhi during rush hour in the pouring rain"), then individual car manufacturers could write translators which allowed their software to interact with this virtual environment.

It wouldn't be perfect, because you'd be approximating the work that the cars sensors do. But for testing the decision-making ability of the cars it would be massively better than nothing. And the development of a shared set of virtual tests would be a boon to all the different self-driving car manufacturers. Heck, if you open-sourced the test suite/environment then anyone at all would be able to try and write self-driving technology which passed all of the tests.

Obviously, this wouldn't be enough to then sell them to the general public. You'd want to then test that the cars worked just as well in real life conditions as they did in virtual ones. But it seems like something worthwhile, which it would be worth regulators investing in.

Comments

[momentsmusicaux]

You could test the software by feeding it a VR input, but in addition to that, there could be a standardized real-world test where a self-driving car has to navigate a course with junctions, buggies, cyclists, and so on.

But hang on. Are you suggesting the software for self-driving cars DOESN'T have tests?

[andrewducker]

I suspect that each manufacturer does.

But I'd like to see a set of standardised conditions that need to be met, as well as closed ones that regulators don't have oversight of.

[jack]

Huh. I might even push it back a level and require that the cars pass a *physical* version of a test, and then everyone can make sure that the existing simulations they have cover those cases before they try to certify the car for real.

I guess in actual fact, there's nowhere self-driving cars can legally drive by themselves, so currently there's no approval process because there's officially no approval (other than "you can try it out if you pass these regulatory hurdles and have someone ready to take over").

[andrewducker]

Arizona, Nevada, and California all allow it.
https://www.brookings.edu/blog/techtank/2018/05/01/the-state-of-self-driving-car-laws-across-the-u-s/

[jack]

Oh!

I guess I jumped too quickly from "there's no way this could be done safely" to "I assume it hasn't been legalised yet". I mean, I *want* progress in self driving cars, and I'm honestly pleasantly surprised that anyone has been willing to update regulations to allow testing them on roads, which is an obviously necessary step.

But last I heard, they could drive a long distance without needing human intervention but failed badly in (a) some unexpected circumstances, like someone stepping/pulling out in front of them unexpectedly or (b) outside extensively surveyed areas, so just letting companies decide if they're safe to go seems an unaccounted for risk...

[andrewducker]

Waymo are launching this year. Admittedly in just one city : https://www.cnbc.com/2018/10/10/waymo-hits-10-millionth-mile-prepares-for-public-ride-hailing.html

[skington]

Isn't this the sort of thing that the aviation and/or space industries do? Like, as part of their policy of buying two redundant things from different suppliers that work in different ways, so if one is buggy chances are the other was written in a different way so doesn't have the bug or design flaw, so you can fall back to the spare if you need to. I assume that the spec includes a robust test suite.

[andrewducker]

I expect so.

Aerospace is a long way ahead on safety though.

[rhythmaning]

45?!

[andrewducker]

Typo! Fixed! Thanks!

[rhythmaning]

And there I was thinking it was in hexadecimal or something! ;)

[danieldwilliam]

We do quite a lot of testing in simulation and testing of the simulation against real world data.

I don't think what you are asking for is particularly difficult.

Simulate the environment and run a simulation of the vehicle in it.

Test the both the simulated environment and the simulation of the vehicle against the real thing.

Then simulate the environment and run the simulation with the actual hardware in the loop.

Then run the actual hardware in real test environment.

There's a lot of things to simulate and then test the simulation against and a lot of things to test. How good is your simulation of a toddler? Can your ATR tell the difference between an eight year old with a football under it's arm and a dwarf carrying a 12" single?

But I don't think a testing and QA process is conceptually difficult.

I expect we'll get an EU test regime a few years after an early autonomous vehicle kills double digits of people.

[mountainkiss]

The EU might get one, yes.

[danieldwilliam]

and when they do we will have to abide by whatever comes out of it because we buy a lot of cars from the EU and (assuming we have a car industry left in the 2020's) we sell quite a lot of cars to the EU.

[mountainkiss]

I do not automatically make that assumption. But I hope you are right.

[danieldwilliam]

I don't automatically make that assumption.

In fact I'm more worried about the car industry than almost any other industry in the UK.

We're good at making cars. We're good at designing them (Land Rover and F1) and good at building them (Sunderland Nissan). But we're good at that in the context of being in the Single Market. One of the things that Sunderland Nissan did really well was adopt Japanese production concepts of Just-in-Time, kanban and waste / muda elimination. Better, in fact, than the Japanese managed. And that got a small but long-running and persistent cost and quality advantage over other car plants. But only so long as we were plugged in to the production and value chain at both ends.

The car industry is about to go through a period of profound and rapid change with a shift to electric drive trains and probably a shift to transport as a service using automated autonomous vehicles (likely very much fewer vehicles.) Now is not the time to be damaging the UK car industry.

But you know, foreigners, sovereignty , straight bananas and the political adventures of Boris Johnson are worth the price.

[mountainkiss]

Don't forget the blue passports.

[danieldwilliam]

I already have a blue passport.

https://www.google.com/imgres?imgurl=https://upload.wikimedia.org/wikipedia/commons/thumb/0/0f/Australian_Passport_%2528%2522P%2522_Series%2529.jpg/190px-Australian_Passport_%2528%2522P%2522_Series%2529.jpg&imgrefurl=https://en.wikipedia.org/wiki/Australian_passport&h=268&w=190&tbnid=cfFTOmDOkkLRRM:&q=australian+passport&tbnh=186&tbnw=131&usg=AI4_-kSdRyw_cO6gu4P46jVCyvi0fSh1Ug&vet=12ahUKEwj2nJTqjK7eAhXqIMAKHfDmBYMQ_B0wFHoECAUQBg..i&docid=jia92NT5-2YdYM&itg=1&client=firefox-b-ab&sa=X&ved=2ahUKEwj2nJTqjK7eAhXqIMAKHfDmBYMQ_B0wFHoECAUQBg

[mountainkiss]

There's also a lot of contextual change that threatens the industry: more flexible working, cities filling up, less infrastructural investment, more concern about climate impact.

[danieldwilliam]

Indeed - all the more reason to not needlessly mess about it.

[mountainkiss]

You'd think.

[danieldwilliam]

This is the sort of thing I'm worried about with the car industry and Brexit. We're good at making bits for cars but not good enough to overcome the market barriers that Brexit creates and one by one factories get shutdown as they come to the end of their natural lives.

https://www.bbc.co.uk/news/uk-scotland-tayside-central-46109213

[calimac]

They do that. I read an article not long ago about the huge test facility that Google keeps on a decommissioned air force base in central California to run self-driving cars around in.

The problem is that real-life traffic is far more complex than anything you can effectively simulate. The ethos in computer design is innovate fast, break things, and thereby learn how to do them better. That's OK if what breaks is a smartphone app, but not if the result is a fatal auto accident, which has happened more than once.

[danieldwilliam]

That depends on how disposable one thinks people are.

[birguslatro]

When Musk used to talk about Tesla's auto-pilot, (he's been quite for a while about it), it was how many miles per death compared to human-driven cars. I felt that was wrong, as it should've been how many miles per death compared to human-driven Teslas not under auto-pilot control.

My guess would be it's too early to come up with a simulator to test cars in. Both would be moving targets, design-wise. Plus it'd let the manufacturers off the hook when there was an accident, as they could claim it wasn't their fault, as it had passed the test.

Both regulators and the manufacturers want autonomous cars as soon as possible, so 'must have a human at the wheel but with hands off while testing' is probably seen as the quickest way to get to the desired result.

[danieldwilliam]

This got me thinking about trolley problems

https://danieldwilliam.dreamwidth.org/188151.html