Where to take driver out?

Introduction

When preparing for a deployment along a specific route, AV developers may test their AV in simulation against scenarios that are statistically likely to take place in that given location. They will also usually send out their AV along the route with a safety driver. This way, they can collect information about how their AV is performing at different sections of the route in a safe manner.

Figuring out when taking the safety driver out can be a complex task with high-stakes. In the following demo, we use Edge to determine failure modes in specific areas, see where those failure modes don’t occur, and use that understanding as a place to consider a deployment without a safety driver.

Open Autonomous Vehicle Sandbox in Edge →

Gathering what we need

Annotations

In order to analyse the performance of our AV, we have gathered a set of scenarios and key scenario annotations into the view title "Scenario Results - Taxonomy View":

Test Scenarios

The scenarios we will analyse during this demo are the successors of the header node Test Scenarios which can be found on the RHS of our taxonomy view. We can open up the spatial view titled "Scenario Results - Spatial View" to see these same scenario nodes positioned along the route on which they take place.

Let's prepare the view for our investigation:

Which regions of our route contain the most VRUs?

Let's start off by getting an overview of how many of our scenarios contained a high-risk hazard such as VRUs (pedestrians, cyclists, etc.) and where along the route they took place, so that we know which regions of the route we should treat with caution.

The following insights are clear:

• The majority of scenarios involve a car hazard (as opposed to pedestrians, objects, trucks)
• Scenarios involving pedestrians took place predominantly at the central junction of the map and as such, we may want to treat this region with caution and keep the driver in here
• Scenarios involving obstacles and vehicles are fairly well-distributed across the map

Where is our AV performing best?

Now let’s find out where on the map our AV is performing best. To do so, we will colour the scenario nodes according to whether the AV entered a collision with the hazard, or whether the scenario was navigated safely and successfully.

Colour the scenarios which involved no collision green:

At this point, we can already see our AV could be safely deployed along the LHS of our map, due to good performance on the scenarios which are likely to take place there.

Where is our AV still failing?

It is known that incidents involving head-on collisions between vehicles are the most deadly, so let’s treat any spots along the route at which our AV has entered such a collision very carefully. To visualise risk, let's colour head-on scenarios red, and all other collision types orange, using the same 4 steps outlined above. As an extra step, increase the node size of head-on scenarios such they can be easily seen.

This has reinforced what we discovered previously: roads along the LHS of our map would be the least risky spot to first deploy our AV without a safety driver.

Any other factors we wish to consider?

Lastly, it's worth noting that another category of high-risk areas are those where vehicles tend to operate at high speeds. Let's check out the speed profile of our scenario set by opening up the "Histogram - Maximum Speed of Hazard" view.

To ensure the region of our map which we plan to take the driver out is indeed lower risk, highlight the LHS of the spatial view and observe where the scenarios light up in the histogram.

The region we hypothesised it might be the safest to take the driver out does indeed contain scenarios at the lower end of hazard speeds!