“UberImpact,” a questionable branding decision (Pride Parade in Seattle, Jun. 29, 2014) © Jiong Gong posted to flicker under CC BY-NC-ND license, at https://www.flickr.com/ photos/124869658@N02/16016091244/
You might remember a rash of people on social media becoming suddenly interested in “the Trolley Problem” a few years ago. A classic ethical thought experiment about the dilemma posed by a runaway trolley, contemporary interest in it has been revived as corporations and engineers (PDF) begin grappling with programming and design decisions that will govern self-driving cars.
This post is in part a response and counterpoint to Michael Daigh’s recent post that presented a hopeful view of self-driving cars that always (obviously!) comply with the law and never make unsafe maneuvers. In that post, the author supposed a world where the robots were programmed to adhere to Asimov’s three laws of robotics, which emphasize avoiding harm to humans. I think it’s important to understand that this outcome for our self-driving car future isn’t self-evident or guaranteed—it is going to take deliberate effort and vigilance because there are already forces at work guiding our streets toward a different, less utopian outcome.
As a matter of practical realism, self-driving cars can and will be safe only directly in inverse proportion to their marketability. That is, you can either have a safe autonomous vehicle, or you can have a mass-marketable one that drives the way contemporary human American passengers and drivers want them to drive, but not both. The reason for this is simple: humans are unsafe drivers, and the customs of the roadway reflect that.
Exhibit 1: the self-driving Uber that killed a pedestrian in Arizona earlier this year had its emergency braking programming disabled to “reduce the potential for erratic vehicle behavior.” Here’s an analysis of what might have happened if the safety feature were enabled.
Self-driving cars that don’t drive the way a human would drive have a “potential for erratic behavior.” They’re prone to, say, driving under the speed limit, or slowing, stopping, and avoiding potential hazards that a human might disregard. That’s erratic in relation to the current culture of the road, which is to out-drive drivers’ realistic reaction time, use the vehicle’s mass and speed as a threat to keep the way clear, and assume things will all work out.
Modern car- and road-culture expectations assume unsafe driving. When people are forced to contemplate what safe driving really looks like, the way a properly programmed robot would do it, it’s not going to sell cars. Nobody will want to own one—not if it’s going to stop at every intersection for pedestrians, even though that’s already the law. Not if it’s going to give bicycles sufficient following distance and no less than three feet of passing buffer, but also not unsafely cross into the lane of oncoming traffic, instead waiting patiently for a safe opportunity to pass, even though that’s all already the law. Not if it routinely drives under the speed limit to avoid reasonably foreseeable hazards or to adapt to road conditions, even though that’s already the law.
American road culture includes all these ways that car drivers already break the law or otherwise drive unsafely. Self-driving car manufacturers, who want to sell to the public and have their product share the road with human drivers (something that will have to happen, unless and until we accomplish an overnight switch to ban human driving!), quickly discovered that their products need to match our cultural expectations.
A car following Asimov’s laws probably would never reach 30 mph on a typical Minneapolis street, for example, and would virtually always drive below the speed limit, completely unlike any human driver any of us have ever known. This is why the autonomous car manufacturers are in the early stages of a long, multi-faceted war to let the robots drive as unsafely as humans do.
Evidence of this war can be found in Elon Musk’s “playing the refs” efforts to discourage media coverage of autonomous car collisions. It can be seen in efforts to ease and deter regulations governing these cars, their sale and manufacture, and their use on the roads. And it can be seen in their brazen determination to put the cars on the roads with safety programming disabled to use public streets and everyone on them as their guinea pigs.
An illustration of the ethical choices available under the current proposition of autonomous cars.
The profit incentive to disregard laws and regulations is going to be just as strong, if not stronger, than a human driver’s incentive to engage in the same unsafe and socially malignant behaviors to get where they want to go quickly. Only, its going to be baked, universally, into the design of the vehicles themselves. Unless maybe autonomous car owners will get to choose whether to “be safe” or “get me there quickly” to match their own personal ethical calculus.
When a manufacturer decides it’s too “erratic” to stop for pedestrians who want to cross at an unmarked intersection, or too unacceptable to drive at speeds slow enough to be safe—and the car that doesn’t do those things sells better—will existing law prevail, or will the allure of the market bootstrap unsafe driving into a new law of the road, hard coded by car manufacturers to best suit their customers?
This is why we can’t take for granted that autonomous vehicles will be safe, but we will have to insist on it, repeatedly, and in many forums: legislative, regulatory, product engineering & design, and public opinion. As has already been amply demonstrated, compliance with existing law and local preference is not a given with all the financial incentives at play in this developing industry.
If we aren’t careful, we will miss this opportunity to make roads safer by compelling the robots to drive more safely than we do. Instead, we risk codifying a bunch of unsafe things our culture is willing to tolerate from human drivers, programming the cars to reproduce them, for marketing purposes.
I disagree with the ultimate premise in that self-driving cars in total will result in many fewer traffic fatalities. Redesign can reduce humans’ proclivity to over-ride certain systems. Typical cars invite all of us to fiddle with the radio and air conditioning, talk with our cellphones and generally become inattentive to various degrees. That makes typical cars potentially occasionally unsafe.
I don’t disagree that self-driving cars can be vastly more safe, I just doubt that their safety benefit potential will survive the market.
Simple example: if a car’s self-driving features can be overridden, they will be if the driver–occupant is frustrated with its courtesy to other road users. Safe in theory, not in practice.
So you don’t accept that the vast majority of automotive casualties are due to human error?
The thing is those things we do to “get there quickly” actually make very little difference to actual travel time, especially in the city, where things like lights and congestion erase any gains our dangerous behavior might create.
And we don’t really do them to get there quicker, we do them to assuage the frustration we feel about not being able to go as fast as feels safe to us inside our cars. We may even feel more frustration if we think our autonomous vehicle can safely go even faster because it won’t make “mistakes.”
In addition to regulation that requires autonomous cars to drive in compliance with traffic laws, we also need to remake our treats so that they reduce driver comfort at high speeds.
I was having a conversation with my friend who works for Google last winter, about autonomous cars. He brought up how everyone expects them to fail in the snow, that’s not fair, we can work those issues out.
I said: it’s not the snow that changes things, it’s the culture of driving in the snow. People no longer follow the street markings, they don’t stop, there’s basically no rules. You can’t make a car work in those conditions. The culture is too different!
the problem, as usual, is people.
“I can’t come to work today, my self-driving car has evaluated the risks and has estimated a safe arrival time of 3pm.” Gonna sell like hotcakes!
I have a slight issue with the premise of this article, which is that people will expect that autonomous vehicles will drive the way they do. I think the psychology of being a passenger in an autonomous vehicle will be much more like the psychology of being a passenger in a taxi or Uber, and I think most people who are passengers would prefer their driver to err on the side of safety rather than speed. It is only the illusion of power that driving gives which leads to the kinds of dominance games which define our current road environment. So, I don’t agree that cautious driving will make autonomous vehicles unmarketable; in fact I’d assume that, to the extent that companies continue to market to individuals, that their cautious driving will be the main marketing point, along with the ability to play Candy Crush while driving.
That being said, I agree that the question of how autonomous vehicles should behave, and how our roads should be designed in the age of autonomous vehicles, are important philosophical questions which will need to be addressed, and those discussions will be contentious. The road in Tempe where the fatality occurred is dangerous by design, and will continue to be hostile to pedestrians and bikes no matter who or what is controlling the cars. The future of the crosswalk will be very interesting. I did a blog about this a while back: https://bike-lab.org/2018/04/04/smarter-cities/
Yes. Chances are if you’re trying to take a nap on your commute from the single family house you can afford in Kenyon (because you couldn’t afford a single family house any closer in) to your job in downtown Minneapolis, you’re going too be annoyed if you get woken up if your car stomps on the accelerator after a traffic light or tailgates and then has to slam on the brakes.
This kind of gets at something I’ve been thinking about with regard to commuting, which is that most people think of their commutes quantitatively (i.e., how much time did it take me to get from my house to my job) versus qualitatively (i.e., did I enjoy my time in my car). I’ve been exclusively commuting via bike and train over the last 5 years and one of the rationales I’ve come up with for why I prefer both over driving is that they’re better qualitative commuting experiences. I don’t experience stress (unless there’s an aggressive driver or some kind of misbehavior on a bus/train), I get some modicum of activity in, whether that’s walking or biking, and I get a chance to mentally prepare/decompress from the day without having to focus on driving.
Granted, I think I mostly started thinking about commuting in qualitative terms as a way to rationalize why I don’t mind longer commute times, but I think it’s a useful exercise. After all, for folks who drive, I wouldn’t be surprised if the commute is one of the most stressful parts of their day.
Thanks, your first paragraph covers my thoughts as well. As long as people get where they are going, I think they’ll zone out and do something else more often than not.
Maybe. But when the speed limit is 35 and I take the lane in front of you on bike and an automated car obligingly slows down to my pace, I question how chill the owner is going to be. Especially when it happens repeatedly.
I think there are big psychological differences between a hailed/hired car and a car you *own*, when it comes to how much control you’re comfortable giving up. Attitudes and preferences along these lines can change, but I’m pretty sure we’re not there today.
Yeah, I think we need to realize that most driving today, even when driving to work or for errands, is recreational. Recreational in the sense that we don’t drive like professional drivers (bus or truck drivers). If we’re late, we speed. If we’re distracted, we don’t look up.
Autonomous cars need to drive like professional drivers. Your bus driver doesn’t speed up because you’re late.
The other thing about calling it recreational driving is how cars are marketed. Sure, some are marketed as practical, but many are marketed as recreation: speed, cornering, acceleration, braking. Most of those things make zero sense in an urban environment, where you’re lucky if you can safely drive 25 mph, the roads are straight, and the stop lights & signs are predictable.
So, currently, autonomous cars have to be sold to that mindset, because they are being marketed against other “recreational” driving vehicles, so they have to meet or exceed them on the same terms (look at Tesla’s bragging about acceleration, which is pointless in a city or even suburb at rush hour).
In the future, however, with “TaaS”/MaaS (Transportation as a Service or Mobility as a Service), then cars themselves are no longer marketed to you. Instead, the service is marketed to you. In that case, acceleration, speed, cornering, & braking make no difference if you’re trying to get from A to B (with other passengers).
It’s fun to drive a car and accelerate & corner hard, but it’s no fun being the passenger in such a car, which is what happens with TaaS/MaaS.
So then the only selling point beyond utility is entertainment (I’ve replied elsewhere on that) and time-to-destination. As others point out, even when we speed we really get to our destination no faster than if we don’t. So I don’t think that will be a viable selling point when cars are no longer marketed to individuals.
The thing is there’s a disconnect between how cars are marketed and what kind of cars Americans actually buy. The guy that drives a Camry to work fantasizes about taking it on the Nürburgring the next weekend and marketing really taps into it. In reality Toyota doesn’t even bother to make a performance oriented car, and most other car-makers make one model. That doesn’t mean we’d tolerate cars with Malaise era specs anymore, but Toyota finally dropped offering a V6 with the RAV 4 a few models ago because so few people were actually buying it despite it being an absolute speed rocket compared to the typical I4 on that class of vehicle.
My stepfather’s experienced (albeit with used cars) is that people buy cars more based on emotion and practicality than which has the best 0-60 time.
Do you honestly believe that cabbies drive cautiously? Some of the stomach turning rides I have ever had have been in a taxi. Taxis whether autonomous or not will have a motive to maximize revenue.
I don’t believe cabbies tend to drive cautiously, but I do believe that most passengers would prefer their cabbies to drive cautiously. A cab company is more likely to be successful marketing “We will get you there safely” than “We will drive aggressively to get you there faster.”
A TaaS company would have an incentive to prioritize speed over caution, but the mass market for autonomous vehicles (which is what the article is proposing) will probably value safety over speed.
Do you honestly believe that cabbies drive cautiously? Some of the most stomach turning rides I have ever had have been in a taxi. Taxis, whether autonomous or not will have a motive to maximize revenue.
Thanks for this. So important. These AVs will only be as safe and friendly for our cities as we insist them to be.
Right now we are letting markets and tech advancements lead the way on autonomous vehicles but how the next 100 years goes, may well be decided now how decide to democratically control this game-changing tech in next few years.
It’s not just a safety rules and regulations, its all sorts of things:
city livability issues (speed, where traffic flows)?
equity issues (does the larger privately provided system include means to service to subsidize rural or poorer neighborhoods areas)?
city planning issues galore
what do they do with user data and what access does the public, city government have to it?
who gets priority with street use/drop off areas?
what about delivery services?
how do incentivize more occupants per vehicle?
how to deal with peaks/congestion?
what do we do to replace revenue of gas taxes, tab fees, tickets, – how do AVs/riders pay for roads?
pricing rules (does surge pricing all go to supplier to incent supply, subsidize off peak use, make them super profits for them or should taxpayers have congestion taxes).
do we ban certain vehicles in certain areas or close more streets to motor vehicles altogether?
where do these fleets park?
what if vehicle miles are so cheap that these vehicles create “mobile” real estate by simple driving around endlessly like sharks on our streets, with or with riders?
how will these systems work with or against public transit systems?
I’m sure others could add to this list.
Usually tech and economics come first, than regulations, taxes later.
So, If we are not careful, operators of driverless ride-hail autonomous vehicle fleets will become entrenched monopolies or cartels in our cities, imposing their will on us rather than other way around.
Before these AV ride-hail systems become entrenched is the time to regulate them. Exerting democratic controls after they are established much more difficult. And we need many cities to band together and insist on similar democratic controls and standards that become the norm.
As an example of how regulation/taxing schemes are easier at first than later – one of the countries getting the most overall benefit for all their citizens from natural oil resources is Norway – this is because they had no oil, no one knew they had any oil until it was suddenly discovered. Before any company, companies were already established in the industry, the government was able to agree that the money from this oil would stay in public hands and be invested elsewhere in an attempt to avoid the “oil curse” on their economy.
We have similar opportunity in regards to fleets of ride-hail AVs now, as Norway did with oil. We now know this change to our economy is coming via tech, but its not quite here yet. We have a chance to get ahead of it.
I suspect cities that already have good policies in place are best suited to apply these to AVs as they show up. Cities like London with congestion pricing already can apply it to AVs to incentivize off peak use and use that revenue for benefit of citizens. Cities like NYC that already require ride-hail companies give them their data will no doubt demand same from AV ride-hail services. Cities that already have lower speed limits will apply those lower speeds to AVs also.
If there is a central organization already dedicated to thinking and developing progressive policy around AV systems, I’d love to know who it is. If not, we need to create one now. We need cities to work together so they can’t be isolated and made to heel to big corporate interests. We need data and rigor to determine best balance of competing issues. We need to try different approaches to see what works best overall.
Yes, this, so many issues.
But I feel like this will eventually mostly be taken out of the hands of cities, and instead “for consistency” will be in the hands of states or federal agencies.
Correct – this is why cities should be very firm and united.
I like this report by NATCO, this article references.
https://dirt.asla.org/2018/05/29/how-cities-can-prepare-for-autonomous-vehicles/
Major and medium sized cities banded together in each state can have major influence on the Senators – this pressure needs to be applied early and often as AVs take over out city transportation systems, so we don’t get federal pre-emption. Also, more progressive states and help make sure bad policies don’t get nationalized.
We can do this, this is battle we can win. But we can’t let them own us like Big Tobacco did and then spend 50+ years trying to fix it, we have to be on our game, in front on this, before any big corporations have that much on the line.
More sensible regs we have up front – easy it will be to maintain them, rather than apply them a new on a industry unaccustomed to them.
We also need to make allies with many communities and make sure they are served to have united democratic pressure on these companies – make sure our policies also deliver for elderly, disabled, rural residents, rich and poor.
We got this, we have a good idea how this may go bad, we are in more knowledgeable position and people were when we switched from car to horse, we know how to organize democratic pressure, we have a good window of opportunity now.
I love the trolley illustration.
I don’t deny the points that some people are going to be irritated if their car drives at a speed that they perceive as too slow, but even if you don’t believe in TaaS, as I don’t, there’s two huge advantages to them that I believe will make them marketable
1) You can still have a car at any age or ability. I’m hoping that self driving cars will be available in the next 30 years so when I reach the point where I can no longer safely drive a “manual” car I won’t lose the freedom that having a car provides.
2) Right now time spent in your car is pretty much written off unless you actually enjoy driving. To reclaim that time opens up all kinds of possibilities. Even if it takes longer because the car drives more conservatively, you can take a nap or watch Harry Potter movies on the way to work.
I just pretty skeptical of 2. It will be more flexible than time spent in an airplane, but you’re still going to be trapped ina box with limitations on what you can do. Maybe that makes 45 minutes comparable to 30 now, but I don’t think it’s going to make two hours all that much more palatable.
Monte, for #1, why not bike, walk, or take transit?
Adam, for #2, this is all about subscriptions and mass consumption. Gaming & VR, binge-watching your favorite TV shows, listening to music & podcasts, reading books, texting & social media (I mean *everyone* is glued to their phones *today*, which is why we have so many distracted driving deaths). All the guilty pleasures that we want to do today, but no problems with having an accident because you won’t be driving.
I could theoretically walk or take the bus for shopping, since there’s a grocery store 3/4 of a mile away and a Route 535 bus stop two houses down and I doubt at that age I’ll be hauling huge bags of wood chips home. But what if I want to visit my friend in suburban Shakopee or take a road trip to Wadena?
Marketable depends on a host of variables. What if insurance companies say that level 5 self driving cars pay $200/yr, level 4 pay $1,000/yr and all others pay $4,000/yr. What if registration and license fees follow a similar pattern?
How much is your time worth? What if you can get work done on your laptop or read a good book while being driven to work and back home?
You’re making an assumption that road behavior remains the same. If AV’s are driving slower and more cautiously, will non AV’s behind them be able to go much faster?
Our current road system is designed for speed over safety. What if U.S. traffic engineers put away their 1960’s thinking and catch up to engineers in other countries and begin designing a much safer road system that forces all vehicles to be driven more cautiously?
More: https://streets.mn/2017/06/09/preparing-our-roads-for-autonomous-vehicles/
NATCO has a good report on these topics, this article provides an overview.
TaaS ride-hail AVs have much potential for creating better transportation systems, good cheap mobility for many more people (elderly, disabled, blind, children etc) and for improving our cities, but there are no doubt downsides.
If we are on our game and use our birthright, our democratic systems, and political pressure to shape TaaS in way to maximize safety for car occupants,peds and bikers, improve our cities/streets, reduce transportation costs, create more usable real estate in already developed areas, work in concert with our public transit systems (not as enemies, competition) etc…we can make this transportation revolution really amazing.
Or we can sit around and let the corporations mow over us, and make things worse, or less better than they could have been.
Ride-hail AVs are coming, like it or not, It’s just a matter of when and how, not if.
The economics and convenience will be a irresistible market forces.
Swe have once in 50+ year chance to greatly change the safety, livability of our cities for the better.
We can rise to the challenge and do this really well and make things so much better for next generations. We can do this.
https://dirt.asla.org/2018/05/29/how-cities-can-prepare-for-autonomous-vehicles/