Friday, June 10, 2016

Update on Me: 8 years ago I pointed out that the 'experts' are wrong about the Monty Hall problem and 8 years later it's still a thing.

This week, strangely, two separate webcomics had panels about the "Monty Hall" problem.  At left is the Saturday Morning Breakfast Cereal comic; the Wondermark (which has the best solution I've ever heard to this question) is at the end of this post.

The "Monty Hall" problem is a riddle that is supposed to help demonstrate probability, I think. It first became 'famous' in 1990, when it was featured in a letter to Marilyn Vos Savant, who runs (ran?) a newspaper column.

Here is the way the problem is formulated: You are a contestant on "Let's Make a Deal." In the final round, you have a pick of 3 doors. Behind 1 is a prize. After you pick one, Monty opens 1 of the other 2 doors, and then gives you a choice: stick with your original door, or opt for the other, as yet unchosen and unopened door.

This problem was first formulated in 1975 or so but was based on an earlier problem from 1959.

Most "experts" say that you should definitely definitely switch.  They reason that when you first picked a door, you had a 1 in 3 chance of picking correctly, but that after Monty opens a door you have a 1 in 2 chance of picking correctly.

That result has been very controversial, as it goes against what our common sense tells us, and even great thinkers have argued with it (Mathematician Paul Erdos is said to have refused to believe the answer was correct until he saw a computer simulation of it.)

I think the answer is incorrect, and that switching does not improve the chances at all. I believe this because I think that the experts are wrong about your initial odds of being correct.

I wrote about this in a post back in 2008. The problem with the 'experts' analysis is that they assume Monty has changed something in your system.

He hasn't. And the reason he hasn't is a combination of what's known as "Magician's Choice" and the kind of thinking behind is that your final answer from Who Wants To Be A Millionaire?

It was from Robert Lynn Asprin's "Myth" books -- silly (but good) books about a magician named "Skeeve" who has various adventures -- that I learned about "magician's choice." "Magician's choice" is making the onlooker choose something without telling them why they're choosing -- giving them the illusion that they're making a choice and controlling the outcome when they are not at all doing that. Suppose I hold up my hands in fists. In the left is a $10 bill, and in the right is nothing. I tell you that I've got a $10 bill in one hand and say "Choose one." You say "Right," and I say "Okay, that's yours. You get nothing, I keep the $10." Now, suppose instead that you say "Left." All I do is say "Okay, that's the one I keep. The right is yours." You still get nothing, and because I run the game, you were always going to get nothing. I made it look like you were getting a choice but you had no choice. The game is rigged.

That's the first thing that helped me crack "The Monty Hall Problem". The second was "Who Wants To Be A Millionaire," which I used to love before it broke my heart. Remember how, when people played, they'd say "A" and Regis would say "Final answer?" and they'd have to say "Final answer" or they could switch? That's pretty important here. 

The reason it's important is because final answer and Magician's choice help demonstrate that  contestants only have the illusion of a 1-in-3 choice at the outset; their choice always 1-in-2 because Monty removes the third choice. That he does this after you've chosen doesn't matter: If you assume Monty will open a door, then the contestant always only had two doors to choose from.

A contestant looking at Doors 1, 2 and 3 thinks he has three choices. But he has only two, in reality, because Monty is going to remove one of those choices. One of the three doors, the one chosen by Monty, is already eliminated from his universe of options. He just doesn't know, yet, which door is eliminated from contention.

This is where Final answer comes in. On Millionaire a contestant could say "A" and Regis would give him a chance to switch: Is that your final answer? When Regis does that, the contestant has the same number of answers to choose from. Assuming he didn't eliminate wrong answers through a lifeline (I miss that show!) he has four possibilities: A, B, C, and D. When he said "A" and Regis says Is that your final answer? the contestant could switch to any of the other three. He has all of his options still open.

But in the Monty Hall problem, when the contestant gets to give a final answer, when given the option to switch, there are only two doors. Which means the contestant never had the option of picking one of the doors. It's as if Regis said "You can pick A, B, or C, but not D." The fact that the contestant doesn't know in advance that one door is ineligible doesn't change the fact that one door is, in fact, ineligible. The Magicians Choice applies: the contestant is given a choice without knowing the conditions of the choice.

A contestant, when looked at this way, makes a preliminary choice -- Door 1, say. Monty then removes Door 2 from the equation, and asks the contestant if he wants to switch.  Because Monty removed door 2 from the equation, it was never available in the first place.

There are other problems with the claim that switching is always best. People who say you should switch focus on the odds that the third, as yet unchosen door, is best based on the change in probabilities.

Probability is calculated by taking the number of outcomes in which the event will occur divided by the total number of possible outcomes, or:

P = n/t

(My nomenclature: P is the probability the event you want to happen, will happen. N is the total number of outcomes in which that event could occur; t is the total number of possible outcomes. So if you roll a die and want to predict the number of times a roll will result in a number higher than four, the probability is 1/3: N is 2 (the numbers 5 and 6) and T is 6 (all possible numbers you could roll. 2/6 = 1/3).

So in the Monty Hall problem, P (the car behind the door) is the event we want to happen. The odds of it being behind any door seem to be 1 in 3, which is where everyone goes wrong. Since 1 door cannot be chosen, the odds you will pick correctly are 1/2.

But even if the initial odds were 1/3 that you'd pick correctly, after Monty chooses, switching presents no better odds. This is because once Monty eliminates a door, he gives you a brand new choice.  This is, again, where you can see that you always only had two choices.  Just as in Final Answer, you are free to switch or stay. It is not a choice of picking the other door. It is a choice of picking either door.

If Contestant picks Door 1, and Monty opens Door 2, and then says Do you want to switch (Final answer?), Contestant now has a choice between two doors.  Phrasing it as do you want to switch helps invoke psychological conditions making it tougher on people (people tend to see value in a choice they've made even where there is none, for example.)

So when Monty says do you want to switch, the odds that the car is behind either door are 1/2. You have the same odds of winning by staying as by switching.

Picture this: I offer to give you $10 if you pick a coin toss correctly. You can have "Heads," or "Tails," or "Both." You're no dummy; you know it can't be both, so you say "Heads." I then say "All right, I'll tell you what. You can't pick "Both." Do you want to switch?" Only an idiot would say "Switch!" based on the assumption that I have somehow changed the odds. Heads and tails were equally likely all along; both was never in play. 

This analysis doesn't change if I flip the coin and hide the result from you. I flip it. You say heads. I say OK, now I'm eliminating the option of you choosing BOTH. Do you want to switch? 

This isn't a perfect comparison (because both cannot occur, so the probability of both is zero) but it helps demonstrate the problem with the experts' solution -- because the door that Monty opens cannot be chosen by the contestant. They just don't know that yet.

So, in "The Monty Hall Problem," your odds of winning are always 1 in 2 from the start. The third door is there to distract you and make you do dumb things like change doors, to give you the illusion that you are choosing more than you really are; you're making one choice between two doors.  It doesn't matter if you switch or not.

No comments: