How Good Are You At Assessing Evidence?

How good are people at receiving some information and assessing the probabilistic evidence instinctively?  I think people overestimate their ability, as this simple test probably will show.  

In front of you are two fields full of sheep.  Field A has 70 black sheep and 30 white sheep, and Field B has 70 white sheep and 30 black sheep, but you don't know which field is which. You're blindfolded and asked to go into one of the fields and lead 12 sheep out of the gate. You lead out 8 white sheep and 4 black sheep.  From which field is it likely that your selection came, A or B?  I'm guessing you said Field B. You're wise to say Field B, but here's the real question - what percentage of confidence would you say is justifiable in claiming Field B to be a wise estimate?

A) 10%

B) 25%

C) 40%

D) 65%

Remember what you chose.  Now here's another question.  Which of these cities has the distinction of being the world's most populated city?

A) London

B) New York

C) Paris

D) Madrid

Which did you choose?  If you have a good sense of the world populations you would have likely thought 'None of the above' (and your instinct would be right - the actual answer is Shanghai). 

Now returning to the first question - what is the actual answer for the justifiable percentage of confidence in claiming Field B to be a wise choice? - you should have said 'None of the above', because the answer is 98%.  Almost nobody would look at my sheep test and think instinctively that the probability would be that high.  

Here’s something else to consider.  I didn't invent this one - it's a selection task devised by psychologist Peter Wason, to show that people don't naturally think well when logical symbols are required. According to statistics over 90% of you will get this wrong.

You are shown a set of four cards placed on a table, which must conform to the rule "If  P then Q". That means that whenever there is a card with P on one side, the reverse side of the card must show Q. The visible faces of the cards show as follows:

CARD 1 - P

CARD 2 - not-P

CARD 3 - Q

CARD 4 - not-Q

Wason asks which card(s) must you definitely turn over in order to test the truth of the proposition that "If P then Q" holds? Have a think about it for a few seconds. 

If you're one of the 90+% you've probably reasoned that you need card 1 and card 3. You want to make sure that P has a Q on the other side, and equally you want to make sure that Q has a P on the other side.  But that's not right - the cards you need to choose are cards 1 and 4, because the rule is "If P then Q".  So indeed you need to check card 1 (P) to ensure there's a Q on the back, but you also need to check card 4 (not-Q) to ensure there is 'no' P on the back. If there is a P on the other side of card 4, then the rule "If P then Q" has been disobeyed.  Cards 2 and 3 don't need to be touched.

Now what's strange, Wason found, is that although people struggle in this task when using logical symbols, they don't when those symbols are changed to more familiar real life situations, despite the logical connection between facts being exactly the same. For example, if instead of "If P then Q" the rule used is "If you come into my pub and drink alcohol you must be 18 or over" people get that one right. The visible faces of the cards show as follows:

CARD 1 - Age 15

CARD 2 - Drinking coke

CARD 3 - Age 18

CARD 4 - Drinking vodka

When presented with the task in the social context of under age drinking, virtually nobody has any trouble choosing cards 1 and 4, even though the logical connection is exactly the same.

I think a lot of this apparent irrationality is down to the mental shortcuts people take.  At least, one would assume so, given that once things have been explained nobody doubts them any more. It's like Daniel Kahneman's now world famous bat and ball problem:

"A bat and ball cost a dollar and ten cents. The bat costs a dollar more than the ball. How much does the ball cost?"

Almost immediately many people go straight for '10 cents' instead of 'five cents'.

It might also be a case of low hanging fruit. I mean, here's one that's alarmingly obvious that surprisingly some people get wrong. Which of the following is the largest set:

A} The set that contains all words in which the third from last letter is 'i'.
B} The set that contains all words in which the last three letters are 'ing'.

It is obvious that A is the larger set, because set A contains all of set B, but yet not everyone sees it. I think it's because they can easily bring to mind words that end in 'ing' so they somehow fail to assess this correctly.

Here is a further puzzle that's really uncomplicated, but yet it leaves many people giving the wrong answer:

X is looking at Y, but Y is looking at Z. X is married but Z is unmarried. Is a married person looking at an unmarried person?

A} Yes
B} No
C} Cannot be determined

Many people answer C, but that’s not right – the answer is A.  Here’s why.  If Y is married then Y is the married person looking at the unmarried Z. If Y is unmarried then X is the married person looking at the unmarried Y. Either way, you know a married person is looking at an unmarried person.

Try some of these on your friends or work colleagues; you’ll be amazed how often they get their answers wrong.  You’ll also see them admit how obvious the answers were all along, once you’d told them.  I suppose that is the nature of riddles in life; in prospect they often seem tricky, but in retrospect they were obvious all along.  Not only are these puzzles fun – they inform us that we humans are perhaps not as rational and not as instinctively good at assessing probabilistic evidence as we might think.