Right Choice

Steve is very shy and withdrawn invariably helpful but very little interest in people or in the world of reality. A meek and tidy soul, he has a need for order and structure and a passion for detail.

Which of the following do you find more likely;

Steve is a Librarian?

OR 

Steve is a Farmer?

This is a study conducted by Daniel Kahneman and Amos Trevesky. This work was very popular they won the Nobel prize it was on human judgment. With a focus when these judgments irrationally contradict with laws of probability.

Our example depicts one specific type of irrationality. According to Kahneman and Tversky, after people were given the description of Steve as “Meek and Tidy soul” most say he is more likely to be a Librarian. After all the description aligns with the stereotypical view of a librarian than a Farmer. According to Kahneman and Tversky, this is irrational. It’s not about stereotyping there is something else. Almost nobody incorporates the information you already have in the judgment. The ratio of Farmer to Librarian. Let say it is 1:20. That is  1 Librarian and 20 farmers. I am not expecting someone to have the exact same information but why at least a rough estimate was not considered. Let’s add that bit of information and reason about the question again. So Let’s say we have 10 farmers and 200 farmers.

How many of them are a meek and tidy soul. Some random number. Be it 40%. That is 4 out of 10 are a meek and tidy soul. And similarly, in farmers 10% fit the description. So for us out of 200, this comes to 20.

See this now, 4 Librarians and 20 farmers fit the description of Meek and Tidy soul. Do a small calculation now. What is the probability of a random person among those who fits the description is a librarian is 4/24 isn’t. that is 16.7%. Meaning Steve being a Farmer is 6 times probably than Steve being a Librarian

What does this mean?

Even if you think a librarian is four-time more as likely to fit the description “Meek and Tidy soul”  that is not enough to overcome the fact that there are way more Farmers.

Let pause and think back to what we did here. a bit of reverse engineering.

First we checked out of the population what is the ratio of Liberians to Farmer ie 1/20 or 5%. The prior knowledge we have.

Then we checked off all who fit the description meek and tidy soul how many are the librarian. That is 4/24 and we got the result 16.7%. So we combined 2 pieces of information, the number of Librarian and farmers and trait the meek and tidy soul being common among Librarian and update our prior belief about Steve from 5% confidence to 16.7% confidence

The description Meek and Tidy soul was an evidence a clue for us to know who Steve is?. Steve can be a librarian or Steve can be a Farmer. So we chose to check the hypothesis Steve is a Librarian given the evidence.

"New evidence does not determine your belief in a vacuum It should update your prior beliefs it's important because we collect evidence and use it to build our belief about the world."

This integrating thing is called, you might have heard, Bayesian theorem. Bayes' theorem, named after 18th-century British mathematician Thomas Bayes. The theorem provides a way to revise existing predictions or theories given new or additional evidence. It has a vast application, in Finance for risk assessment, Scientist uses this to validate or invalidate their models based on new data. Any coder here? You can code this and explicitly control a machine's belief. And you do this you become a machine learning expert.

 But more importantly

We should use this when we make a choice or judgment. Our judgment is strongly influenced, unconsciously, by which side we want to win. This shapes how we think about our health, our relationships, how we decide how to vote, what we consider fair or ethical. What's most is the fact that it is unconscious. We can think we're being objective and fair-minded and still wind up ruining the life of an innocent man. We need to cut through their own prejudices and biases and motivations to make good judgments. Whenever you want to make a choice or decision. Pause, step back, and remember the 3 rules of Bayes theorem.

1.        Remember your Priors.

2.        Imagine your theory is wrong

3.        Update your belief incrementally

Courtesy: 3blue and 1brown