Index > Course > 2021-03-02: Everyone is Overconfident, About Everything, All the Time

2021-03-02: Everyone is Overconfident, About Everything, All the Time

Administrative

• Assignment 2 is coming up - the 4th?
  • The rest of it got delayed, it might be delayed
• Assignment 3 is released
• There is a take home test coming up
  • 4 hour time limit
  • it should take 1 hour
  • it might take 2 hours

Chapter 6 Discussion

Heuristics:

• availability: prob of occurrence of event
  • prob(running into a cop on the drive to work)
• representativeness

How to account for subjective probability?

Non-experts show overconfidence. Calibration attempts to account for this.

Chapter 7 discussion

How to perform a probability assessment

• stanford assessment
• “epi” protocol?

How to get reliable, consistent, accurate data from experts?

The Delphi Method:

• ask everybody the question
• compile a report and send it to everybody
• Ask the question again, letting people change their answers.

Availability bias: we only really consider things that we know about.

Anchor and adjust bias: we know this number, and adjust to account for problems that we know about.

PDFs and CDFs

Am I gonna have to turn on mathjax lol

Probability Density function: What is the probability of X happening, given parameter Y? This is the classic bell curve.

Cumulative Density Function: The probability of X happening, given that parameter Y is at or less than some value. It’s the integral of the PDF.

This is going to take tikz in the pdfs. I barely know tikz.

Expert Elicitation

When google fails.

• We have a question
• It’s very hard to measure
• It’s very hard to estimate

It’s time to find an adult who knows what they’re doing. Or lots of them. But we can’t just ask them what they think:

• Bias!
  • The weatherman says it’s going to snow A LOT.
  • (It turns out they get the percentages roughly right, but I’m still convinced they overestimate amounts.)
• The person we’re asking might not be good at statistics
  • The 70th percentile of grilled-cheese doneness is sunflower golden.

Estimation Exercise

A series of questions with numerical answers. No looking things up. Provide your best guess and your 90% confidence interval.

• Example: How many ice cream shops are in the city of Rochester?
  • Probably 40?
  • Almost certainly between 5 and 300
• How long is the Erie canal, in miles?
  • Like, 100 miles?
  • Almost certainly between 5 and 200
  • Answer: 363 miles. (3 students got it)
• How long is the suez canal? (I’ve never heard of this canal)
  • Could be 40 miles
  • Probably between 2 and 100
  • Answer: 120 miles. (6 students got it)
• Area, in square miles, of the city of Rochester
  • I’m guessing a square, 10 miles to a side, so 100 sq miles
  • Probably more than 9sqm, less than 400
  • Answer: 37.1 (4 correct)
• Area, in square miles, of the city of Los Angeles?
  • A square, 50 miles to a side, 2500 sqm
  • More than 100, less than 10,000
  • Answer: 502 sq miles (5)
• What is the population of New York State?
  • 50 million
  • Between 10 mil and 200 mil
  • Answer: 19.5 million (2)
• What is the population of Australia?
  • 10 million
  • 1 mil to 80 mil
  • Answer: 25.3 mil (5)

Question	Correct Intervals	Percentage
1	3	30%
2	6	60%
3	4	40%
4	5	50%
5	2	20%
6	5	50%

If we were good at making confidence intervals, we should capture the correct value 90% of the time. But at best, we saw 60%. Our ranges were too small, we were too confident in our answers.

I captured the right answer in 4 of 6 questions, or 66% of the time.

We were overconfident in the familiar questions, and still overconfident with unfamiliar questions.

This is graphable

We say that infrequent things, never happen. And common things, always happen.

“Everyone is overconfident; in Everything; All the time.” - Prof.

Except weather forecasters, apparently. They predict things, with percentages, every day. And they get quick feedback.

Good job, weatherman.

---

Index > Course > 2021-03-02: Everyone is Overconfident, About Everything, All the Time