How many times do you come across people in your profession who lack competency but sound more confident? And how many do you come across who are competent but sound less confident?
This isn’t just with others. It is very possible that others might see us as less confident inspite we are competent and vice versa.
As I am into professional development, learning, especially in areas of coaching, business I always wondered why do so many professionals don’t invest in their learning! They spend lakhs to educate their kids but don’t like to invest in their professional development.
That is when I came across the Dunning-Kruger effect which answered this question.
Justin Kruger and David Dunning are psychologists who did a famous experiment back in 1999.
The experiment was conducted on a group of students entering introductory courses into fields they had not studied before. The students were given tests to see how much they already knew about the material and then asked about how well they thought they had done on the tests?
Two interesting results emerged, regardless of the field of study in question. The students that did the best on tests thought they are about the average, and the students that did worst on the tests thought they were above average.
In other words, those who have the low ability tend to feel more confident as they do not have the understanding to see they are lacking the ability (Stupid don’t know how stupid they are)
Conversely, those who performed best feel less confident as they underestimate their ability assuming that everyone is able to perform at a similar level. (Imposter syndrome)
In a way, this explains the negative effects of overconfidence. The adage, “little knowledge is a dangerous thing” applies here.
This is prevalent in most fields but far higher where outcomes have a probabilistic nature than real measurable.
Let me give an example of what it means. In some hard skills like programming, math, it is somewhat easier to know where one is in terms of real learning. A simple test can reveal the competence. So being overconfident can expose quite quickly!!
But when it comes to fields like management, sales, finance, one can ride on confidence without having the backing of necessary learning or competence! The reason is the outcome has many factors involved.
For example in stock trading, many new traders enter the market. They start learning by watching the internet, other pro traders talks, etc. Now as they get some initial success, it is easy to fool oneself that it happened bcos of newly acquired knowledge!
But the reality is, it is can be due to the current trend, market forces, and nothing to do with what trader new learning! Unless the trader can repeat success across multiple cycles of the market, his confidence is not justified and overrides mostly the competence!
Similarly in the career of salespeople, when they start newly or enter new markets if few sales happen, it is easy to fool oneself that due to their competence they clinched the deals. But many times it may be just that they met a needy buyer or the market was good, competition didn’t exist, etc!
Again, unless a salesperson can repeat success across markets and that too for a good time period, it is better to not let confidence grow!
As the Dunning-Kruger experiment showed, those who lack competence exhibit (!over) confidence. This is a mental bias or blind spot. This problem with this overconfidence is, it makes one stop taking feedback from others, the external world, and eventually stop learning!
Being confident is good by itself. But what Dunning-Kruger says, there is a boundary condition that one needs to be aware of w.r.t confidence! Dunning Kruger studies show, when there are new learnings, most fall into to trap of overconfidence.
The Beginners Bubble: It Takes Little Learning For Overconfidence to Develop
How to become proficient and not get carried away by the overconfidence is best explained again by Dunning & Sanchez in their Beginner’s Bubble Hypothesis.
People approach a new task|skill being quite cautious and unconfident in their decisions. Then, after a little learning, they quickly become overconfident (they enter the beginner’s bubble) before going through a correction phase in which performance improves and confidence levels flatten.
Studies also show ‘confidence’ outperforms ‘accuracy’.
The above picture is from studies where participants were asked to perform a task that involved diagnosing patients following multiple steps. Participants were initially given all instructions and training to diagnose patients and record their observations.
They were asked to perform 60 diagnoses. As part of the test, before performing each diagnosis they were asked to rate their confidence levels.
As can be seen from the picture above as initially participants didn’t know the steps and had lower confidence and as they became a little proficient, their confidence shot up! But the study also showed their confidence was high, they induced more errors!
What study also revealed in this test was, over time with multiple errors, participant’s confidence reduced and errors also came down. In general, most continued have errors without realizing it and are always more confident than accurate.
To give an example of people entering a new field like investing in stocks. Most people are a bit cautious when they start investing and are keen to learn at the beginning.
But the moment they get little success, what generally happens is their confidence shoots up, and with little more success quickly it turns to overconfidence.
In this phase of overconfidence, they stop learning, taking feedback from their peers, markets, or following checklists, processes, etc. They believe they know the things in their head!
At this time many hit a plateau or start losing! After this failure, many get discouraged and even leave that stock market itself and switch to other fields!!
But those few who stick on, accept mistakes, and learn from their failures, will do very well moving forward as their learning will improve and confidence is under check!
My stock market experience
If I have to relate my experience, let me give with investments in stocks. Back in 2004 or so, I got interested in stock market investment.
Now, with no real mentors around me, I took to learning the stock market myself. I started learning the value investment of Warren Buffet, Benjamin Graham, Peter Lynch, etc. Started reading 10’s of books. These are all new learning and I found it fascinating!. Now what this learning did to me was 2 fold!
One way, I was gaining a lot of new information and I valued this very highly. What I didn’t perceive was the bias that was developing in me of increased confidence. This confidence got a further boost whenever I got a little success also.
With hindsight I can relate now, the confidence overtook my learning and ultimately as always happens with (!over) confidence, few years down the line in 2007-8 I lost big time in the stock market!!
After that I started reflecting, I could sense that I had actually stopped real learning, stop correcting myself due to my mistakes and was acting too ahead of my experience or knowledge, and lacked practical experience.
The way to become proficient
Sanchez and Dunning’s data suggest that a ‘boundary’ condition for the Dunning-Kruger effect might be that a person is an absolute beginner at a task or skill. This ‘condition’ only lasts a short time after which they become “some of the most vulnerable individuals to the Dunning-Kruger effect”
The blind spot of the Dunning-Kruger effect can prove very expensive as the confidence growing beyond competence is what makes people take up unknown risks!
The (over!) confidence forces to make mistakes but if the mistakes are small, one is saved. If mistakes are big, then it is possible many will find it tough to recover.
Conclusion
Everyone needs to be aware of the mental bias of confidence getting ahead of learning!! Many learned people always highlight this!! But as it is a mental bias, it gets ignored!!
As per Dunning Kruger, the practical way to get over mental bias is, creating an environment where sooner the confidence bubble bursts better!
So, when starting new, be it a business, investing, sales, create an environment where mistakes are allowed early in the game. The cost of mistakes is small if done early and learning is big and also it won’t let confidence grow high!
There is also another way, that is go-to mentors, coaches who have been in the path you wish to take, and who can guide with the right perspectives, and avoid the blindspots early in the game.
Though my experience shows, first-timers prefer not to go to mentors, coaches. Most cherish the experience of doing on own, to experience the kick one gets from (!over) confidence even at cost of actual results!!
Learning new, doing things the first time is like a teenage experience, and as teenager one likes to explore in own way!!
Seeking help from others by itself is a sign of maturity and comes later after repeated setbacks.
Finally developing the right attitude of not getting into the overconfidence trap by being humble and reminding oneself that we know only a little, there is always a lot to learn help.
