In a 2014 article in the Atlantic magazine, journalists Katty Kay and Claire Shipman call this difference between men and women a “confidence gap”. Through interviews they conducted for their book Womenomics, they reveal the widespread real-life consequences of this gap. Women managers in the UK are less confident than their male counterparts. Male business school students are four times more likely to negotiate their salaries and ask for more money. Men are more likely to apply for a promotion even when they meet only 60 per cent of the criteria, women typically apply only when they meet 100 per cent of the criteria. Kay and Shipman implicate men’s ability to be honestly overconfident – without faking it – as one reason for the confidence gap.

The other side to the gap, they say, stems from women’s perfectionism and risk aversion. Avoiding competitive environments prevents women from gaining valuable experience that could help them thrive better in those very same environments. It’s a vicious cycle that’s further perpetuated by the negative reactions women face when they actually do compete or assert their confidence. Incidentally, Kay herself realises that she attributes her professional profile to having an English accent in the US, echoing how people who feel like impostors can come up with the most creative reasons to explain away their success.

People have also looked at the implications gender differences in competition could have for leadership in groups. Depending on the stereotypes of the task involved, confidence and self-beliefs can be detrimental to performance in competitive environments. Groups choosing leaders based on claims of past performance are more likely to choose men than women. This is driven by the overconfidence of the men and their biased estimates of past performance, rather than underconfidence on the part of women. Male overconfidence increases if leadership comes with financial incentives. In light of results such as these, it is easy to view overconfidence as a male trait, and perhaps underconfidence as being more female.

Things are not so simple, though. Take, for example, a study comparing the tendency of men and women to choose competitive environments among the Masai in Tanzania and the Khasi in India. People participating could get rewarded for every successful performance on a task. Or they could choose to compete with another person and get rewarded three times as much for each success if they did better than their competitor. The task was a simple one – they had to toss a tennis ball into a bucket from three metres away, with a score based on how many times they succeeded out of ten attempts. The decision to compete or not was made before the people attempted to toss the ball into the bucket. In the more patriarchal Masai society, men were twice as likely to compete as women. In contrast, in the matrilineal Khasi society in Meghalaya, it was women who were more likely to compete than men. Another study used the same task and pay-offs in the Nandi society in Kenya.

Since a woman’s property in this society is inherited only by men, women without sons take on the role of a female husband to a younger woman. The sons of this younger woman then go on to inherit the property of the husband. When economics professor Ignacio Palacios-Huerte ran a competition study similar to those we’ve been talking about, he found that female husbands were just as overconfident as men and were markedly different to the wives. The roles we play in society, it would appear, cast longer shadows than we give them credit for. The social and cultural environment clearly influences how competitive men and women are, and could also have an influence on who ends up being overconfident and underconfident.

Some researchers have sought to explain the differences in confidence between men and women based on the social role played by confidence. The suggestion is that women (at least in some societies) perceive a greater social cost to being overconfident – and prefer to make lower self-estimates rather than risk future effects on their social image. This seems to occur even when there isn’t an explicitly strategic advantage to being underconfident, suggesting it could well be an internalised attitude. Of course, this could be also true for any underconfident person.

If this is true and we do use confidence strategically in social settings, there should be tell-tale signs showing that underconfidence occurs differently in social settings compared to private settings. Studies that set out to look for the social role of underconfidence have found some evidence for it. Take the worse-than-average effect. Does believing that they are worse than average help people by giving them suitable social outcomes or motivating them towards better future performance? Some studies suggest this is true. Believing you are worse than your peers or receiving a negative evaluation often leads immediately to poor performance. However, it could motivate people to seek out helpful advice and do better in the long term. It could be argued that it motivates people to put in more effort. For instance, students who start off at university believing they had made fewer friends than average go on to make more close friends over three months. That’s true even after we take into account the actual number of friends they have to begin with.

The impostor phenomenon also shows a social pattern. People who score highly on the impostor scale make different self-predictions if they think their predictions are going to be publicly discussed compared to when the predictions are anonymous. Their predictions are lower when public and they are more accurate if made anonymously. Impostors thus might not always have a strongly held private belief in their low performance. It is only when they present themselves to others that they differ from non-impostors in how they rate themselves. The evidence points towards their professed underconfidence serving a social function.

Excerpted with permission from Beyond Doubt: Overconfidence and What It Means for Modern Society, Vivek Nityananda, Westland.