The expectancy vs surprise continuum: music, art, suicide, and the good life

A rule of thumb about how to live a good life and what makes good content.

There’s a thread that connects sounds in nature, music, suicide and what makes a good film: the expectancy vs surprise continuum. Understanding or at least being aware of this continuum can help us see life through a different lens and helps us make sense of why we enjoy the media we consume and the things we produce and the work we do.


Let’s start with at the bleakest so we can move into the light. Suicide. Often regarded as the forefather of sociology Emile Durkheim wrote about suicide in his treatise Le Suicide that was published in 1897. Durkheim described four types of suicide using two sociological variables: integration and regulation. He argued that too little or too much of either creates conditions which make suicide more likely.

Along the regulation axis sits fatalistic suicides caused by excessive structure and control, such as slaves who are unable to influence the rules under which they must live. On the other side of this axis sits anomic suicide, where there are no rules or clear conventions on how to act and behave, which can lead to a breakdown of social equilibrium. For example, suicide because of bankruptcy or loss of a job or loss of close family members.

The parallels between Durkheim’s work and the expectancy vs surprise continuum are obvious. If too much or too little structure increases the likelihood of suicide, we can surmise that the conditions for the good life lie somewhere in a sweet spot between those two extremes.

Pink noise

Halfway between the entirely uncorrelated random notes of white noise and the entirely correlated drunkard’s walk of brown noise sits pink noise (or 1/f noise or flicker noise). Tunes based on pink noise are moderately correlated over short and long runs. Benoit Mandelbrot was the first to recognise that pink noise and 1/f fluctuations are everywhere in nature, from the annual flood levels of the Nile and variations in sunspots to the wobble of the Earth’s axis and currents in the nervous system of animals and ourselves.

Our perception of the world seems to cluster around pink noise.

From the cradle to the grave our brain is processing the fluctuating data that come to it from its sensors. If we measure this noise at the peripheries of the nervous system under the skin of the fingers it tends, Mandelbrot says, to be white. The closer one gets to the brain, however, the closer the electrical fluctuations approach 1/f. The nervous system seems to act like a complex filtering device, screening out irrelevant elements and processing only the patterns of change that are useful for intelligent behaviour.

Martin Gardner, p.24, Mathematical Games (1979)

The thread between pink noise and music, and the expectancy vs surprise continuum is clear.

It is commonplace in musical criticism to say that we enjoy good music because it offers a mixture of order and surprise. How could it be otherwise? Surprise would not be surprise if there were not sufficient order for us to anticipate what is likely to come next. If we guessed too accurately, say in listening to a tune that is no more than walking up and down the keyboard in one step intervals, there is no surprise at all. Good music, like a person’s life or the pageant of history, is a wondrous mixture of expectation and unanticipated turns.

Martin Gardner, p.28, Mathematical Games (1979)

The same applies to all media we consume, from films and novels to newspaper articles. While the format is similar the content must be sufficiently different to pique and retain our interest. Hollywood can’t be sustained on Marvel sequels and prequels alone. As consumers get bored, new forms will replace them, much like the new schools of art that have emerged over the past 400 years.

Bore out versus burn out

What are the conditions that encourage the state of flow, when work comes easily and minutes drift into hours and we become super productive? According to Steven Kotier, a key psychological trigger is to find the balance between the challenge of the task in hand and our skills and ability to perform that task. We need to find tasks that stretch our abilities to force us into the present, but not too much that we snap.

This bore-out-vs-burn-out dichotomy maps directly on to the expectancy vs surprise continuum.

Kotier suggests that the task should be around 4% greater than the skills one brings to it as a rough heuristic for finding flow. This varies per person. High achievers may blow way past this threshold without any of the motivational reward of flow and risk burn out. While underachievers need to get comfortable with feeling uncomfortable and stretching themselves.

Watch out for

Everyone’s tolerance for structure or surprise varies. And for each of us it waxes and wanes over time.  

You may boost overall happiness and fulfilment by compensating when one area of your life feels overly restrictive, or, on the other side of the ledger, disorderly. For example, a natural creative might feel stifled working for a large, bureaucratic company, but find solace in creative writing or abstract art or, indeed, writing an irreverent blog.


Gardner, Martin. (April 1978), Mathematical Games, Scientific American, Vol. 238, No. 4, pp. 16-33
Jones, Robert Alun. (1986) Emile Durkheim: An Introduction to Four Major Works. Beverly Hills, CA: Sage Publications, Inc. pp. 82-114
Kotier, Steven. (May, 2014), Create a Work Environment That Fosters Flow, Harvard Business Review

Maslow’s Theory of Motivation: the five steps to Nirvana

Everyone has seen some version of this pyramid. But what exactly does it do?

American psychologist Abraham Maslow created his hierarchy of needs to explain what motivates certain behaviours. His theory was that once you’ve largely satisfied one need, it no longer motivates you. The next need in the hierarchy becomes the main motivation, and you feel its absence more noticeably.

Managers can use it to work out what does and doesn’t motivate their team members.

The basic needs are:

  1. Physiological: the most basic needs for survival, such as food and water.
  2. Safety: not just physical security, but also structure, stability and order. This is why people prefer secure jobs and savings accounts to starting a business or investing.
  3. Belongingness and love: friends, family, socialising and working with familiar colleagues. Maslow blamed urbanisation, mobility and the breakdown of traditional groupings for the rise in feelings of alienation and loneliness, half a century ago. Plus ça change.
  4. Esteem: this comes in two parts. The need for respect, recognition, attention and appreciation from others. And the need for self-respect and self-confidence — ultimately, liking yourself.
  5. Self-actualization: a feeling of fulfilment, achieving your potential and being true to your own nature. As Maslow described it, “a musician must make music, an artist must paint, a poet must write”.

To put it another way, if you’re alone on a desert island with plenty of food and water and no obvious dangers, you’ll probably wish you had some friends with you. You won’t think about food or safety much, or yearn to write poetry.

Maslow argued that people who satisfy all five basic needs are more intuitive, creative, curious, calm and open. They lose prejudice and “robot-like conventionality.”

Therefore, employees who are respected and have opportunities to pursue their goals and win recognition will not only be motivated to succeed, but more effective in getting there.

Watch out for

Achieving self-actualization doesn’t mean job done, and they lived happily ever after. There will always be a new goal to reach, a new desire to motivate you. Maybe the poet will one day find they don’t like writing poetry any more. If you expect eternal satisfaction, prepare to be dissatisfied.

You also don’t need 100% achievement in one need to move up to the next. In fact, Maslow said that most people are “partially satisfied in all their basic needs and partially unsatisfied in all their basic needs at the same time.” Of course, it’s possible to move down as well as up the hierarchy as your needs and motivations change over time.

Despite the popularity of the Maslow’s hierarchy, there’s scant recent data to support it…

Contemporary science — specifically Dr. Edward Deci, hundreds of Self-Determination Theory researchers, and thousands of studies — instead points to three universal psychological needs. If you really want to [take] advantage of this new science – rather than focusing on a pyramid of needs – you should focus on: autonomy, relatedness, and competence.

Susan Fowler, (2014, HBR)

Still, the pyramid could be a useful lens through which to assess your own life choices and consider whether the needs of your family and work colleagues are being met.

With thanks to Ivan Edwards who wrote most of this post. Thanks Ivan!


Abulof, U., (2017) Introduction: Why We Need Maslow in the Twenty-First Century.Society 54. Springer.
Bolton, J. & Naybour, P., (2014) The APM Project Management Qualification Study Guide. Association for Project Management.
Fowler, S. (November, 2014) What Maslow’s Hierarchy Won’t Tell You About Motivation. Harvard Business Review
Kremer, W. & Hammond, C., Abraham Maslow and the pyramid that beguiled business (2013), BBC News Magazine.
Maslow, A. H., (1970). Motivation and Personality.Harper & Row.

Dunning-Kruger effect: Are you pickaxing to the plateau or plummeting past the peak?

This is the model that explains why after two hours into an edX Python course you’re stuffing a backpack, searching for motels in central Menlo Park and muttering gibberish to your cat about s3 buckets and a world-beating app that’s the Uber of dentistry.

Still, on the flipside, the model offers solace when you know nothing after six months. Keep going, it gets better.  

This is a phenomenon that, in the words of David Dunning, “visits us all.” It’s not a model that’s about others, or them, it afflicts everyone. It forms part of a wider cognitive bias called naive realism. Our brains are authors of our own reality. We rationalize. Our “self-talk” normally paints us – our ego, what we’ve done, who we are – in a glowing, radiant light.

Unfortunately, this witness cannot be trusted, m’lud.

The first rule of the Dunning-Kruger club is you don’t know you’re a member of the Dunning-Kruger club.

David Dunning

Watch out for

The original effect has morphed incorrectly from the original research. It’s “poor performers are overconfident,” not “beginners are overconfident.”

Try to think in terms of probabilities rather than certainties.

Don’t confuse facts (which can be found true or false) with opinions (that can’t). Opinions can usually be prefaced with words like “I think”, “we should” and “they ought.” They are beliefs. The trouble comes when we bend the facts to fit our opinions. Facts shouldn’t bend. And, yes, that’s an opinion.   

This is the is-ought problem or Hume’s law. This brilliant, animated illustration from BBC Radio 4 read by Harry Shearer (aka Principal Skinner) explains it sweetly in less than 90 seconds. The whole series is great, so go listen.

Also see the will vs skill coaching matrix. Managers will often see this behaviour.


Dunning, D. (2011). The Dunning–Kruger Effect: On Being Ignorant of One’s Own Ignorance. Advances in Experimental Social Psychology, Volume 44, Pages 247-296
Kruger, J. & Dunning, D. (1999). Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology
Resnick, B. (June 26, 2019). An expert on human blind spots gives advice on how to think. Vox

Learn to learn better in 3 steps

With three, brutalist green triangles because they’re memorable.

How can we get better at getting better? How can become more efficient learners? This model offers a useful starting point.

Ulrich Boser sets out three ways we can get better at learning in his Learn Better book.

1. Set goals

Boser argues that at heart learning is often a type of project management, and if we treat it as such – with clearly defined, specific, time-bound tasks – we have more chance of successfully completing the learning “project.” Imagine trying to take on a huge task. For example, trying to teach yourself JavaScript. It can seem like an impossible task, but if you spilt this herculean feat into much smaller, discrete steps such as first completing an introductory online course, secondly replicate a project on Medium etc. it becomes much more achievable and you’re less prone to negative thinking. Afterall, how do you eat an elephant? One bite at a time.

2. Think about thinking

Also known as metacognition, thinking about thinking is about introspection and asking ourselves: have I really understood what I’ve just learned?

When it comes to learning, one of the biggest issues is that people don’t engage in metacognition enough. They don’t stop to ask themselves if they really get a skill or concept.

Ulrich Boser

We can adopt routines and behaviours to force ourselves to do this. For example, can you summarize from a lesson or your reading the salient points in simple language. Could you explain it to your grandmother? This is the Richard Feynman learn-by-teaching technique. When I was studying A-level physics, for each module I honed into a routine of copying out the key learning objectives and the syllabus and wrote out the answers in full against each one. I then condensed the notes onto 5 or 56 sheets of A4 and then condensed them once more onto flash cards. Throughout this condensing stage I would be testing my learning by working through past exam papers. I’m not saying this is the golden technique but it worked well for me, and worked relatively quickly.

3. Reflect on your learning

Get some time and space away from your learning to process it. Again a period of introspection, but less focused than step #2. Look to set aside a moment of reflection when calm. This is what good rest and sleep is all about. Often our best breakthroughs are when we’re not confronting a problem straight on, but obliquely in the shower or on a walk when our mind is wandering.

Watch out for

Boser’s work is underpinned on the principle that learning can be learned -learning is not an innate skill. And there’s research to back this up. For example, Marcel Veenman found that focusing on how we understand is more important than intelligence at achieving mastery.

…intellectual ability uniquely accounts for 10 percent of variance in learning, metacognitive skills uniquely account for 17 percent of variance in learning…

Marcel V. J. Veenman & Bernadette H. A. M. Van Hout-Wolters & Peter Afflerbach


Bosner, U. (2017). Learn Better: Mastering the Skills for Success in Life, Business, and School, or How to Become an Expert in Just about Anything. Rodale
Bosner, U. (2018). Learning Is a Learned Behavior. Here’s How to Get Better at It. Harvard Business Review
Veenman, M., Van Hout-Wolters, B. & Afflerbac, P. (2006) Metacognition and learning: conceptual and methodological considerations. Metacognition Learning. Springer

Always Be Learning

It’s more a credo to live by rather than a mental model, but I’ve categorized it as a mental model. Sue me.

A central tenet in Ray Dalio’s Principles and beloved by Warren Buffett, Charlie Munger, Elon Musk and other highly successful investors and businesspeople, Always Be Learning – or ABL – is about how we learn, and how we learn from our mistakes and our failures.

I constantly see people rise in life who are not the smartest, sometimes not even the most diligent, but they are learning machines. They go to bed every night a little wiser than they were when they got up and boy does that help, particularly when you have a long run ahead of you.

Charile T. Munger

I will write more (and more coherently) on this later. Probably.

It’s better to have tried and failed than to live life wondering what would’ve happened if I had tried.

Alfred Lord Tennyson

Watch out for

It works best with dispassionate reflection, which is easier said than done since we’re programmed to rationalize our mistakes.

It’s not about failing fast. It’s about learning fast.


Baid, G. (2020) The Joys of Compounding: The passionate pursuit of lifelong learning. Columbia University Press
Dalio, R. (2017) Principles. Simon & Schuster