Every writer has her own unique process of putting their thoughts out there. Some are pantsers – they “fly by the seat of their pants,” putting down words as they tumble along the way. Others would be more deliberate about the process of writing – carefully listing out ideas and key messages and then drafting out the story. For both forms of writing, there is a part where you are spontaneously jotting down whatever comes to your mind, which I would call the “Writing phase”. And then you later go in and revise what you’ve written to publish or ship it out to the world. This is what is typically called the “Revision phase” of writing.
On my laptop, you would find scores of spontaneous drafts which have been untouched and never graduated to the stage of getting published. That, I’m afraid, is a discomfort with the revisions – my least favorite part of the writing process.
Writing the first draft is like falling in love. You have discovered a new idea, you are exploring it, there are more connections and concepts spawning in your head which you rush to compose. It is heady , reckless and also liberating in a certain measure to be able to conjure up and give life to ideas out of nothing. Along the way, you also realize that some of them need improvement, and a few could be gems, but you don’t judge them as they arrive. You just keep writing and documenting, and your heart glows with pride as the page fills up with words that are your own creation.
The “Revision phase” gives you the exact opposite feeling. You now have to judge the words you’ve written, and realize that the message that you sought to convey is somehow not there. The flaws are slowly getting revealed, and the feeling is of languidly falling out of love, or being in a marriage.
You have to “kill your darlings” and judge ruthlessly. What is especially painful is culling pieces that you thought were very cleverly written and get attached to, but now seem like clumsy appendages which do not fit in the overall story.
But if you keep going and survive this phase, you would realize that eventually – in the process of revision – you compromise and find a deeper, more long lasting love.
As I now analyze this writing process, what is increasingly becoming clear to me is that the root cause of my lack of fondness for the “Revision phase” is the fear that what I’ve written might not be good enough.
Somewhere in our subconscious self, we are reassured by the thought of being a brilliant work-in progress than a confirmed failure I think.
Alright. The first step to solving a problem is recognizing that it exists.
Fear of failure is normal. If you are worried about failing, then that is a good sign – you will do your best to succeed. Not committing or finishing because of the fear of failure is a problem.
Overcome the fear of revision to powerfully close your story. Diagnosing your own work, uncovering flaws and correcting them is the first step to clarity — and eventual success!
In my job, I work closely with all things Data. And the magical words you’d hear most likely after that is Artificial Intelligence and Machine Learning. We work with clients to help them use Machine Learning – gleaning insights from their data to gain a sustainable advantage in their business.
In other words, we help them discover what they do not know yet with the Data they already have. And that’s made possible with programming and building algorithms that can learn from the past to predict the future.
The concept of teaching algorithms to learn from the past and replicating the future is a powerful one – and a lot of it has roots in observation of human behavior.
Look back at the history of how the various branches of Machine learning and AI evolved. Most of the thinking that contributed to this discipline was around making machines intelligent by mimicking inner workings of a human brain. Dig a little deeper and you would find branches like Neural networks & Reinforcement learning – entire paradigms of Machine learning inspired off human thinking processes.
After having worked in this industry quite a bit, and getting familiar with the inner workings of these algorithms, an insight that struck me was how much of the reverse is true.
Of course, many types of Algorithms have been taught to learn based on how Humans think and learn. However, there is a also lot that we Humans can learn from how algorithms get trained , tested and then and perform in the Real World.
Here are some examples:
You learn from what you Observe: Any machine learning algorithm you develop has this computationally intensive phase called the learning phase. You train the algorithm with a certain set of inputs and outputs – the machine picks up the patterns in the data to build a model of the world. Now, when you give it a new set of data to make a prediction – it generates an output based on the representation of the world that it has built. Isn’t this how real life works? Oftentimes we lament on our lack of ability to respond favorably to unexpected scenarios. The reality is – you always learn from what you observe.
Generalizations from scant Data leads to Overfitting – Developing your life’s principles from scant data gives you an inaccurate representation of reality. When models learn from too little data, then they fall into the peril of Overfitting. What that means is – they perform very well in test scenarios i.e. the environment where they have learnt, but fail miserably in the real world. In real life too, when you develop very strong viewpoints based on little data – it is quite certain that you might be wrong. One observation of that in the workplace is how every person’s world view gets skewed by what they have seen in their previous roles and organizations – with learnings that might not be completely transferable. Hence, if you have a limited perspective and a new world before you – anticipate that you might be wrong. Look for new data that challenges your established beliefs, and that would help you be aware of the biases you have.
Exposing yourself to new Data enriches you to the next level: When a model does not give us good results – there are usually two ways of improving the accuracy. Either you feed the model new data – which is called ‘feature engineering’, or try a new way of looking at the data which is ‘Algorithm selection’. Considering that you’ve done your homework right in the first place, in my experience – ‘Feature engineering’ (almost) always trumps ‘Algorithm selection’. The more relevant data you expose an algorithm to, the better it learns. And the reason that happens is that more and varied data helps the algorithm develop an understanding of a wide variety of scenarios. In real life, the advice you hear is – get out of your comfort zone. So, while the advice is to go ahead and do something that challenges you, what we are really saying is that expose yourself to a situation that you have not dealt with before. More data helps you develop a worldview that is diverse and captures the intricacies that enable superior decision making.
You need many models to map the complexity of the world: With one viewpoint, your understanding of reality is most likely biased. So, don’t depend too much on the opinions of those who are very similar to you. Research, ask questions – seek out diverse viewpoints. Pursue varied opinions because you achieve wisdom through a multiplicity of lenses. Otherwise, if all you know to use is a hammer – everything seems to look like a nail. Taking the parallel from machine learning, we observe that various models perform differently in different data dimensions, and a combination of models usually gives us superior results. So, the learning here is that if you want get a more accurate understanding of reality – think of multiple approaches for solving a problem. “Get a toolbox, not a hammer.”
The world is not Binary: One of my key instincts after years of management experience was to obsessively simplify messaging – get to the heart of the problem and find simple solutions. What I have realized over time is – the world is complex, and working with data and algorithms has helped me appreciate and embrace that complexity. For example, when we build machine learning models – say propensity to upgrade a product, there is usually no single data point that is overwhelmingly predictive of the outcome, but a combination of scores of signals or features that can accurately predict how a customer would behave. Similarly, machine learning also reveals that there can be hundreds of micro-segments in your data – customers with their own unique needs, wants and aspirations, which can be addressed uniquely. The world is not binary, even though we have strong instincts to view it so — ‘We are losing our jobs because immigrants are coming in and taking them’, ‘Equal pay for equal work will solve all women’s problems’. Binary answers are usually not accurate – and can sometimes be downright dangerous.
“Beware of simple ideas and simple solutions. History is full of visionaries who used simple utopian visions to justify terrible actions. Welcome complexity. Combine ideas. Compromise.”
In summary – as researchers and practitioners, we have built AI and Machine Learning systems by replicating the learning processes of human neurons and building patterns in the data that is fed to them. Unknowingly, we might have created a mirror image of real Life in these self-learning systems.
One which powerful, dynamic and feeds not just from Human learnings, but also informs Humans on how to Learn!
My fascination with Algorithms started when I was quite young. In my teens perhaps – when algorithms and the emerging world of computers seemed to be enticing and promising in equal measure. My brush with them began in a high school computer class, when we were introduced to these archaic boxes of off-white bulky computers with a grey or black hard keyboards.
It was the early days, computers were a relatively new invention and being able to see one live in front of us was quite exciting. The first language we learnt was BASIC, and then graduated to more cognitively expensive ones like C and C++. You could make the computer do enchanting things, by giving it the most complex earth shattering instructions and then watch with pleasure as it bends over backwards to do your bidding. Indeed, how dramatic!
And to add to that, these computers were primitive and heated up rather quickly so they needed enclosed and air-tight rooms with air conditioners in what was known as the “Computer department”. If you grew up in a small town with the harsh unforgiving Indian summer – spending time in there was quite a treat. Computer classes were the favorite even among students who didn’t fancy programming.
For myself – I must say that even though I was quite fascinated by the concept of programming, we never really hit it off. I remember having read through the dense “Algorithms and Data Structures” book in my Engineering to capture any nuggets of wisdom that programming would bring. There was a promise , a connection to all the wonderful happenings in the Tech industry. Dramatic advances in technology with a vision to transform the world. However, making loops in my head and if-then-else-break statements began to feel like a chore very soon.
Until, one day, almost ten years later – I discovered the magical world of Machine Learning.
Machine learning, as a concept was a new paradigm where computers do not need to be programmed with explicit instructions about what needs to be done, but can be taught to learn purely by observation. And at the core of it is the concept of Learning. So first you train your algorithms to learn from the past, and with this knowledge of the past learnt primarily by observation, your algorithms can predict the future and take actions. All this may sound very mysterious but there is plain logic and and a lot of math behind all this.
My discovery of Machine Learning was not accidental. I started with reading books and spinning experiments of my own. And slowly, applying these experiments in many work projects exposed me to the inner workings of these digital beasts.
And the more I knew – the more it astounded me. Imagine a machine that can observe how a system has performed in the past and develop a complete knowledge of the system from Day One. The power of this capability is mind boggling , and frightening in equal measure.
Today, this is the core of what I do. And yet, the more I discover it, the more enamored I am by how much of this world fits into certain patterns, and how much of it can be discovered through pure math. It is also surprising to me how these algorithms reveal our hidden beliefs and desires, some of them which we might not be aware of ourselves.
There are very many fears on what this means for our future, and where this technology will take us. And it also raises provocative questions.
What can we learn from these super powerful algorithms?
What are the benefits and limitations of using these technologies?
How do we leverage these technologies without succumbing to the inherent biases they come with ?
What are the key challenges we would face — as strategists, programmers, individuals, society and humankind in general?
There are many versions of answers to these. I am hoping to discover my own answers through these pages.
The view from my window is an ordinary one. Right outside, there are wiry branches of this tree which has lost all its leaves. I do not remember if the leaves were lost in fall or this tree has been stricken like this most of its life.
I look beyond these dry branches and at first sight , there is darkness and the sky is silhouetted with varying shades of black and dark blue. You can see an outline of the Cupertino hills far away, dotted with tiny specks of lights – like fireflies.
Except that they are houses with real people . Million dollar houses nestled in the slopes and carpeted by dense woods and popular trails. And as I observe closely, I can see thousands of them – or perhaps those are streetlights. It is hard to tell.
The Silicon Valley is a valley in the true sense. Take any major freeway or expressway, and you would be able to view hills hugging the horizon. Like this one from my window, where the summits manifest themselves even in complete nightfall.
But in a few minutes, everything would change.
The sun would rise, and then darkness would be transformed with light. Shades of black and dusk blue would suddenly morph into a multitude of colors. The light would reveal endless details in the landscape before me – like nuggets of surprises to color an ordinary day. I have been looking at this view for many months, but every time there is a new detail which emerges,
Like somewhere between me and mountains ranges before me where there is a house which has two very tall palm trees in it. They stand out and next to each other like an Eleven. Is it a sign ? Or like the moments when the sun strikes the houses nestled on the hills, and they sparkle back!. Figment of my imagination ? Or solar panels striking the sun’s rays at an angle ?
Every night, the world outside dies and awakens in the morning with these brilliant details.
My spot on the couch by the window connects me to two different lenses of the world.
As the light fills in and wakes up the world outside me, I can hear stirring sounds from the rooms inside. A trickle of water in the sink.. the faint hum of the microwave singing with morning coffee..the pitter patter of little feet .. tiny fingers that tug at your hair with sleepy good mornings .. The warmth of love enveloping as my dear ones wake up , and embrace a new day
This is the moment my reverie is broken. There is work to be done, to do lists to be completed. Moments of reflection transform into “military moments” – as I began planning my day ahead , identifying and attacking hurdles, problem solving,
It was a cloudy Saturday morning, with the overcast skies threatening to rain any moment. Rain and cold is not a very reassuring combination, hence we decided that a drive through the California waterfront in the protected confines of our car would be a good way to spend the 2nd day on our trip to Monterey.
So after a hearty breakfast , we packed into our six seater and started on the 17 mile drive from Carmel-at-Sea. The 17 mile drive snakes through a picture postcard vignette of the West Coast. You can drive through the Del Monte forest, glassy green golf courses, and breathtaking views of the Pacific ocean kissing precariously perched rocks on the coastline.
Along the drive we stopped to observe landmarks of tourist interest. Bird rock was a boulder with a mysterious attraction to birds – you could see a swathe of them swooning in and enveloping it on the foaming ocean. Ghost trees were a collection of dried out trees which have been smitten by some kind of affliction and stand out as stark reminders of their glorious past.
And among all these sights we came across this legendary landmark on Pebble Beach, the Lone Cypress tree.
So when you get down and stand at the farthest corner of land, you see this single Cypress tree standing out on a shelf separated from the mainland. Strong gusts of wind brush your face, reminding you of the harsh conditions here. A tour guide standing next to us in the crowd observes that this tree is more than 250 years old.
We are transfixed in this powerful moment, and the symbolism of what we see.
You can see the silhouette of the Cypress leaves across the endless backdrop of brilliant blue beyond. If you peer closer, you would be able to see a faint curvature of the earth as the ocean engulfs you on all three sides. It feels like you’ve reached a cliff – go beyond into the ocean and you’d fall off the edge of the earth.
And amongst all this is the Lone Cypress that stands out starkly, almost with an invitation which says – “Look at me, I am still here!” . Like a Howard Roark laughing at the edge of a cliff.
What you do not see is that a few hundred years ago – a bird that plucked the cypress seed and innocently dropped it out on the rock. You do not see a tiny seedling emerging out, unaware of the glory it would be destined to – just by refusing to give up.
What you do not observe is the furious storm that almost ripped this cypress tree away from its roots. Lashed by a hurricane and stricken by lightning flashes, the tree has been downed once but was never out.
You also cannot see the thin transparent wires that hold it upright now.
In one picture you observe individualism. heroism and empathy entwined together.
Grocery lists, invitation lists , travel to-do lists, tasks-of-the-day lists. I summon lists when I have a problem to solve and am not sure where to start off at. Any new assignment comes my way and I begin making lists.
What started off as a harmless way of staying organized, has now become my chosen warfare for attacking problems. And thinking. And writing. Even when I am writing my journal and the words flow into long sentences and paragraphs, jumbled thoughts flowing out as they arrive. Then after a few paragraphs there is a surge of anxiety. I have to stop.
Break it up into paragraphs
Make logical sections
Put the key points in bullets
Thinking in bullets, is what I call it . Which is wonderful if you want to make a point, but can be a handicap when you want to share your thoughts and emotions, articulate a story. Is there a right brain part of you which decimates as the left brain becomes more powerful ?
My worries exacerbated when I observed my writing style. As I braced myself to sit down to write, there was a ruthlessly drilled habit ingrained in my head.
List ideas
Organize them in logical groups
Build connections and make a story
Sort according to the order in which you want to convey your message
Bullet them for clarity of reading
It was this management consulting routine that had been honed over hours spent painstakingly on presentations, many of which were mercilessly ripped apart in reviews with my managers – “You made this long list of recommendations – but what exactly is the story ?”. I always wondered, what can they see that I am not able to see?!
More practice, list-making and exacting reviews made me realize what I was missing. And also that my fears were unfounded. The link between list making, consulting techniques and creative writing, is after-all – the Story!
Take this piece of writing for example:
A quiet room. Sheafs of magazines stacked at the corner of the sofa.. Pillow under the crook of my arm. Two broken pencils, crumbs of wooden shavings, Off-white lighting throwing long shadows in the living room. The cat purring the corner.. This is a lifeless list, but what is the message here. Is it boredom ? is it loneliness ? As writers that’s the story we are trying hard to convey.
A list is a medium to lay out all the facts and data points out there.
A Story is the heart of message – what is it that you are REALLY trying to say? And oftentimes it is the hardest part of writing, because you aren’t really sure of what you are trying to say in the beginning. But if you are lucky, or think hard enough, or keep going at it – you would stumble on the right answer. The Aha moment of your writing.
You suddenly realize – Yes, this is what I want to say!
Thought Experiments 