Observability: Long Past, Daunting Future

"Observed failure of certain organ systems of the body could predict behavior of the whole person, which could in turn predict behavior in the social body en masse."

Steven Johnson, The Ghost Map (2007)

Our observability timeline starts in the Paleolithic era, evolving over innumerable generations, encumbered by limitations of synthesis and instrumentation.  That human timeline is now joined by our machine partners.

500,000 BCE, Proto-observability.  A shivering Neanderthal man awakened to ashes, remnants of an extinguished fire.  His memory recalled a similar experience from days before, as he traced the pattern of coalescing dark clouds above.  The sky’s rumbling was his alarm.  It was his prehistoric monitor, his noise gate.

1400 CE, Mechanical observability.  Upon a medieval and bloodied landscape, stone masons proudly scoured their fortress keep, having withstood a failed enemy siege the day before.  They saw reparable cracks in the mighty walls, the wounds from catapulted boulders.  But unseen, beneath the superficial damage, was subterranean moisture, eating at a footing, patiently waiting to crumble a vulnerable and critical support column.

1854 CE, Hypothesis-driven observability.  Renowned in Victorian London, a brilliant physician painstakingly sought the cause of a localized cholera epidemic.  It had taken the lives of a half-thousand souls, within a death radius of only three hundred meters.  Germ theory was not formally established, still many years off.  He needed something more than the instrumentation of the day.

2026 CE, Computational observability.  With deep subsystem monitoring, a cloud-minding engineer faced excessive performance degradation, while management and client frustration was turning to anger.  But all software components were blinking green, satisfactorily within tolerance. Her analytical tools scoured the landscape, the APIs, the geographies.  The “unknown unknowns” were new mysteries waiting for AI to solve,

---

From prehistoric survival to debates regarding our own survival, observability’s coinage is steadily widening, implying a mandate of holism.  “Ceteris paribus” becomes less tenable, as variables are hardly ever constant.  The search for root cause runs far and wide in our present day, as digital pulses instigate distant agentic behaviors.

At this moment in observability’s history, information technology monitoring is about to face its most daunting challenge, one that goes beyond avenues of reliability and toward a machine’s human-impacting intention.  Tracing the trigger of suspect technology demands balanced paradigms, of both behavioral analysis and technical levers.  A 19^th century epidemic exemplifies the theme...  

Hard Sciences and Social Sciences, Allies Seeking Truth

Impediments on our analysis have existed since the dawn of humanity.  In prehistoric times, it was reliance on personal observation, limited synthesis.  In medieval times, rudimentary instrumentation was based upon risky trial and error.  And even during the height of the Enlightenment, discrete instruments and technical know-how were still not enough.

Written in 2007, Steven Johnson’s The Ghost Map is not horror fiction.  It is horror reality.  It is a brilliantly researched tale that chronicles the cholera outbreak near London's Broad Street in 1854.  The demon was a contaminated water pump.   

At the time, the so-called miasma (“bad air”) theory was the predisposition of the scientific community.  And bad air had a warning flag -- stench.   But unfortunately, that theory is no less direct, no less useful than the blame factor in late 20th century IT incident management.  Outage frenzies of decades ago were often “a network problem,” even if there were other causes.  Even when resolved issues turned out to be database overflows, application hiccups, or user errors, the Rodney Dangerfield of the OSI stack was always cited: Network Problem.   The lack of specificity is of course unacceptable.  In 1854, it was tragic.      

At the time, most of the health professionals and the bureaucracy believed that cholera was passed through airborne contact, not waterborne.  The faulty consensus was difficult to change, as the cholera outbreak coincided with a stench in the Broad Street region.   Smells were not usually outliers and the center of the unfolding disease was an ordinary neighborhood with a general sense of trust in the local environs: things were working.   Like the view of our present-day, cloud-minding engineer, the infrastructure and associated components were within tolerances.

Johnson’s descriptions of cholera death are difficult to read.   From old to young, elderly to infants, the spread of the disease was elusive, but its signature was diabolical, sometimes beckoning death within hours.   

Those deaths were painstakingly enumerated by physician John Snow, a remarkable scientist who analyzed the sudden up-tick in cholera deaths, by way of the proximate footpaths and contacts occurred near the Broad Street water pump.   His metrics, the resident victims in the area and their symptoms, could not just be explained by “bad air.”   Simply living with one already infected, or walking past the contaminated pump and breathing the air, was not a death knell.  His hard data falsified the air-born theories of the time.

In concert with John Snow was the Reverend Henry Whitehead, a clergyman who was, like most everyone else, biased toward miasma theory.  Yet unlike tales of didactic and bombastic preachers banging the pulpit, Whitehead had an open mind.  His research focused on behavior.  He was a man who delivered last rites to those unfortunate souls who suffered from the disease.  He was the perfect situational detective, a man of the cloth who was a man of the people.  He knew the neighborhood, the families, their day-to-day walks and habits.  

For Snow and Whitehead, observability needed outcome analysis and behavioral knowledge.  The hundreds of souls lost in only two weeks must have had a different root cause than "bad air" miasma.  The two gentlemen balanced two complementary paradigms – the hard science of physiology, and the social science of human behavior.   While Snow documented victim deaths, Whitehead documented their now-gone actions.  Snow considered symptoms, demographics, previous health.  Whitehead considered victims’ routines, where they had walked and where they collected their water.

But their profound research did not penetrate the scientific consensus of the time.  Despite Snow’s meticulous data, strong chemical hypotheses, and Whitehead’s complementary behavioral analysis, institutional pressures were still not moved. 

Old beliefs and rigid tales of superficial experience are hard to break.

Like all tech, the faulty pump was an amoral agent.  Tools and pipes and check valves are mechanical.  They are not nefarious, nor were the limitations in mid-19th century infrastructure monitoring.  They are dumb components in a chain, only as safe as what they pass to the next link.   As stewards of the ecosystem, we assess physical integrity.   As cyber space detectives and technologists, we capture deviant behaviors of our software, not lifeless intent.

But intention will soon become part of the AI lexicon. 

Decision-making, AI’s Future Intent

Ethical observability enters our timeline as the most recent variation on a theme.

Facing a roadblock in real-world scenarios, we cannot predict every use case’s outcome. “Rewindability” is impossible.  With a universe of alternative pathways, blind spots will recur, because the ecosystems change. Incidents are not going to be eliminated, even when the symptoms and observable components are all accountable for.  

Software environments and solutions are evolving in microseconds, under queries, responses, mechanical and physical reactions.  Across diverse AI scenarios.   What cannot be accounted for?  Behaviors among billions, new use cases that are nuanced and subtly different under the prompting, motivational, requestable catalysts.

Agentic AI under one tiny ecosystem may seem like a tiny ant climbing the plant in my garden, a natural nuisance, a call for easy removal.  Is that all it is?  Or is it a harbinger in planet-sized garden, a forest, a jungle?

Our AI training now rests upon ethical intent. How to assign accountability and guarantee that AI systems are transparent in their value weights, transparent among spectrums and cross sections of potentially impacted individuals.

The next frontier is delayering decision-making.  How was a disaster-causing exposure not foreseen? How is progress measured if development is halted when an infinity of futures is possible under an eternally evolving society and humanity?  This is far more onerous than reliance on a policy document.

The future of observability is turning AI models on their heads.  Beginning with the end, deconstruct agentic causation and algorithms, not as mindless bit-like states of 1 or 0, but rather recursive calculations via value-based thinking. "Values" in terms of human benefit derived by machine responses.  Our observability technology and those who lead it shall be experts in both holistic assessment and computational analysis.

Renaissance thinkers are individuals who can dance in the systems sphere just as easily as they can walk along behavioral footpaths – a combination of John Snow and Henry Whitehead.   This is the profile of those assigned to advance the next frontier in observability.    

Currently agentic AI has autonomy based on value assignment.   Here is where the values align to the impact on a human being -- within a setting, protected by guard rails, at satisfactory distance to probabilistic errors.  

In the 19^th century cholera outbreak, statistics were not just numbers.  They were stories of life, behavioral actions, and suffering under faulty systems.

Observability analysis has a new directionality, from outcomes backward, not from systems outward.  Rather than starting inside the system, we start with the real-world consequences.  While the outcomes cross ownership, geography and time, there is a preeminence of factors: human lives.  Across all weightings and purpose, human beings are the first-class, ecosystem passengers. They are never to be collectively quantified but rather individually acknowledged, each having personal protection, veto-proofed and weighted the same as each and every other human being.   That value parameter shall not be overridden.

Is your AI explainable?

Can your AI story-tell its reasoning?

Is it able to communicate this reasoning to an individual who is not steeped in machine learning or computational analysis?

Does AI identify the limits of its own knowledge and conclusions, its decision-making authority, and the consequences of being wrong?  In plain and sobering language?

As a leader, a professional, or executive, can you?

---

Afterword:

Scientific consensus, especially under limited instrumentation, moves laboriously. While the brilliant John Snow had convinced the regulatory board to remove the Broad Street pump handle, thereby saving countless lives, miasma theory was still entrenched. His theory of waterborne cholera was still not accepted by the body politic nor most scientists.

In the mid to late 1860s, epidemiologists slowly, but hesitantly began accepting his theory. But it wasn’t until 1883 that Robert Koch isolated the vibrio cholerae bacterium, validating Snow’s waterborne contagion theory; he was utterly correct in root cause analysis.

Sadly, John Snow’s vindication was posthumous.  He had died of a stroke in 1858, when he was only 45 years old.