Ernie.SG 
One of the things that I am very excited about “The Sound of Stories” project is around the idea of allowing users to generate their own unique drum sounds, then dump them all into an embedding space and have them work with someone else on-site to use an interactive device such as the ERAE Touch to locate their particular sound - sliding, turning knobs, etc. that map onto latent spaces and values - to locate the “coordinate” of their particular sound in an embedding space. And then once they find it, maybe the on-site location will have a huge display that then projects the story that generated that sound, for instance. I thought this will be interesting from an A.I. explainability POV, somewhat inspired by this and this. Though of course the fun twist is that latent spaces may not be directly interpretable - but I still think that it would be a fun and meaningful way to weave these different technologies together on-site.
So much so for this setup, as I was listening to this podcast though, it occurred to me that if we were to reconstruct the brain and its relationship to the world from first principles, if we accept that the true nature of reality is fundamentally unknowable (as suggested by Plato and Kant), the logic seems to follow that all of life is probabilistic so not only is it unreasonable to expect such systems to be deterministic, but that A.I. explainability is a bit of a red herring as well? Why should we expect any further explanation or reason from A.I. when it makes good predictions for the most part, from the perspective of probability and statistics? It almost seems unfair. The only other area that bears such an extra burden of explainability might just be the nature of consciousness itself.
If it is the case that the brain operates top-down, and what we call reality is simply controlled hallucinations that most of us agree on based on our Bayesian inference of what we see in the world and our priors as informed by our culture, environment, beliefs, etc. If we are all Bayesian inference machines now - the fundamental test for any intelligent system, setting aside the problem of consciousness, could just be the accuracy of its predictions; or to take it one step further, how capable it is of in utilising relevant actions from a repertoire to achieve desired goals. In other words, success is its own explanation much as how the march, the progress of scientific theories is measured by how good it is at predictions or how much a theory minimises error.
That fundamental mystery to our brains, its hierarchical predictive processing, how information taken from the world is transformed into electrochemical signals is not all that different to the “black box” that is A.I. at the moment. But it’s not a black box that is entirely opaque to us in the sense that a healthy dose of neuroscience, probability and statistics probably goes a long way in laying the groundwork for accepting these prediction machines as they are, as well as knowing their limitations.
So perhaps the onus for A.I. explainability really lies in the education system?
Originally published on PubPub at erniesg.pubpub.org/pub/6p2l328i.