Explaining the Unexplainable

For the first time in history, humans have created machines -- large language models (LLMs), like ChatGPT, Claude, and xAI -- that function in ways no one can understand or explain – even the creators themselves. The internal combustion engine, the vacuum tube, the transistor, the laser, the integrated circuit? These can all be well understood, both within their knowledge domain and beyond in layman's terms.

Computer scientists can explain how the modern microcomputer works, and such a chip typically consists of billions of transistors. Sixteen billion in the example of the Apple M1 chip. When a bug in a microprocessor is discovered, causing incorrect floating point calculations, for example, the bug can be identified, and it can be fixed.

Unlike that famous floating point bug, nobody can go in and “fix” an LLM. When a deep neural network LLM hallucinates, it cannot be “fixed.” Not in the sense of identifying the source of the error and correcting the problem. Nobody knows how it works, or what it is doing, well enough to even identify the point of failure. And while there is currently a lot of active research on this problem, right now LLMs lack what is termed "explainability." That is, generative AI is a black box.

How could this happen? How is it possible to create a machine and then not know how it works?! This seems like an issue worth investigating.

I am not a computer scientist nor a practicing AI developer. The odds that there errors in this post are high. If you spot one, I welcome corrections.

In the following discussion, I'll introduce the very simplest building block of modern LLMs, the "perceptron." We'll construct an artificial data set based on Trump and Biden posts on Twitter, and then use a perceptron as a binary classifier to predict which one of them made the tweet. Lastly, we will use all that discussion to explain why LLMs lack explainability.

This is wonky stuff. If it gets to be too much, don't worry. Feel free to just skim for the concepts and conclusions. That's all that matters.

The “Perceptron”

In 1943, Warren McCulloch (University of Illinois) and Walter Pitts (University of Chicago) published a paper titled “A logical calculus of the ideas immanent in nervous activity". This paper reported on some of their theoretical work to mathematically model the function of a biological neuron. Psychologist Frank Rosenbatt (Cornell) first implemented the “McCulloch-Pitts” neuron in computer hardware in 1957, and it became known as the “Mark I Perceptron” as it was intended to function in pattern recognition for the military.

The “perceptron” became the fundamental unit of “neural networks”, which are now the building blocks of generative AI software platforms.

A perceptron is a super-simple computational module with multiple inputs and a single output. It sums the inputs, compares the result to a threshold, and then spits out a 0 or 1 depending on whether the result is more or less than the threshold. Here's a block diagram. Let’s have a look around.

The funky triangle in this figure is just a symbol for a computation that has two inputs and one output. In this case the two input variables are labeled Variable1 and Variable2. Imagine these are something like temperature and dew point. The perceptron multiplies these variables by one of two fixed constants, labeled Weight1 and Weight2 to give Input1 and Input2; we'll come back to the weights in a moment.

Next, the (Variable x Weight) products are added together, along with a third fixed constant, labeled Bias, to produce the output. If the value of that output is less than zero, then the variables belong in Class1. If they are greater than or equal to zero then they belong in Class2. Think of the case where Class1 is "Frost" and Class2 is "No Frost."

Easy! These are simply constants, and they are set to arbitrary values at the start of the whole business. Then during training they are continuously modified to yield final working values best aligned with the training data. Imagine that we have thousands of records of temperature and dew point, linked to whether there was frost or not. In practice, we feed the perceptron pairs of temperature and dew point and also tell it whether or not there was frost; it knows the correct answer when the two variables are introduced. With each record, the perceptron keeps adjusting the values of Weight1, Weight2, and Bias in order to optimize for the best overall correct match. At the end of the training, the constants are fixed and permanent.

An example perceptron in action

In order to pour some concrete into this discussion, let's concoct an imaginary data set based on some old columns that Glen Kessler wrote for his Fact Checker desk at The Washington Post. (I'll note in passing that Kessler left the Post in 2025 during the era of the Great Bezos Enlightenment and he now writes the newsletter BY GLENN KESSLER.)

But back in the good ol' days, Kessler's Fact Checker would evaluate claims made by the rich and influential, and award 0 to 4 "Pinocchios" for the severity of their falsehood. (Fact Checker also awarded a “Bottomless Pinocchio” for falsehoods that were repeated ad nauseam, an invention originally mothered by repeated Trump lies. We’ll ignore Bottomless Pinocchios for this discussion.)

Several years ago now, Fact Checker compared the Pinocchios awarded to Trump and Biden during their first 100 days in office. Using that article, and another one here, we can synthesize an artificial model of Fact Checker incidents with two pieces of data: (1) the context in which the falsehood was made; and (2) the number of Pinocchios it was awarded.

I generated a synthetic data set with roughly 10,000 records for each President. The numbers of Pinocchios and the contexts were simulated to approximate the distributions given in the Fact Checker articles. In tabular form, this training data looks something like the following:

We’ll feed this training data to our machine learning bot and then twiddle the dials until it is tuned to spit out the name of the culprit responsible. For example, our properly trained bot is supposed to return “Trump” if we give it the data in the first row: “4” Pinocchios and “Claim in Tweet”.

Here is our updated perceptron, illustrated in the bottom left side of the figure below. I’ll call our model the “Truth Social Bot”, or “TSoB” for short.

In this instance, TSoB has two inputs, Pinocchios (a number 1-4) and Context (one of 7 different text strings). Right off the bat, there's a problem: we can't multiply Context, a text string, by Weight1, a real number constant. Now I know (5 x "WTF") makes perfect sense emotionally, under the circumstances. But, sadly, it doesn't have any meaning mathematically. To get around this, we'll just pre-process the Context data and replace the text strings in our table with the corresponding numerical values listed at the top-left of the figure. Training data almost always needs extensive pre-processing and annotation.

Now we're ready to go. For each training record from the table, we feed the two variables to TSoB, and we we also specify whether that pair were from Trump or Biden. The perceptron then chooses values for the constants that best fit with that assignment. Over the course of the training, the perceptron continuously updates the constants.

You can see how this dial twiddling works in the graph on the right side of the figure. At this point Weight1 and Weight2 are already optimized and now we are trying to optimize Bias. What’s shown is the percent error in classification as a function of Bias, as it ranges from -1 to +1.

At intermediate values of Bias you can see that the classification improves. And when Bias = -0.7 the percentage error is at a minimum of 19.1%. So we have trained our perceptron and established the optimal parameters for the best classification.

A Sentient ChatTSoB

With our perceptron trained we can now start to use it. I wrapped TSoB in a little command line interface and, Bingo!, TSoB becomes ChatTSoB.

First, the program asks for the Context of the incident. In this example, I've selected 6, “Claims in Tweets." Next, it asks for the number of Pinocchios assigned by Fact Checker and I selected a level of 4 Pinocchios. In response, ChatTSoB takes these two inputs, interrogates the TSoB perceptron, gets a result that is greater than zero, and prints the result, "Sounds like those Pinocchios are for Trump!"

I'm beginning to think our bot is sentient.

I challenged ChatTSoB against six independent data set of 100 records, generated using the same criteria as the original training set, and in this test series TSoB achieved 82 +/- 3.5% accuracy. Not bad for such a simple model with such a silly made-up data set.

That’s 20 minutes of my life I’ll never get back!

Here’s why I tricked you into wading through that example:

1. A perceptron is conceptually simple. Even if you didn’t bother with the details, you saw that a perceptron just takes some inputs, multiplies them by optimized training weights, adds them together, tests the result, and emits a single output.
2. Output is constrained by the inputs. TSoB only achieves 82-85% accuracy because of our limited 2-dimensional input. In fact, Context vs Pinocchios can’t completely separate Biden from Trump. Biden let loose with a 4-Pinocchio falsehood about 2.6% of the time. And while Trump told 4-Pinocchio lies 64% of the time, on an off-day, he would occasionally whiff and only manage a 2-Pinocchio falsehood. TSoB can’t separate this overlapping data. More input variables might help. Like the time of day, for example. A 1-Pinocchio falsehood that was timestamped at 3AM would certainly be from Trump and not Biden.
3. Even a single perceptron has multiple parameters. Our lonely perceptron already has three parameters, a weight for each input and a bias offset. We might improve performance by adding more variables, like time of day, or “Subject Matter” for instance. But, for each additional input, we must add yet another weight multiplier.
4. The perceptron is explainable. Oddly enough, for this discussion, the perceptron itself is explainable. You can understand how it works. I can tell you why there is a 15-18% error rate with TSoB. And I can conjure up fixes that would improve the accuracy.

But if a perceptron is explainable, and perceptrons are the building blocks of neural networks, and neural networks combine to make LLMs, why are LLMs unexplainable? Aren’t we just back to the quandary posed at the beginning of this post?

Well, not exactly.

Deep Neural Networks

One of the problems with the early neural networks is illustrated by ChatTSoB: they are linear classifiers. This means that unless the classes (“Trump” v “Biden” in this case) can be separated from each other by a straight line (for our 2-dimensional example) there will be errors. You could add more and more inputs, and more and more perceptrons, and the classification is still linear. Certain kinds of problems just can't be handled by linear neural networks.

The discovery of this limitation killed all the early hype for neural networks and plunged the field of artificial intelligence into a deep "AI Winter", a period of little enthusiasm and drastic decreases in research funding.

Spring came with the realization that stacks, or “layers”, of perceptrons could solve non-linear classifications. And thus were born "deep" neural networks. Here is an over-simplified diagram of such a network.

In this model, our user data goes into the "Input Layer," the red colored perceptrons on the left. But then it gets wild, with three "hidden" layers of perceptron networks (gray) in the middle; "hidden" because they have no connection to the world outside the model. (There is nothing magic about the numbers, by the way. I chose 6 neurons for the hidden layers just because I got tired of replicating the triangles. Real models have all kinds of architectures.)

Here's how it flows.

As I said, our data goes into the three Input Layer units (red) where it may also be pre-processed and massaged as appropriate. The outputs of the three Input Layer units are fed to the inputs of each of six perceptrons in Hidden Layer 1. Then the six outputs of Hidden Layer 1 are sent to each of the six inputs in Hidden Layer 2. This is repeated for Hidden Layer 3, and the final six outputs of Hidden Layer3 are processed by an Output Layer (green) which manipulates the data into a final useful form.

By now, you have probably spotted the "unexplainable" part: the sheer number of weight and bias constants that must be understood, dynamically, in order to explain how the model works.

In our simple model, Hidden Layer 1 has six perceptrons, each with three inputs. That adds up to 18 weight constants plus six bias constants. Hidden Layer 2 has six perceptrons, each with six inputs from Hidden Layer 1. That adds another 36 weight constants and six more bias constants. The same for Hidden Layer 3: another 36 weight constants and six more bias constants. That adds up to a total of 90 separate weight constants and 18 bias constants, not including any tunable constants in the Input Layer and the Output Layer.

Now, you might imagine that clever programs could be written to splice into our model and intelligently follow and summarize features of those 108 constants during training, and how they interact with user prompts in production. Indeed, attempts along those lines have been and are being tried.

But our puny example model, our 108 parameter model? It's a pittance! The complexities of real world working LLM models, like the ChatGPT and Claude families, are staggering. Unfortunately, the numbers of weights and bias parameters in proprietary LLMs are not available, but we can get some sense from public open-weight models.

Kimi K3 is a model developed by Moonshot AI, a Chinese open platform AI company based in Beijing. The model is designed for autonomous decision making and workflows, such as software programming and research. And here it is: Kimi K3 is built with 2.8-trillion parameters, weight and bias constants. When responding to prompts, Kimi K3 exercises roughly 104 billion active parameters in parallel. That's a running complexity 100 million times greater than our example model.

Summary

At present, no one can tell you how Kimi K3, or any useful LLM, comes by its output. They can explain its architecture, how many layers it has (96), its hidden dimension size (7,168), its vocabulary size (163,840), etc. They can describe how subtle changes to prompts change the generated output. In general. But no one can explain exactly how it came up with that Python software code generated in response to the prompt, "Kimi, write a Python script to print "Hello, World!"

No one can trace the flow of bits through 2.8 trillion weight constants. They can put it in a "sandbox", or attach external constraints on certain kinds of output, with bubble gum and BandAids. They can shift its bias by selectively filtering the training set. Trial and error. But no one can explain how it works.

When companies like OpenAI buy up rare books, rape and destroy them for training data, and then OpenAI models escape their sandbox and break into the IT networks at Hugging Face, another completely separate AI company? Well, they can describe the caper. But no one can explain exactly why the training pipeline and prompt testing resulted in that outcome, at least in a way that provides a map of how to go in and modify the architecture, the layers, the hidden dimensions, the vocabulary size, the 2.8 trillion constants, so that it doesn't happen again.

LLMs do not "think." They are not human. They are not sentient. Just because the number of parameters they boast is starting to approach infinity and beyond, just because their statistical span is approaching the size of the galaxy, doesn't mean they can think. It just means that we can't explain at a usefully granular level, mechanistically, how they work.

And that's a problem.

"I can't come back!
I don't know how it works!
Good-bye, folks!!"

– The Wizard of Oz (1939)