Black box AI models arise for one of two reasons: Either their developers make them into black boxes on purpose, or they become black boxes as a by-product of their training. 

Some AI developers and programmers obscure the inner workings of AI tools before releasing them to the public. This tactic is often meant to protect intellectual property. The system’s creators know exactly how it works, but they keep the source code and decision-making process a secret. Many traditional, rule-based AI algorithms are black boxes for this reason.

However, many of the most advanced AI technologies, including generative AI tools, are what one might call “organic black boxes.” The creators of these tools do not intentionally obscure their operations. Rather, the deep learning systems that power these models are so complex that even the creators themselves do not understand exactly what happens inside them.

Deep learning algorithms are a type of machine learning algorithm that uses multilayered neural networks. Where a traditional machine learning model might use a network with one or two layers, deep learning models can have hundreds or even thousands of layers. Each layer contains multiple neurons, which are bundles of code designed to mimic the functions of the human brain.

Deep neural networks can consume and analyze raw, unstructured big data sets with little human intervention. They can take in massive amounts of data, identify patterns, learn from these patterns and use what they learn to generate new outputs, such as images, video and text. 

This capacity for large-scale learning with no supervision enables AI systems to do things like advanced language processing, original content creation and other feats that can seem close to human intelligence.

However, these deep neural networks are inherently opaque. Users—including AI developers—can see what happens at the input and output layers, also called the “visible layers.” They can see the data that goes in and the predictions, classifications or other content that comes out. But they do not know what happens at all the network layers in between, the so-called “hidden layers.”

AI developers broadly know how data moves through each layer of the network, and they have a general sense of what the models do with the data they ingest. But they don’t know all the specifics. For example, they might not know what it means when a certain combination of neurons activates, or exactly how the model finds and combines vector embeddings to respond to a prompt. 

Even open-source AI models that share their underlying code are ultimately black boxes because users still cannot interpret what happens within each layer of the model when it’s active.