A black box model is a model whose internal reasoning or decision process is difficult or impossible for humans to understand directly. Users may observe inputs and outputs, but the path from one to the other lacks transparency or interpretability. Many deep learning models and large language models are treated as black box systems for governance purposes.
A model whose "reasoning" is impossible or difficult for humans to understand. That is, although humans can see how prompts affect responses, humans can't determine exactly how a black box model determines the response. In other words, a black box model is lacking interpretability. Most deep models and large language models are black boxes.