Онлайн-инференс generates predictions on demand when an application or user submits an input. Он contrasts with offline inference, where predictions are precomputed and served from storage.
Generating predictions on demand. For example, suppose an app passes input to a model and issues a request for a prediction. A system using online inference responds to the request by running the model (and returning the prediction to the app). Contrast with offline inference. See Production ML systems: Static versus dynamic inference in Machine Learning Crash Course for more information.