Skip to main content

Batch Endpoint

Overview​

Batch endpoints are used to do asynchronous batch inferencing on large volumes of data. They receive pointers to data and run jobs asynchronously to process the data in parallel on compute clusters. Finally, they store outputs on a data store for further analysis.

You may use them when:

  • you have expensive models or pipelines that require a longer time to run.
  • you want to operationalize machine learning pipelines and reuse components.
  • you need to perform inference over large amounts of data, distributed in multiple files.
  • you don't have low latency requirements.
  • your model's inputs are stored in a Storage Account or in an Azure Machine Learning data asset.
  • you can take advantage of parallelization.

Batch endpoints can be divided int two types:

  • Model deployment
  • Pipeline deployment

Examples​

Model deployment​

warning

WIP

Description

Create and manage a batch endpoint for inferencing

This notebook example, taken from Azure ML, focuses entirely on deploying pre-trained models for asynchronous batch inference.

This notebook does the following:

  1. Creates a batch endpoint using BatchEndpoint.
  2. Registers two models:
    • One trained with PyTorch
    • One trained with TensorFlow/Keras
  3. Creates and configures two batch deployments (ModelBatchDeployment):
    • One for the PyTorch model
    • One for the Keras model
  4. Tests both deployments:
    • Invokes the batch endpoint with sample image data
    • Downloads and inspects the predictions
  5. Switches default deployments
  6. Optionally deletes unused deployments and endpoints

Instructions

  1. Go to the repository folder with the notebook example and associated files and folders:

    • LINK GOES HERE

  2. Copy the files and folders to your development environment in AI platform.

  3. Before running the <NOTEBOOK_NAME> notebook to deploy the pipeline, ensure your environment is properly set up and the few configuration values are filled in.

  4. Follow the instructions in the notebook and run the code cells.

Simple pipeline deployment​

Description

Hello Batch Pipeline Deployment

In this example, we're going to deploy a pipeline component consisting of a simple command job that prints "hello world!". This component requires no inputs or outputs and is the simplest pipeline deployment scenario.

The notebook:

  • Connects to Azure ML workspace
  • Defines a simple pipeline component (hello_batch)
  • Creates a Batch Endpoint
  • Deploys a pipeline component to the batch endpoint using PipelineComponentBatchDeployment
  • Invokes the batch endpoint
  • Monitors job execution and retrieves logs
  • Cleans up the endpoint

Instructions

  1. Go to the repository folder with the notebook example and associated files and folders:

  1. Copy the files and folders to your development environment in AI platform.

  2. Before running the sdk-deploy-and-test.ipynb notebook to deploy the pipeline, ensure your environment is properly set up and the few configuration values (like workspace and compute names) are filled in.

  3. Follow the instructions in the notebook and run the code cells.

Pipeline deployment for batch scoring with preprocessing​

Description

How to deploy a pipeline to perform batch scoring with preprocessing

This is a clear example of deploying a reusable pipeline component to a batch endpoint. This example is taken from the Azure ML repository.

This notebook:

  • Defines a batch scoring pipeline that includes:
  • Preprocessing using a custom transformation model
  • Inference using a registered MLflow model
  • Converts the scoring pipeline into a pipeline component
  • Deploys the pipeline to a Batch Endpoint via PipelineComponentBatchDeployment
  • Tests the deployment by invoking the endpoint with sample data
  • Tracks intermediate steps (preprocessing and scoring) as separate child jobs
  • Provides detailed instructions for managing and monitoring the batch scoring job
  • Cleans up endpoint resources at the end

Instructions

  1. Go to the repository folder with the notebook example and associated files and folders:

  2. Copy the files and folders to your development environment in AI platform.

  3. Before running the sdk-deploy-and-test.ipynb notebook to deploy the pipeline, ensure your environment is properly set up and the few configuration values (like workspace and computeTarget) are filled in.

  4. Follow the instructions in the notebook and run the code cells.