In this article we would see how we can use the MLflow Registry feature and how can we access the model from the registry using spark apis and pandas api, if you have not gone through previous articles (part1, part2)on MLflow please read them before you go through this article. links are provided above.

The architecture:

The architecture of the article that we are going to implement is shown as below. Here we are going to take a trained model and its artifacts and register them with the help of MLflow registry, push to different stages and access them using pandas API and Spark API.

The Introduction:

MLflow Registry is a component of the MLflow platform, which provides a centralized repository to manage and organize machine learning models, artifacts, and other artifacts produced by the MLflow platform. It provides a set of features for versioning, sharing, and tracking models, including model lineage, model stages, and model version control.

The MLflow Registry serves as a hub for model artifacts, and it enables users to manage their models and collaborate with other team members effectively. It also provides support for model governance, enabling users to track the performance of their models over time, monitor model usage, and ensure that models are compliant with their organization’s policies and regulations.

The Importance:

Some of the key features of the MLflow Registry include:

The Implementation:

If you have followed my last article on tracking there is a command in start_experiment method as below.

mlflow.sklearn.log_model(model, "model", registered_model_name="iris-classification")

log_model method will log the model to MLflow registry only if the mlflow server is attached to back end also observe clearly that log_model is taking registered_mode_name as parameter which will tell mlflow to register this model else model will be just tracked but not registered.

To attach MLflow with back end database (in my case its postgreSQL) by using the below command

once you run the above command with out any issues, we can see the database getting populated with many tables as below

if you are using mlflow ui then file system tracking is enabled where you can not use the registry feature at all but just track all the features. once the registry has got the model registered with it, then we can access it in multiple ways

Once the experiment is run and tracked then a run_uuid is created and you can see it in tracking page of the UI or else you can query the database as below.

Once you have the run id of the experiment run, then you can start using the model as below.

logged_model = 'runs:/4178a1c5129b40c5ac4e147667f4f19d/model'

# Load model as a PyFuncModel.loaded_model = mlflow.pyfunc.load_model(logged_model)

# Predict on a Pandas DataFrame.import pandas as pd

predicted = loaded_model.predict(pd.DataFrame(X_test))print(predicted[:10])

Now let’s see where and how the model is registered from UI, go to tracking page and in model section top right corner see the registered navigation of the model as below or else navigate to model main menu at the top

As soon as the model is registered then then stage is not decided. but a name and version to that registerd model is associated with it. with those name and version we can access the model and this is method 2. the code is shown below.

model_name = "iris-classification"model_version = 1

model = mlflow.pyfunc.load_model( model_uri=f"models:/{model_name}/{model_version}")

y_pred = model.predict(X_test)print(y_pred)

# load model to predict probabilitysklearn_model = mlflow.sklearn.load_model( model_uri=f"models:/{model_name}/{model_version}")y_pred_prob = sklearn_model.predict_proba(X_test)print(y_pred_prob)

Once we are good with our model utilization then we can promote the model to different stages as stage or production either programatically or by using UI as below.

once the above code is run, the version 1 mode is promoted to staging level.

now lets take help of UI to promote version1 to production . click on the version number and you see the below screen

select the stage that we want to push the model into, in our case production. with this our model will be in its final stage and ready to be served to general public.

The Conclusion:

In conclusion, MLflow’s registry feature is a powerful tool for managing the lifecycle of machine learning models. By providing a central repository for storing, versioning, and sharing models, MLflow registry helps teams collaborate more effectively and improve the efficiency of the model development process.

With MLflow registry, data scientists can easily track changes to their models over time, compare performance metrics, and deploy models to production with confidence. The registry also provides a convenient interface for managing model artifacts, including code, data, and trained weights.

Furthermore, MLflow registry supports integration with popular machine learning frameworks, such as TensorFlow and PyTorch, making it a flexible tool for managing models across a wide range of use cases.

Overall, MLflow’s registry feature is a valuable addition to the machine learning toolkit, helping data scientists and engineers streamline their model development workflows and improve the reliability and reproducibility of their models.

stay tuned to my blog for further components in MLflow, also the working code has been pushed to my github along with instruction please fork and play along with it.