computer science kubernetes ai, Schemes and Mind Maps of Computer Applications

Download computer science kubernetes ai and more Schemes and Mind Maps Computer Applications in PDF only on Docsity!

Red Hat OpenShift AI Self-Managed

Monitoring data science models

Monitor your OpenShift AI models for fairness

Last Updated: 2025-10-

Legal Notice

Copyright © 2025 Red Hat, Inc.

The text of and illustrations in this document are licensed by Red Hat under a Creative Commons

Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is

available at

http://creativecommons.org/licenses/by-sa/3.0/

. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must

provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert,

Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift,

Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States

and other countries.

Linux ® is the registered trademark of Linus Torvalds in the United States and other countries.

Java ® is a registered trademark of Oracle and/or its affiliates.

XFS ® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States

and/or other countries.

MySQL ® is a registered trademark of MySQL AB in the United States, the European Union and

other countries.

Node.js ® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the

official Joyent Node.js open source or commercial project.

The OpenStack ® Word Mark and OpenStack logo are either registered trademarks/service marks

or trademarks/service marks of the OpenStack Foundation, in the United States and other

countries and are used with the OpenStack Foundation's permission. We are not affiliated with,

endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.

Abstract

Monitor your machine-learning models in OpenShift AI to ensure they are transparent, fair, and

reliable.

.......................................................................................................................... .......................................................................................................................... .......................................................................................................................... .......................................................................................................................... .......................................................................................................................... ..........................................................................................................................

Table of Contents

CHAPTER 1. OVERVIEW OF MODEL MONITORING CHAPTER 2. CONFIGURING TRUSTYAI 2.1. CONFIGURING MONITORING FOR YOUR MODEL SERVING PLATFORM 2.2. ENABLING THE TRUSTYAI COMPONENT 2.3. CONFIGURING TRUSTYAI WITH A DATABASE 2.4. INSTALLING THE TRUSTYAI SERVICE FOR A PROJECT 2.4.1. Installing the TrustyAI service by using the CLI 2.5. ENABLING TRUSTYAI INTEGRATION WITH KSERVE RAWDEPLOYMENT CHAPTER 3. SETTING UP TRUSTYAI FOR YOUR PROJECT 3.1. AUTHENTICATING THE TRUSTYAI SERVICE 3.2. UPLOADING TRAINING DATA TO TRUSTYAI 3.3. SENDING TRAINING DATA TO TRUSTYAI 3.4. LABELING DATA FIELDS CHAPTER 4. MONITORING MODEL BIAS 4.1. CREATING A BIAS METRIC 4.1.1. Creating a bias metric by using the dashboard 4.1.2. Creating a bias metric by using the CLI 4.1.3. Duplicating a bias metric 4.2. DELETING A BIAS METRIC 4.2.1. Deleting a bias metric by using the dashboard 4.2.2. Deleting a bias metric by using the CLI 4.3. VIEWING BIAS METRICS FOR A MODEL 4.4. USING BIAS METRICS CHAPTER 5. MONITORING DATA DRIFT 5.1. CREATING A DRIFT METRIC 5.1.1. Creating a drift metric by using the CLI 5.2. DELETING A DRIFT METRIC BY USING THE CLI 5.3. VIEWING DATA DRIFT METRICS FOR A MODEL 5.4. USING DRIFT METRICS CHAPTER 6. EVALUATING LARGE LANGUAGE MODELS 6.1. SETTING UP LM-EVAL 6.2. ENABLING EXTERNAL RESOURCE ACCESS FOR LMEVAL JOBS 6.2.1. Enabling online access and remote code execution for LMEval Jobs using the CLI 6.2.2. Updating LMEval job configuration using the web console 6.3. LM-EVAL EVALUATION JOB 6.4. LM-EVAL EVALUATION JOB PROPERTIES 6.4.1. Properties for setting up custom Unitxt cards, templates, or system prompts 6.5. PERFORMING MODEL EVALUATIONS IN THE DASHBOARD 6.6. LM-EVAL SCENARIOS 6.6.1. Accessing Hugging Face models with an environment variable token 6.6.2. Using a custom Unitxt card 6.6.3. Using PVCs as storage 6.6.3.1. Managed PVCs 6.6.3.2. Existing PVCs 6.6.4. Using a KServe Inference Service 6.6.5. Setting up LM-Eval S3 Support 6.6.6. Using LLM-as-a-Judge metrics with LM-Eval 4 5 5 5 6 9 9 12 14 14 15 16 17 19 19 19 21 23 24 24 24 25 26 28 28 28 29 30 31 33 33 34 34 37 38 40 43 44 47 47 48 50 50 51 51 53 55 Table of Contents

Table of Contents

CHAPTER 1. OVERVIEW OF MODEL MONITORING To ensure that machine learning models are transparent, fair, and reliable, data scientists can use TrustyAI in OpenShift AI to monitor and assess their data science models. Data scientists can monitor their data science and machine learning models in OpenShift AI for the following metrics: Bias: Check for unfair patterns or biases in data and model predictions to ensure your model’s decisions are unbiased. Data drift: Detect changes in input data distributions over time by comparing the latest real- world data to the original training data. Comparing the data identifies shifts or deviations that could impact model performance, ensuring that the model remains accurate and reliable. Data scientists can assess their data science and machine learning models in OpenShift AI using the following services: LLM evaluation: Monitor your Large Language Models (LLMs) against a range of metrics, in order to ensure the accuracy and quality of its output. Guardrails: Safeguard text generation inputs and outputs of Large Language Models (LLMs). The Guardrails Orchestrator manages the network requests between the user, the generative model, and the various detector services, and the Guardrails detectors identify and flag content that violates predefined rules, such as the presence of sensitive data, harmful language, or prompt injection attacks, as well as perform standalone detections. Red Hat OpenShift AI Self-Managed 2.25 Monitoring data science models

3. Click the Data Science Cluster tab.
4. Click the default instance name (for example, default-dsc) to open the instance details page.
5. Click the YAML tab to show the instance specifications.
6. In the spec:components section, set the managementState field for the trustyai component to Managed : trustyai: managementState: Managed
7. Click Save.

Verification

Check the status of the trustyai-service-operator pod:

1. In the OpenShift console, from the Project list, select redhat-ods-applications.
2. Click Workloads → Deployments.
3. Search for the trustyai-service-operator-controller-manager deployment. Check the status: a. Click the deployment name to open the deployment details page. b. Click the Pods tab. c. View the pod status. When the status of the trustyai-service-operator-controller-manager- <pod-id> pod is Running, the pod is ready to use. 2.3. CONFIGURING TRUSTYAI WITH A DATABASE If you have a relational database in your OpenShift cluster such as MySQL or MariaDB, you can configure TrustyAI to use your database instead of a persistent volume claim (PVC). Using a database instead of a PVC for storage can improve scalability, performance, and data management in TrustyAI. Provide TrustyAI with a database configuration secret before deployment. You can create a secret or specify the name of an existing Kubernetes secret within your project. Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have installed the OpenShift CLI ( oc ) as described in the appropriate documentation for your cluster: Installing the OpenShift CLI for OpenShift Container Platform Installing the OpenShift CLI for Red Hat OpenShift Service on AWS You have enabled the TrustyAI component, as described in Enabling the TrustyAI component. The data scientist has created a data science project, as described in Creating a data science project, that contains the models that the data scientist wants to monitor. If you are configuring the TrustyAI service with an external MySQL database, your database Red Hat OpenShift AI Self-Managed 2.25 Monitoring data science models

If you are configuring the TrustyAI service with an external MySQL database, your database must already be in your cluster and use at least MySQL version 5.x. However, Red Hat recommends that you use MySQL version 8.x. If you are configuring the TrustyAI service with a MariaDB database, your database must already be in your cluster and use MariaDB version 10.3 or later. However, Red Hat recommends that you use at least MariaDB version 10.5.

NOTE

The transport security layer (TLS) protocol does not work with the MariaDB operator 0.29 or later versions. Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI ( oc ) as shown in the following example: $ oc login <openshift_cluster_url> -u <admin_username> -p
2. Optional: If you want to use a TLS connection between TrustyAI and the database, create a TrustyAI service database TLS secret that uses the same certificates that you want to use for the database. a. Create a YAML file to contain your TLS secret and add the following code: apiVersion: v kind: Secret metadata: name: <service_name> -db-tls type: kubernetes.io/tls data: tls.crt: | tls.key: | b. Save the file with the file name <service_name> -db-tls.yaml. For example, if your service name is trustyai-service , save the file as trustyai-service-db-tls.yaml. c. Apply the YAML file in the data science project that contains the models that the data scientist wants to monitor: $ oc apply -f <service_name> -db-tls.yaml -n <project_name>
3. Create a secret (or specify an existing one) that has your database credentials. a. Create a YAML file to contain your secret and add the following code: apiVersion: v kind: Secret metadata: name: db-credentials type: Opaque CHAPTER 2. CONFIGURING TRUSTYAI

format: "DATABASE" 3 folder: "/inputs" 4 size: "1Gi" 5 databaseConfigurations: <database_secret_credentials> 6 data: filename: "data.csv" 7 metrics: schedule: "5s" 8 Set to true to prompt the migration from PVC to database storage. The name of the TrustyAI service instance. The storage format for the data. Set this field to DATABASE. The location within the PVC where you were storing the data. This must match the value specified in the existing CR. The size of the data to request. The name of the secret with your database credentials that you created in an earlier step. For example, db-credentials. The suffix for the existing stored data files. This must match the value specified in the existing CR. The interval at which to calculate the metrics. The default is 5s. The duration is specified with the ISO-8601 format. For example, 5s for 5 seconds, 5m for 5 minutes, and 5h for 5 hours. b. Save the file. For example, trustyai_crd.yaml. c. Apply the new TrustyAI service CR to the data science project that contains the models that the data scientist wants to monitor: $ oc apply -f trustyai_crd.yaml -n <project_name> 2.4. INSTALLING THE TRUSTYAI SERVICE FOR A PROJECT Install the TrustyAI service on a data science project to provide access to its features for all models deployed within that project. An instance of the TrustyAI service is required for each data science project, or namespace, that contains models that the data scientists want to monitor.

NOTE

Install only one instance of the TrustyAI service in a project. Multiple instances in the same project can result in unexpected behavior. Installing TrustyAI into a namespace where non-OVMS models are deployed can cause errors in the TrustyAI service.

2.4.1. Installing the TrustyAI service by using the CLI

CHAPTER 2. CONFIGURING TRUSTYAI

You can use the OpenShift CLI ( oc ) to install an instance of the TrustyAI service. Prerequisites You have cluster administrator privileges for your OpenShift cluster. You have installed the OpenShift CLI ( oc ) as described in the appropriate documentation for your cluster: Installing the OpenShift CLI for OpenShift Container Platform Installing the OpenShift CLI for Red Hat OpenShift Service on AWS You have configured monitoring for the model serving platform, as described in Configuring monitoring for the multi-model serving platform. You have enabled the TrustyAI component, as described in Enabling the TrustyAI component. If you are using TrustyAI with a database instead of PVC, you have configured TrustyAI to use the database, as described in Configuring TrustyAI with a database. The data scientist has created a data science project, as described in Creating a data science project, that contains the models that the data scientist wants to monitor. Procedure

1. Open a new terminal window.
2. Follow these steps to log in to your OpenShift cluster as a cluster administrator: a. In the OpenShift web console, click your user name and select Copy login command. b. After you have logged in, click Display token. c. Copy the Log in with this token command and paste it in the OpenShift CLI ( oc ). $ oc login --token= --server= <openshift_cluster_url>
3. Navigate to the data science project that contains the models that the data scientist wants to monitor. oc project <project_name> For example: oc project my-project
4. Create a TrustyAIService custom resource (CR) file, for example trustyai_crd.yaml :

Example CR file for TrustyAI using a database

apiVersion: trustyai.opendatahub.io/v kind: TrustyAIService metadata: name: trustyai-service 1 Red Hat OpenShift AI Self-Managed 2.25 Monitoring data science models

The interval at which to calculate the metrics. The default is 5s. The duration is specified with the ISO-8601 format. For example, 5s for 5 seconds, 5m for 5 minutes, and 5h for 5 (Optional) The observation’s historical window size to use for metrics calculation. The default is 5000 , which means that the metrics are calculated using the 5,000 latest inferences.

5. Add the TrustyAI service’s CR to your project: oc apply -f trustyai_crd.yaml This command returns output similar to the following: trusty-service created

Verification

Verify that you installed the TrustyAI service: oc get pods | grep trustyai You should see a response similar to the following: trustyai-service-5d45b5884f-96h5z 1/1 Running 2.5. ENABLING TRUSTYAI INTEGRATION WITH KSERVE RAWDEPLOYMENT TrustyAI can be used with KServe RawDeployment and Knative Serverless deployment modes. For more information about deployment modes, see About KServe deployment modes. To use the TrustyAI service with KServe RawDeployment mode, you must first update the KServe ConfigMap, then create another ConfigMap in your model’s namespace to hold the Certificate Authority (CA) certificate.

NOTE

Knative Serverless deployment mode does not require this additional setup. Prerequisites You have installed Red Hat OpenShift AI. You have cluster administrator privileges for your OpenShift AI cluster. You have access to a data science cluster that has TrustyAI enabled. You have enabled the model serving platform in either KServe RawDeployment or Knative Serverless mode. Procedure Red Hat OpenShift AI Self-Managed 2.25 Monitoring data science models

1. Update the KServe ConfigMap ( inferenceservice-config ) in the OpenShift AI UI: a. From the OpenShift console, click Workloads → ConfigMaps. b. From the project drop-down list, select the redhat-ods-applications namespace. c. Find the inferenceservice-config ConfigMap. d. Click the options menu (⋮) for that ConfigMap, and then click Edit ConfigMap. e. Add the following parameters to the logger key: f. Click Save.
2. Create a ConfigMap in your model’s namespace to hold the CA certificate: a. Click Create Config Map. b. Enter the following code in the created ConfigMap:
3. Click Save.

NOTE

The caBundle name can be any valid Kubernetes name, as long as it matches the name you used in the KServe ConfigMap. The caCertFile needs to match the cert name available in the CA bundle.

Verification

When you send inferences to your KServe Raw model, TrustyAI acknowledges the data capture in the output logs.

NOTE

If you do not observe any data on the Trusty AI logs, complete these configuration steps and redeploy the pod. "caBundle": "kserve-logger-ca-bundle", "caCertFile": "service-ca.crt", "tlsSkipVerify": false apiVersion: v 1 kind: ConfigMap metadata: name: kserve-logger-ca-bundle namespace: <your-model-namespace> annotations: service.beta.openshift.io/inject-cabundle: "true" data: {} CHAPTER 2. CONFIGURING TRUSTYAI

echo $TOKEN

Next step

When running curl commands, pass the token to the TrustyAI service using the Authorization header. For example: curl -H "Authorization: Bearer $TOKEN" $TRUSTY_ROUTE 3.2. UPLOADING TRAINING DATA TO TRUSTYAI Upload training data to use with TrustyAI for bias monitoring or data drift detection. Prerequisites Your cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. You have model training data to upload. You authenticated the TrustyAI service as described in Authenticating the TrustyAI service. Procedure

1. Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service in your project: TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})
2. Send the training data to the /data/upload endpoint: curl -sk $TRUSTY_ROUTE/data/upload
   --header 'Authorization: Bearer ${TOKEN}'
   --header 'Content-Type: application/json'
   -d @data/training_data.json The following message is displayed if the upload was successful: 1000 datapoints successfully added to gaussian-credit-model data. Verification Verify that TrustyAI has received the data via the /info endpoint by inputting this query: curl -H 'Authorization: Bearer ${TOKEN}'
   $TRUSTY_ROUTE/info | jq ".[0].data" The output returns a json file containing the following information for the model: The names, data types, and positions of fields in the input and output. The observed values that these fields take. This value is usually null because there are too many unique feature values to enumerate. The total number of input-output pairs observed. It should be 1000. CHAPTER 3. SETTING UP TRUSTYAI FOR YOUR PROJECT

3.3. SENDING TRAINING DATA TO TRUSTYAI To use TrustyAI for bias monitoring or data drift detection, you must send training data for your model to TrustyAI. Prerequisites Your OpenShift cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. You authenticated the TrustyAI service as described in Authenticating the TrustyAI service. You have uploaded model training data to TrustyAI. Your deployed model is registered with TrustyAI. Verify that the TrustyAI service has registered your deployed model, as follows:

1. In the OpenShift web console, navigate to Workloads → Pods.
2. From the project list, select the project that contains your deployed model.
3. Select the pod for your serving platform (for example, modelmesh-serving-ovms-1.x- xxxxx ).
4. On the Environment tab, verify that the MM_PAYLOAD_PROCESSORS environment variable is set. Procedure
5. Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service pod. TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})
6. Get the inference endpoints for the deployed model, as described in Accessing the inference endpoint for a deployed model.
7. Send data to this endpoint. For more information, see the KServe v2 Inference Protocol documentation.

Verification

Follow these steps to view cluster metrics and verify that TrustyAI is receiving data.

1. Log in to the OpenShift web console.
2. Switch to the Developer perspective.
3. In the left menu, click Observe.
4. On the Metrics page, click the Select query list and then select Custom query.
5. In the Expression field, enter trustyai_model_observations_total and press Enter. Your model should be listed and reporting observed inferences.
6. Optional: Select a time range from the list above the graph. For example, select 5m. Red Hat OpenShift AI Self-Managed 2.25 Monitoring data science models