Process ID

MLE.1

Process name

Machine Learning Requirements Analysis

Process purpose

The purpose is to refine the machine learning-related software requirements into a set of ML requirements.

Process outcomes

1) the ML requirements, including ML data requirements, are identified and specified based on the software requirements and the components of the software architecture

2) ML requirements are structured and prioritized

3) ML requirements are analyzed for correctness and verifiability

4) the impact of ML requirements on the ML operating environment is analyzed

5) consistency and bidirectional traceability are established between ML requirements and software requirements, and between ML requirements and software architecture

6) the ML requirements are agreed and communicated to all affected parties

Base practices

MLE.1.BP1: Specify ML requirements. Use the software requirements and the software architecture to identify and specify functional and non-functional ML requirements, as well as ML data requirements specifying data characteristics and their expected distributions.

Note 1: Non-functional requirements may include relevant characteristics of the ODD and KPIs as robustness, performance and level of trustworthiness.

Note 2: The ML data requirements are input for SUP.11 Machine Learning Data Management but also for other MLE processes.

Note 3: In case of ML development only, stakeholder requirements represent the software requirements.

MLE.1.BP2: Structure ML requirements. Structure and prioritize the ML requirements.

Note 4: Examples for structuring criteria can be grouping (e.g. by functionality) or variants identification.

Note 5: Prioritization can be done according to project or stakeholder needs via e.g. definition of release scopes. Refer to SPL.2.BP1.

MLE.1.BP3: Analyze ML requirements. Analyze the specified ML requirements including their interdependencies to ensure correctness, technical feasibility, and ability for machine learning model testing, and to support project management regarding project estimates.

Note 6: See MAN.3.BP3 for project feasibility and MAN.3.BP5 for project estimates.

MLE.1.BP4: Analyze the impact on the ML operating environment. Analyze the impact that the ML requirements will have on interfaces of software components and the ML operating environment.

Note 7: The ML operating environment is defined as the infrastructure and information which both the trained ML model and the deployed ML model need for execution.

MLE.1.BP5: Ensure consistency and establish bidirectional traceability. Ensure consistency and establish bidirectional traceability between ML requirements and software requirements and between ML requirements and the software architecture.

Note 8: Bidirectional traceability supports consistency, facilitates impact analyses of change requests, and verification coverage demonstration.

Note 9: Redundant traceability is not intended, but at least one out of the given traceability paths.

MLE.1.BP6: Communicate agreed ML requirements. Communicate the agreed ML requirements, and the results of the impact analysis on the ML operating environment to all affected parties.

MLE.1 – Machine Learning Requirements Analysis

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Outcome 6

Output Information Items

17-00 Requirement

X

X

17-54 Requirement attribute

X

X

13-52 Communication evidence

X

13-51 Consistency evidence

X

15-51 Analysis results

X

X

Base Practices

BP1: Specify ML requirements

X

BP2: Structure ML requirements

X

BP3: Analyze ML requirements

X

BP4: Analyze the impact on the ML operating environment

X

BP5: Ensure consistency and establish bidirectional traceability

X

BP6: Communicate agreed ML requirements

X

Process ID

MLE.2

Process name

Machine Learning Architecture

Process purpose

The purpose is to establish an ML architecture supporting training and deployment, consistent with the ML requirements, and to evaluate the ML architecture against defined criteria.

Process outcomes

1) an ML architecture is developed

2) hyperparameter ranges and initial values are determined as a basis for the training

3) evaluation of ML architectural elements is conducted

4) interfaces of the ML architectural elements are defined

5) resource consumption objectives for the ML architectural elements are defined

6) consistency and bidirectional traceability are established between the ML architectural elements and the ML requirements

7) the ML architecture is agreed and communicated to all affected parties

Base practices

MLE.2.BP1: Develop ML architecture. Develop and document the ML architecture that specifies ML architectural elements including details of the ML model, pre- and postprocessing, and hyperparameters which are required to create, train, test, and deploy the ML model.

Note 1: Necessary details of the ML model may include layers, activation functions, and backpropagation. The level of detail of the ML model may not need to cover aspects like single neurons.

Note 2: The details of the ML model may differ between the ML model used during training and the deployed ML model.

MLE.2.BP2: Determine hyperparameter ranges and initial values. Determine and document the hyperparameter ranges and the initial values as a basis for the training.

MLE.2.BP3: Evaluate ML architectural elements. Define evaluation criteria for the ML architectural elements. Evaluate ML architectural elements according to the defined criteria.

Note 3: Trustworthiness and explainability might be criteria for the evaluation of the ML architectural elements.

MLE.2.BP4: Define interfaces of the ML architectural elements. Determine and document the internal and external interfaces of each ML architectural element including its interfaces to related software components.

MLE.2.BP5: Define resource consumption objectives for the ML architectural elements. Determine and document the resource consumption objectives for all relevant ML architectural elements during training and deployment.

MLE.2.BP6: Ensure consistency and establish bidirectional traceability. Ensure consistency and establish bidirectional traceability between the ML architectural elements and the ML requirements.

Note 4: Bidirectional traceability supports consistency, and facilitates impact analyses of change requests, and verification coverage demonstration.

Note 5: The bidirectional traceability should be established on a reasonable level of abstraction to the ML architectural elements.

MLE.2.BP7: Communicate agreed ML architecture. Inform all affected parties about the agreed ML architecture including the details of the ML model and the initial hyperparameter values.

MLE.2 – Machine Learning Architecture

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Outcome 6

Outcome 7

Output Information Items

04-51 ML architecture

X

X

X

X

X

13-52 Communication evidence

X

13-51 Consistency evidence

X

01-54 Hyperparameter

X

X

15-51 Analysis results

X

X

Base Practices

BP1: Develop ML architecture

X

BP2: Determine hyperparameter ranges and initial values.

X

BP3: Evaluate ML architectural elements

X

BP4: Define interfaces of the ML architectural elements

X

BP5: Define resource consumption objectives for the ML architectural elements

X

BP6: Ensure consistency and establish bidirectional traceability

X

BP7: Communicate agreed ML architecture

X

Process ID

MLE.3

Process name

Machine Learning Training

Process purpose

The purpose is to optimize the ML model to meet the defined ML requirements.

Process outcomes

1) an ML training and validation approach is specified

2) the data set for ML training and ML validation is created

3) the ML model, including hyperparameter values, is optimized to meet the defined ML requirements

4) consistency and bidirectional traceability are established between the ML training and validation data set and the ML data requirements

5) results of optimization are summarized, and the trained ML model is agreed and communicated to all affected parties

Base practices

MLE.3.BP1: Specify ML training and validation approach. Specify an approach which supports the training and validation of the ML model to meet the defined ML requirements. The ML training and validation approach includes

 entry and exit criteria of the training and validation

 approaches for hyperparameter tuning / optimization

 approach for data set creation and modification

 training and validation environment

Note 1: The ML training and validation approach may include random dropout and other robustification methods.

Note 2: ML validation is the optimization of the hyperparameters during Machine Learning Training (MLE.3). The term “validation” has a different meaning than VAL.1.

Note 3: The training environment should reflect the environment of the deployed model.

MLE.3.BP2: Create ML training and validation data set. Select data from the ML data collection provided by SUP.11 and assign them to the data set for training and validation of the ML model according to the specified ML training and validation approach.

Note 4: The ML training and validation data set may include corner cases, unexpected cases, and normal cases depending on the ML requirements.

Note 5: A separated data set for training and validation might not be required in some cases (e.g., k- fold cross validation, no optimization of hyperparameters).

MLE.3.BP3: Create and optimize ML model. Create the ML model according to the ML architecture and train it, using the identified ML training and validation data set according to the

ML training and validation approach to meet the defined ML requirements, and training and validation exit criteria.

MLE.3.BP4: Ensure consistency and establish bidirectional traceability. Ensure consistency and establish bidirectional traceability between the ML training and validation data set and the ML data requirements.

Note 6: Bidirectional traceability supports consistency and facilitates impact analyses of change requests.

MLE.3.BP5: Summarize and communicate agreed trained ML model. Summarize the results of the optimization and inform all affected parties about the agreed trained ML model.

MLE.3 – Machine Learning Training

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Output Information Items

08-65 ML training and validation approach

X

03-51 ML data set

X

01-53 Trained ML model

X

01-54 Hyperparameter

X

13-51 Consistency evidence

X

13-52 Communication evidence

X

Base Practices

BP1: Specify ML training and validation approach

X

BP2: Create ML training and validation data set

X

BP3: Create and optimize ML model

X

BP4: Ensure consistency and establish bidirectional traceability

X

BP5: Summarize and communicate agreed trained ML model

X

Process ID

MLE.4

Process name

Machine Learning Model Testing

Process purpose

The purpose is to ensure compliance of the trained ML model and the deployed ML model with the ML requirements.

Process outcomes

1) an ML test approach is defined

2) an ML test data set is created

3) the trained ML model is tested

4) the deployed ML model is derived from the trained ML model and tested

5) consistency and bidirectional traceability are established between the ML test approach and the ML requirements, and the ML test data set and the ML data requirements, and the ML test approach and ML test results

6) results of the ML model testing are summarized and communicated with the deployed ML model to all affected parties

Base practices

MLE.4.BP1: Specify an ML test approach. Specify an ML test approach suitable to provide evidence for compliance of the trained ML model and the deployed ML model with the ML requirements. The ML test approach includes

ML test scenarios with distribution of data characteristics (e.g., gender, weather conditions, street conditions within the ODD) defined by ML requirements;

 distribution and frequency of each ML test scenario inside the ML test data set

 expected test result per test datum

 entry and exit criteria of the testing

 approach for data set creation and modification

 the required testing infrastructure and environment setup

Note 1: Expected test result per test datum might require labeling of test data to support comparison of output of the ML model with the expected output.

Note 2: Test datum is the smallest amount of data which is processed by the ML model into only one output. E.g., one image in photo processing or an audio sequence in voice recognition.

MLE.4.BP2: Create ML test data set. Create the ML test data set needed for testing of the trained ML model and testing of the deployed ML model from the ML data collection provided by SUP.11 considering the ML test approach. The ML test data set shall not be used for training.

Note 3: The ML test data set for the trained ML model might differ from the test data set of the deployed ML model.

Note 4: Additional data sets might be used for special purposes like assurance of safety, fairness, robustness.

MLE.4.BP3: Test trained ML model. Test the trained ML model according to the ML test approach using the created ML test data set. Record and evaluate the test results and logs.

Note 5: Evaluation of test logs might include pattern analysis of failed test data to support e.g., trustworthiness.

MLE.4.BP4: Derive deployed ML model. Derive the deployed ML model from the trained ML model according to the ML architecture. The deployed ML model shall be used for testing and delivery to software integration.

Note 6: The deployed ML model will be integrated into the target system and may differ from the trained ML model which often requires powerful hardware and uses interpretative languages.

MLE4.BP5: Test deployed ML model. Test the deployed ML model according to the ML test approach using the created ML test data set. Record and evaluate the test results and logs.

MLE.4.BP6: Ensure consistency and establish bidirectional traceability. Ensure consistency and establish bidirectional traceability between the ML test approach and the ML requirements, and the ML test data set and the ML data requirements, and the ML test approach and ML test results.

Note 7: Bidirectional traceability supports consistency, and facilitates impact analyses of change requests, and verification coverage demonstration.

MLE.4.BP7: Summarize and communicate results. Summarize the test results of the ML model. Inform all affected parties about the agreed results and the deployed ML model.

MLE.4 – Machine Learning Model Testing

Outcome 1

Outcome 2

Outcome 3

Outcome 4

Outcome 5

Outcome 6

Output Information Items

08-64 ML test approach

X

03-51 ML data set

X

13-50 Test result

X

X

11-50 Deployed ML model

X

13-51 Consistency evidence

X

13-52 Communication evidence

X

Base Practices

BP1: Specify an ML test approach

X

BP2: Create ML test data set

X

BP3: Test trained ML model

X

BP4: Derive deployed ML model

X

BP5: Test deployed ML model

X

BP6: Ensure consistency and establish bidirectional traceability

X

BP7: Summarize and communicate results

X