Date of Award

Spring 5-1-2026

Document Type

Dissertation

Degree Name

Doctor of Philosophy (PhD)

Department

College of Technology

First Advisor

M. Affan Badar

Second Advisor

Arash Rafiey

Third Advisor

Christopher Kluse

Abstract

This dissertation evaluates whether a reusable assurance architecture, the Quality Assurance Machine (QAM), can provide effective product and process quality assurance for ML-enabled software platforms. The QAM is a system-level SQA architecture that turns plans and policies into versioned configurations, executes them in controlled environments, and produces preserved run evidence that supports traceability, auditability, and controlled change. The study follows Design Science Research and evaluates the instantiated artifact using eight assurance requirements (AR1–AR8) synthesized from standards-based guidance, including IEEE 730 and ISO/IEC/IEEE 15026. A four-year longitudinal evaluation combines two methods. First, operational evidence from routine regression and release-validation runs, defect linkages, environment governance artifacts, and preserved logs is analyzed to assess product assurance outcomes and process-quality monitoring outcomes (reported as documented, artifact-grounded examples). Second, Design of Experiments is used to characterize infrastructure-driven variation and dominant performance factors, which are then translated into Performance Analysis engine (PAe) baselines. Once deployed, the PAe adds detection-focused monitoring via sigma-gated checks. The evaluation is bounded by a single organization and a regulated medical-imaging platform context. The study does not assess model-centric properties such as accuracy, fairness, or explainability. Contributions include (1) an evaluated, system-level assurance architecture for unifying product and process assurance of an ML-enabled system, (2) a reusable evidence-bundle pattern that binds intent, execution context, and outcomes at run level, and (3) a DOE-grounded, change-detection approach to operational monitoring that is suitable for ML-enabled systems where important regressions can present as resource and throughput shifts rather than discrete incorrect outputs. Theoretical implication: The findings support system-level SQA as a unifying assurance construct for ML-enabled systems, where assurance depends on evidence-producing architecture and controlled execution rather than model-only evaluation. Practical implication: The results indicate that run-level evidence bundles and PAe baseline monitoring provide a repeatable way for SQA teams to detect operational regressions and support audit-ready release decisions in regulated environments.

Download

Share

COinS