Beyond the ML Model: Applying Safety Engineering Frameworks to Text-to-Image Development

Identifying potential social and ethical risks in emerging machine learning (ML) models and their applications remains challenging. In this work, we applied two well-established safety engineering frameworks (FMEA, STPA) to a case study involving text-to-image models at three stages of the ML produc...

Full description

Saved in:
Bibliographic Details
Published in:arXiv.org 2023-07
Main Authors: Rismani, Shalaleh, Shelby, Renee, Smart, Andrew, Renelito Delos Santos, Moon, AJung, Rostamzadeh, Negar
Format: Article
Language:English
Subjects:
Data processing
Ethics
Hazards
Machine learning
Product development
Safety
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Identifying potential social and ethical risks in emerging machine learning (ML) models and their applications remains challenging. In this work, we applied two well-established safety engineering frameworks (FMEA, STPA) to a case study involving text-to-image models at three stages of the ML product development pipeline: data processing, integration of a T2I model with other models, and use. Results of our analysis demonstrate the safety frameworks - both of which are not designed explicitly examine social and ethical risks - can uncover failure and hazards that pose social and ethical risks. We discovered a broad range of failures and hazards (i.e., functional, social, and ethical) by analyzing interactions (i.e., between different ML models in the product, between the ML product and user, and between development teams) and processes (i.e., preparation of training data or workflows for using an ML service/product). Our findings underscore the value and importance of examining beyond an ML model in examining social and ethical risks, especially when we have minimal information about an ML model.
ISSN:2331-8422