Begin with the human: Designing for safety and trustworthiness in cyber-physical systems

Abstract

Control systems are designed and built to manage and regulate the behavior of other systems. The use of artificial intelligence (AI) in control systems has simultaneously created new opportunities and new challenges in how we create, manage, and govern cyber-physical systems. In this paper, we discuss the challenge of defining and developing a model for contemplating how these systems will potentially learn, evolve, and act without human intervention. We present an analytical framework for thinking about trust and safety in these systems—both key factors for shared context in human-machine teams—and demonstrate its application using an example from history. Show more