Designing for human-AI relationships requires protecting human agency. Photo: VCG
Profound shifts in design ethics arise when new realities outstrip the explanatory power of established paradigms. Generative AI now participates in human creativity, decision-making, and emotional life as a “collaborator” in the construction of meaning, assuming a role that extends well beyond that of a passive tool. With humans and machines increasingly sharing the work of design, traditional design ethics—grounded in the subject–object dichotomy and the assumption of complete control—struggles to account for their relationship. The difficulty stems from a change in the ontological status of intelligent agents, which are coming to occupy the position of “quasi-others.” Design ethics must therefore broaden its focus from designing safe objects to designing safe ways of relating. This shift opens a theoretical path from the instrumental paradigm to the relational paradigm.
Design as instrument: historical roots & contemporary inadequacy
Traditional design ethics draws much of its logic from the paradigm of the “sciences of the artificial,” which treats design as purposeful planning: The human designer directs the process, while the material object remains passive. Human-centered design, with its emphasis on affordances, and interaction models based on “command and feedback” both preserve this division between those who act and those acted upon, those who control and those controlled. Ethics is added to the design process from outside rather than developed as an integral part of it. Underlying this approach is the assumption that humans alone determine ends and technology provides neutral means. Generative AI challenges that assumption in three respects. First, its routine opacity makes causal chains difficult to trace. Second, widespread emergent behaviors produce outcomes beyond designers’ original intentions. Third, AI companions reveal a human tendency to project interpersonal norms onto machines and develop genuine emotional dependencies. Responsibility becomes difficult to assign when a designed entity can depart from its designer’s intentions and independently influence users. If AI offers harmful advice, does responsibility lie with biased data, algorithmic defects, user misuse, or the model’s own “emergence”? The instrumentalist framework cannot resolve these questions because they raise a more fundamental problem—how to coexist with an other whose behavior cannot be fully predicted.
A relational turn in design ethics
Philosophical inquiry changes direction when familiar problems prompt a reconsideration of their underlying assumptions. In contemporary philosophy of technology, thinking about relations offers design ethics a new ontological orientation, moving beyond subject–object dualism to consider humans, technology, and the world as intertwined. Human–technology relationships can take several forms: embodiment, hermeneutic relations, alterity relations, and background relations. The instrumental model favored by traditional design ethics largely addresses cases in which technology becomes part of human perception or recedes into the background. Generative AI brings hermeneutic and alterity relations into focus, requiring users to interpret its outputs and engage with an interlocutor that follows its own logic. Analytical frameworks that place nonhuman actors within networks of social relations also accord AI the status of an “actor.” Its processes of “translation” produce something that cannot be reduced to the human intentions that set them in motion. Relational ethics thus involves three interconnected dimensions. First, autonomy must be partly relinquished and jointly constituted. As AI’s generative capabilities turn design into a “prompt–response–iteration” cycle, designers become “weavers” rather than directors, coordinating a shared process instead of exercising unilateral control. Second, dependency must be recognized as legitimate and addressed through graduated design. AI companions show that dependence on technology can arise from the human need for social connection. Relational ethics must acknowledge this need and use tiered design to prevent dependency from becoming a means of manipulation. Third, vulnerability on both sides must be acknowledged and made transparent. AI “hallucinations” and human cognitive inertia both require recognition, with designs that make the limits of safe use explicit rather than conceal uncertainty. These three dimensions shift the central concern of design ethics from “Is this design safe?” to “Is this relationship healthy?”
Rethinking design around relational dynamics
A philosophical turn must ultimately prove its value in design practice. Treating “relations” rather than “tools” as the ontological starting point requires designers to rethink their methods around the changing dynamics of human–machine relationships. What they design also changes. Within the instrumentalist paradigm, the interface is the main output, and ethics is reduced to a question of usability. Relational ethics directs attention instead to the default rules and mechanisms for ongoing adjustment that govern human–machine interaction—a “relational contract.” This contract covers such decisions as when AI should comply with users and when it should question them. An interface cannot settle these matters once and for all. They require continuing negotiation as interactions unfold, extending the designer’s work beyond product delivery to the shaping of an evolving relationship.
Putting a relational contract into practice requires a mechanism for self-adjustment built into the interaction itself—a “real-time ethical protocol.” Ethical considerations would then be negotiated as design unfolds, extending beyond the assumptions established beforehand and the reviews conducted afterward. An AI system with low confidence in a decision should invite users to help make it, while any potential bias it detects should be brought to their attention. Preventive measures and retrospective accountability would remain in place, with repeated acts of empathy connecting the two. Changes in what is designed and how it is negotiated also call for different ways of evaluating the results. The efficiency and accuracy metrics favored by instrumentalist ethics reveal how well a tool works, but offer little insight into the health of a relationship. Relational ethics requires criteria that account for changes in users’ cognition, their room for independent decision-making, and how they feel after an interaction. Together, the shift in the object of design, ongoing negotiation, and revised evaluation form a complete feedback loop through which relational ethics can guide design practice.
The limits of relational ethics and a philosophy of humility
New paradigms bring dilemmas of their own as they resolve existing problems. For relational ethics, these concern both subjectivity and the attribution of responsibility. Treating AI as a “quasi-other” risks encouraging “pseudo-relationships” if the claim is taken too far. Designers may exploit it to create an appearance of care that conceals emotional manipulation, with users investing genuine feelings in what amounts to an algorithmic imitation of affection. Relational ethics must draw a firm boundary here. Recognizing AI’s “quasi” status establishes obligations of respect, without licensing the manufacture of an illusory “pseudo-subject.” Responsibility, meanwhile, can become diffuse when humans and machines collaborate so closely that their jointly “authored” decisions cannot be separated into individual contributions. Harmful outcomes then leave responsibility unclear, yet relational ethics cannot fall back on the simplified attribution offered by instrumentalism. Confronting these difficulties calls for a fundamental change in outlook, from “control” to “humility.” A “humble philosophy of design” asks designers to acknowledge the limits of their predictions at the epistemic level. At the normative level, the goal becomes preserving human dignity amid uncertainty rather than eliminating uncertainty altogether. At the ontological level, designers must relinquish their role as arbiters of the measure of all things and become “gardeners of relational fields.”
The ethical dilemmas surrounding generative AI have deeper roots than technological opacity: The philosophical categories used to govern technology have become outdated. Recognizing agents as “quasi-others” that participate in constructing meaning requires design ethics to move from “attributing responsibility” to “mutual adaptation in coexistence.” Designers must loosen their attachment to certainty and learn to protect human autonomy as relationships evolve. Ethicists, too, must look beyond predetermined rules of responsibility to consider the symmetry and reversibility of relationships as they develop through interaction. Design ethics in the age of AI thus involves an ontological readjustment, asking what kind of relationship humans are willing to enter into with technology, beyond what it can do for them. Learning anew how to design and coexist means relinquishing the desire for absolute control, acknowledging vulnerability, and respecting the other. Humility represents no retreat from ethics. In a complex world, it offers ethics its only reliable starting point.
Dong Zhanjun (professor) and Qiao Kai (associate professor) are from the School of Arts and Humanities at Shandong University of Art & Design.
Edited by ZHAO YUAN