An Analysis of the Interaction Between Intelligent Software Agents and Human Users

Christopher Burr · Nello Cristianini · James Ladyman

184 citations0/0 files storedMinds and Machines

No readable copy stored yet — this record is metadata only.

No open-access location is known for this record yet.

Literature note Cite in a project Citation map

Ethics and Social Impacts of AI Multi-Agent Systems and Negotiation Innovative Human-Technology Interaction

Abstract

Interactions between an intelligent software agent (ISA) and a human user are ubiquitous in everyday situations such as access to information, entertainment, and purchases. In such interactions, the ISA mediates the user's access to the content, or controls some other aspect of the user experience, and is not designed to be neutral about outcomes of user choices. Like human users, ISAs are driven by goals, make autonomous decisions, and can learn from experience. Using ideas from bounded rationality (and deploying concepts from artificial intelligence, behavioural economics, control theory, and game theory), we frame these interactions as instances of an ISA whose reward depends on actions performed by the user. Such agents benefit by steering the user's behaviour towards outcomes that maximise the ISA's utility, which may or may not be aligned with that of the user. Video games, news recommendation aggregation engines, and fitness trackers can all be instances of this general case. Our analysis facilitates distinguishing various subcases of interaction (i.e. deception, coercion, trading, and nudging), as well as second-order effects that might include the possibility for adaptive interfaces to induce behavioural addiction, and/or change in user belief. We present these types of interaction within a conceptual framework, and review current examples of persuasive technologies and the issues that arise from their use. We argue that the nature of the feedback commonly used by learning agents to update their models and subsequent decisions could steer the behaviour of human users away from what benefits them, and in a direction that can undermine autonomy and cause further disparity between actions and goals as exemplified by addictive and compulsive behaviour. We discuss some of the ethical, social and legal implications of this technology and argue that it can sometimes exploit and reinforce weaknesses in human beings.

Paper brief

Loading…

Cite this work

Citation unavailable for this work.

BibTeX RIS CSL-JSON

No public copy is stored for this record, so only the metadata is cited.

Citations in this library

Citation map →

Stored only when both papers are in the corpus — 184 citations exist upstream.

References

2019Theory of games and economic behavior 2011Thinking, Fast and Slow 1990The structural transformation of the public sphere: an inquiry into a category of bourgeois society 2008Nudge: improving decisions about health, wealth, and happiness 1956Rational choice and the structure of the environment.2008Nudge: Improving Decisions About Health, Wealth and Happiness 2011The Filter Bubble: What the Internet Is Hiding from You 1993The Quality of Life

Cited by

2023How social anxiety leads to problematic use of conversational AI: The roles of loneliness, rumination, and mind perception 2020Smart nudging: How cognitive technologies enable choice architectures for value co-creation 2020Supporting Human Autonomy in AI Systems: A Framework for Ethical Enquiry 2025Why human–AI relationships need socioaffective alignment 2022A Conceptual Model of Trust, Perceived Risk, and Reliance on AI Decision Aids 2023Nudge in perspective: A systematic literature review on the ethical issues with nudging

Explore the map →

Papers that cite the same references — the relational fallback while embeddings are still backfilling.

Imported from OpenAlex (CC0) — nothing here is generated.