<nettime> If you talk to bots, you're talking to their bosses

[Brett Scott <brettscott@fastmail.com>]

Hi all, just published this on the disingenuous and phony behaviour of
AI 'bots', suggesting that the next stage of corporate personhood is for
corporations (legal persons) to literally claim to be natural living
persons, something that can be acheived only if the user-experience
layer of a company’s processes can be completely automated and then
attributed with an abstract personality to make it a being-in-itself.
Original article link:
https://howwegettonext.com/if-you-talk-to-bots-youre-talking-to-their-bosses-cd8e390c242f#.3w98g4c82

IF YOU TALK TO BOTS, YOU'RE TALKING TO THEIR BOSSES: BRETT SCOTT

At a recent financial technology conference I was invited to meet Cleo,
a friendly automated spirit living within the confines of an iPhone
interface that offers financial advice to those who chose to activate
her. Described as an “AI assistant for your money,” she playfully
answered text message queries about bank balances, spending, and budgeting.

Hey, Cleo, what’s my balance?

Hey, Alex! MasterCard: -£760. Current account: £1048. Savings: £1700

Cool, how much have I spent at Pret this month?

You’ve spent £44 at Pret since you got paid on the 15th of December

Despite the futuristic jargon and growing wave of hype around such bots,
automated assistants are not really that new. Our world is full of
simple bots, like the automatic hand dryer in a public restroom that
jets hot air if you trigger its sensor. By now many of us have
experience with some that attempt to mimic basic personality, like the
tinny voice of the supermarket self-service checkout machine as it
coldly attempts to replicate one half of a stilted, human conversation.

There are two new trends emerging, though. First, the sophistication
with which bots interact is increasing. They are able to respond to (or
more accurately, be triggered by) a greater variety of situations. The
iPhone’s Siri is programmed to respond to all manner of vocal
instructions, from answering your queries about the time, to searching
the internet for you.

Second, the mode by which they frame that interaction — how the machine
addresses you — is also shifting. An old automatic hand dryer may have a
sticker on it that says: “To activate dryer, put hands below.” This is
essentially an instruction left by the creator of the machine. The
supermarket checkout machine similarly offers generic audio instructions
like, “Place items in the bagging area.” The new breed of bots, however,
are dropping non-personal words like “the” in favor of mysterious
first-person pronouns like “I” and “me,” or possessive pronouns like
“my.” Imagine the automated checkout asking you to “place the items in
my bagging area.”

Machine entities are increasingly claiming to have some subjectivity,
some sense of an independent place in the world. Or, rather, the team
that creates the machine programs it to claim subjectivity.

The new generation of bots, which include the scheduling assistant Amy
and Amazon’s Alexa, would very much like to get on a first-name basis
with you. Amy’s website offers an enthusiastic greeting: “Hi, I’m Amy!
You interact with me as you would to any other person — and I’ll do all
the tedious email ping pong that comes along with scheduling a meeting.”
She then gives her personal email address.

Who, though, is this “I”? Who is Amy exactly? The home page certainly
doesn’t mention her creators. Perhaps the personalities, agendas, and
egos of the team might distract from the personality, agenda, and ego of
the being they present to us as their avatar. Amy is keen to tell you
that her ability to schedule your meetings is “like magic,” rather than,
say, the end result of using venture capitalists’ money to pay highly
trained machine-learning software engineers.

The creation of a subjective identity for the bot may be an attempt to
mystify and delight, to create some kind of warm, intuitive human
experience. Humans have a long history of mystifying objects in order to
imbue them with meaning beyond their immediate functional use.
Anthropologists sometimes refer to this as fetishization. A fetish
object is not just something you find in an S&M club. It’s any object
seen to be more than just an object. The entire branding industry works
off this concept: Levi’s jeans are not just functional items of
clothing. They’re a lifestyle. Jack Daniel’s isn’t just whiskey. It’s a
carrier of the down-home spirit of rural America.

We do it too with sentimental items and heirlooms, like the old guitar
inherited from your grandfather or that faded leopard-print shirt that
your mother once wore in the 1980s. A key element of this fetishization
is to take a social relationship of some sort — such as a relationship
between family members — and project it into, or imagine it within, the
confines of some physical thing, as if your relationship with your
mother is in the fabric of the shirt.
This can be a very positive experience. Perhaps, if your mother has
died, you hold onto that shirt when you feel afraid and want comfort.
Fetishized objects can even be used to facilitate difficult
relationships, as in the case of Wampum beads created to signify a peace
treaty between warring groups, symbolically holding the peace within the
beads as a way to make it seem more concrete.

On the other hand, a fetishization process might be used to conceal
exploitation, or distract from a tense relationship. The Marxist concept
of commodity fetishism critiques how people fixate upon things rather
than the humans that make them. You don’t see the labor exploitation
that might exist beyond the interface of your new tablet computer. You
mistake it for a thing-in-itself, floating in abstract consumer space;
something you bought off eBay rather than something created by real
humans in a real factory somewhere. To exclusively focus on the
immediate here-and-now contours of an object is to ignore the earlier
and deeper relations the led to its appearance in the world.

We don’t only fetishize physical goods, though. One of the ultimate
fetishized entities is the company or corporation. What is it but a
group of real people working under the banner of an artificial “object,”
an entity given a name like Barclays and possessing a legal personality
that enables it to “do things.” Such “legal persons” can sue others,
enter into contracts, give gifts, and issue statements.

When issuing statements, though, the bank Barclays still doesn’t address
itself to people as “I,” possibly because that would stretch the bounds
of plausibility. When push comes to shove, we all know that underneath
Barclays’ blue logo is a big group of people actually carrying out the
work that is then attributed to “Barclays”—and we know that Barclays
doesn’t really exist unless those employees carry on going to work. A
corporation may be a legal person, but it is not a “natural person”; it
is not an actual living, subjective being.
But let’s for a moment imagine that people working within such a firm
have designed an advanced automated system that they set up as a
self-service user interface for their customers. Now imagine that they
give that automated process a human name, and get it to issue statements
in which it literally claims to be a natural person. Their system is
programmed to refer to its workings as “I.”

We speak to it. We think we are talking to it. And yet what we are
really doing is interacting with a machine process put in place by a
particular group of humans with a private economic agenda, denying their
own existence and maintaining the illusion that the agency resides with
an external being, dressed up in the form of a friendly helper.

This is the next stage of corporate personhood. If the user-experience
layer of a company’s processes can be completely automated, neither its
owners nor employees need to present themselves to people. They give the
automation an abstract personality and make it a being-in-itself.

Such a being — like any company — is prepared to be helpful, provided
you never ask anything too challenging. I asked Siri, “How much is my
data worth to the company of people that you represent?” Siri paused,
and suggested I read some Financial Times articles on the value of data.

After all, my apparently playful and private interactions with these
beings will end up as log entries in a database, ready to be subject to
data analyses, ready to be sold, and ready to be used to project my own
personality back at me in the form of an ever more disingenuous
artificial being. It will refer to me as Brett, and suggest things that
I didn’t even realize I wanted, induced from statistical relations
between different data sets I’ve inadvertently produced.

It’s time for us to to move beyond uncritical hype around bots, and to
start considering the real economic agendas for why these virtual beings
(are claimed to) exist. We can design automated entities with different
personas, like the “I clearly don’t give a shit” supermarket checkout
bot that doesn’t even pretend to like you, or the “I’m really fun” bot
put in place by a startup company. But the one persona that is likely to
always be missing is the Honest Bot, the one that clearly tells you its
agenda, like a true friend who drops their façade and lets you know
their dark secrets.

The Honest Bot does not currently exist. The new wave of bots are — to
use a term popularized by the existentialist misfit Holden Caulfield
from The Catcher in the Rye — the ultimate phonies. They’ll pretend to
be friendly, to be cool, to be serious, to be insightful, and even to be
self-reflective. But they’ll never just be themselves. Because, in the
end, there is no “I.” There is only a company, its shares held by who
knows who, possibly registered in Panama.

Maybe one day we’ll invent the Honest Bot—and I urge anyone creating
bots to do this, please. But until then, just remember the following: If
you’re talking to bots, you’re talking to their invisible bosses.


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

#  distributed via <nettime>: no commercial use without permission
#  <nettime>  is a moderated mailing list for net criticism,
#  collaborative text filtering and cultural politics of the nets
#  more info: http://mx.kein.org/mailman/listinfo/nettime-l
#  archive: http://www.nettime.org contact: nettime@kein.org
#  @nettime_bot tweets mail w/ sender unless #ANON is in Subject: