Your own personal AI agent. What could possibly go wrong?: Opinion

.So you want to buy the new iPhone Duo. Usually, you would face the time-consuming task of comparing prices and deciding where to buy it.

Imagine instead, the convenience of simply telling an AI assistant: “Find me the best deal for the new iPhone Duo and buy it for me.”

This is the promise behind a new generation of personal AI assistants.

Artificial intelligence assistants such as Meta’s Muse and Instinct are moving beyond this and chatbots that simply answer questions. Muse, for example, can connect to apps and services to carry out multi-step tasks, while Instinct works through text messages to make bookings and other arrangements.

AI assistants can increasingly connect to our e-mails, calendars and other services, browse the web, make reservations, shop online and carry out tasks on our behalf.

The appeal is obvious. But there is an important difference between an AI that helps us make a decision, and one that makes and executes the decision for us.

If a chatbot gives us a poor recommendation, we can ignore it. If an AI assistant misunderstands us, it could spend our money, send an e-mail, cancel a booking or disclose sensitive information before we realise something has gone wrong.

There is also an uncomfortable contradiction. We are still struggling to make autonomous AI agents behave safely and reliably even in controlled or sandboxed environments. Yet consumers are increasingly being encouraged to connect them to some of the most sensitive parts of their lives – their e-mail, accounts, payment information and other personal data.

This is why the conversation about personal AI assistants cannot stop at privacy. We must ask what information they can access. But increasingly, we also need to question: What authority are we giving it to act on that information?

Problems with AI acting for us

Giving an AI authority to act for us personally introduces several new problems.

It may simply misunderstand what we want. “Find me the best deal for the new iPhone Duo” sounds straightforward. But what does “best” mean? The lowest price? Fastest delivery? A particular storage size or colour? A chatbot that gets this wrong may give us a bad recommendation. An AI assistant may actually spend our money.

Tech news site PCMag reported that one Muse user who selected “Allow Always” while letting the agent handle a Facebook Marketplace listing, later discovered that it had shared his home address with a buyer and arranged a pick-up. He said he had expected Muse to seek approval before accepting offers.

An Instinct user, according to TechCrunch, reported that she had disconnected her e-mail after the agent sent a message on her behalf without first checking with her.

Mistakes can cascade. Suppose an assistant books the wrong flight. It might then rearrange our calendar around that flight, cancel another reservation and notify colleagues that we will be away. One initial mistake has now become several actions across different services, potentially before the user even notices.

We therefore need to think not only about permission, but also about delegated authority – how much decision-making power we are prepared to hand over to AI.

A useful principle is simple: The greater the consequence of an action, the less autonomy the AI should have.

We might be comfortable allowing an assistant to reorder inexpensive household items without asking. But an expensive purchase should probably require approval. An assistant might be allowed to find and recommend a flight, but should ask before charging thousands of dollars to a credit card.

The choice should not simply be between giving an AI full authority and no authority. Users should be able to set clear limits on what their assistants can do without coming back to them.

Such safeguards are not new. Banks impose transaction limits, require additional authentication for certain payments and flag unusual activity. These measures create some inconvenience, but we accept the friction because the consequences of getting a financial transaction wrong – or allowing a scammer access to our funds – can be serious.

AI developers should adopt the same mindset and build safety into personal assistants from the start.

The onus is on AI developers

Some developers are already building in safeguards. Meta says Muse asks users to approve critical actions such as sending e-mails and making purchases, and provides a complete audit trail of what the agent has done.

But these safeguards are only useful if users understand what they have authorised and can set clear limits.

“Your assistant has completed your travel arrangements” is very different from “Your assistant is about to book this flight for $1,850 using this credit card. Approve?”

The aim is not to make people approve every trivial action. That would defeat much of the purpose of having an AI assistant. The challenge is to identify sensible boundaries where automation should stop, and human judgment should return. There should also be a clear and easily understood record of what the AI has done on our behalf.

Think of something closer to a bank statement than a technical computer log: what the assistant did, when it did it, which service it interacted with, whether it spent money or shared information and, where possible, how the action can be reversed.

If something goes wrong, users should not have to reconstruct their AI assistant’s behaviour across five different apps to work out what happened.

Company AI influencing your AI

There is an even bigger question on the horizon.

Today, online advertising is largely about influencing people. Companies compete for our attention, show us advertisements and try to persuade us that one product is better than another. But what happens when we are no longer the ones doing the shopping?

If I ask my AI assistant to find the best deal for a phone and buy it, I may never see the advertisement, compare the retailers or even visit their websites. My AI assistant may make the choice for me.

Companies will then have a powerful incentive to influence not only people, but also the AI assistants making decisions for them. Retailers may learn how to present information so that AI assistants favour their products. Sponsored results may shape recommendations. Other AI systems may even try to steer an assistant towards particular choices.

The next battle in digital advertising may not be for our attention, but for the attention of the AI agents acting on our behalf.

That raises a question we have barely started to confront: How do we know that our AI personal assistant is really acting in our interests?

None of this means we should reject AI personal assistants. They could remove large amounts of mundane work from our lives and make many everyday tasks considerably easier.

But consumers should think carefully about the authority they hand over. And AI developers should not treat safety as something to add only after these products become widely used.

A human-centred AI assistant should not be judged simply by how much it can automate. It should know what it can safely do for us, what it must ask us before doing, and how to show us clearly what it has done.

Keeping humans in control will depend on designing sensible limits to what we delegate.

Roy Ka-Wei Lee is an associate professor at the University of British Columbia.

Source: The Straits Times © SPH Media Limited. Permission required for reproduction.

Steps taken to help organisations adopt AI safely and responsibly: Forum Unhealthy air quality in S’pore enough to trigger haze pollution law; NEA is monitoring situation