Google recently held its annual developer conference where it announced all the new things it is working on. None of these garnered more buzz than the announcement of Google Duplex.
Duplex is sort of an extension of personal digital assistants such as Google’s own Home product or Amazon’s Alexa or Apple’s Siri. The upgrade here is that Duplex can autonomously perform real world tasks for the user without that user’s involvement. And even more groundbreaking is that this virtual assistant can actually interact with humans who are not the user that makes the request.
What does that mean? Well, Google has only released a few well controlled demonstrations, but what it has shown seems to point to a paradigm shift in how we interact with technology. Duplex is able to make phone calls to set appointments for things such as a haircut or a restaurant reservation. While these tasks may not seem that miraculous, you need to listen to the calls to really understand why this is so groundbreaking. Here are a couple of samples.
First, listen to Duplex call to make an appointment for a haircut:
As you can hear, the Duplex bot is able to respond appropriately to the human salon worker. This is especially impressive as the bot has to handle changing circumstances (i.e. asks for an appointment at noon; is told that is unavailable; bot makes a second request that augments, but does not contradict the first request).
But the real magic here is how natural the Duplex bot sounds in conversation with a real human. When making requests of Siri or Alexa, you can pretty immediately tell that you are talking to a non-human. The speech is measured and things like tone of voice are not as realistic as we would expect when talking to another human. But with the new Duplex system, they have made the bot’s voice more naturalistic in a number of ways.
One of the more impressive features that help give the illusion of speaking to a real human is the inclusion of what are known as “speech disfluencies.” These are the “ahs” and “umms” that we all use in our own everyday speech. Listen to the clip above again and you’ll hear these disfluencies in action.
Now, the clip above is a pretty simple, linear interaction. But humans are messy, behave erratically and are often hard to understand. We speak ambiguously and the receiver of the information must often make a decision about the meaning of a statement based on contextual clues. So let’s increase the complexity. What about phone calls with a noisy background as is often found in a restaurant setting? What if the human that Duplex is speaking with has an accent? Or even harder, what if they are not a native English-speaker? How would the AI handle that situation?
Let’s listen to an example of Duplex calling to make a restaurant reservation in a noisy environment with a non-native speaker who struggles a bit with complete sentences:
As you can hear from the clip above, this is where the magic really begins to shine. The AI is able to successfully navigate this conversation despite the language barrier, incomplete sentences and a noisy background.
But is this the super-intelligent, robotic future we’ve been told about? Probably not quite yet.
You see, the Duplex AI is only designed to work in very narrow situations. It is trained extensively to handle these situations and respond appropriately to the inputs it receives. But Google is quick to note that “[Duplex] cannot carry out general conversations.”
Yet.
Our Robotic Restaurant Future
You don’t need to look very far past where this technology currently is to see a potential future that could be highly disruptive to the restaurant and foodservice industries in general.
First, we must identify situations that are similar to those already shown by Google. That is, situations that are fairly narrow in scope and involve a pretty scripted pattern of back and forth interactions that lead to a result. Making a reservation for something is a good example. The inputs are narrow and fairly well understood. You have the date of the appointment, the time of the appointment and the service being requested. Easy, right?
But you don’t have to stray far from that path to see where it could fall apart. For example, scheduling a doctor’s appointment might be more difficult as the person taking the information may ask follow up questions that are outside of the AI’s ability to answer. The necessary information can easily go beyond the who, what, where type of responses that Duplex appears to handle very well.
In the restaurant industry, we have a number of situations that would appear to fit well into the model for the Duplex AI. One of the most straight forward would be order taking at a drive-thru. The interactions are fairly limited. A customer orders items. A customer may want to customize menu items (e.g. “hold the pickles”). A customer may ask questions about menu items (e.g. “what comes on the club sandwich?”). The AI processes the order, totals it and asks the customer to pull around.
This is not so different from the self-service kiosk ordering that has begun to show up in fast food restaurants.
Artificial Intelligence Meets Artificial Intelligence
We can even use the example Google provides to imagine the future. It’s not hard to imagine a restaurant’s reservation system using a Duplex-like AI to take the reservations as well as make the reservations as shown in the calls heard here. That means, you could have one AI placing the reservation and another AI receiving the reservation.
This type of end-to-end automation is sure to come with unexpected problems and issues. But it isn’t unthinkable that this could be achieved in just the next few years.
Error Handling
Lest you think that Duplex signals the end of a human workforce, fear not! As we said earlier, humans are messy and unpredictable. As a result, we will always need human back up for situations that are beyond the AI’s ability to cope and respond.
But even this can be handled pretty seamlessly. If the AI gets stuck, it can simply transfer the interaction to a live human.
A.I.: “I’m sorry. I’m having an issue with your order. Let me transfer you to a specialist that can help to complete your order. Can you hold a sec?”
An effective AI system will have to be able to handle errors graciously, if not seamlessly. Fortunately, this is a known issue and certainly one that is being actively worked on as AI systems become more and more common.
Disclosure
One of the thornier issues that this technology presents is that of disclosure. When we phone customer service today, we are notified that our “call may be monitored and recorded for quality assurance.”
This is so common that we barely think about it. When was the last time you disconnected a call because you didn’t want to be recorded? Have you ever thought about how that recording is used? Do you care?
And does it matter that an artificial intelligence is one of the participants (instead of two humans)? Guess what? AI already listens to your customer service calls. We certainly aren’t far from the day when an AI more actively participates in those calls.
But the fact remains that federal law requires disclosure before a business call can be monitored or recorded. Under the federal law, only one party on the call needs to agree to the monitoring and recording. But many states go further and require that all parties agree to the recording.
By definition, all of these AI calls are recorded in a sense because they have to be interpreted by a computer. That doesn’t necessarily means that they are stored long term. But it doesn’t explicitly mean that they aren’t saved either. And this could present legal problems if not addressed in advance.
Legal Issues
Let’s go back to the doctor’s appointment scenario. Imagine that you disclose sensitive personal information as a part of setting up an appointment. You might think you are talking to a human who is only going to use that information for the purposes of creating the appointment. But since this is an AI and not a human, your data is not only recorded, but also potentially stored in a way that could be used for purposes that you never dreamed of when simply trying to make an appointment.
This scenario strongly indicates the need for disclosure when interacting with AI entities. It might even necessitate the creation of opt-out paths in the AI routines (i.e. “You will be speaking to an artificial reservation system. If you would prefer to speak with a human customer service representative, please press 1 now.”).
Although not explicitly discussed in Google’s Duplex demos that we have seen thus far, Google acknowledges the need for disclosure even for the types of calls shown in the demos.
“It’s important to us that users and businesses have a good experience with this service, and transparency is a key part of that. We want to be clear about the intent of the call so businesses understand the context” according to a recent Google blog post. “We’ll be experimenting with the right approach over the coming months.”
In other words, before the Duplex AI can begin its reservation call, there may need to be a preamble disclosure for the business recipient.
A.I.: “Hi, I’m a virtual assistant working on behalf of Jane Doe. As part of my programming, I need to record this interaction. Is it OK if I schedule an appointment with you?”
It might ruin part of the magic of speaking to a human-sounding AI, but the legal issues will likely take precedence. But since we are on new ground with the whole AI autonomous virtual assistant front, it may take some time before all the legal issues are sorted out. Unfortunately, these things often get escalated only after some sort of negative incident or lawsuit. It will be interesting to see how our laws and regulations change and adapt to the new technology.
I, For One, Welcome Our New Robotic Overlords
Even with all the potential pitfalls involved, humans have shown an enthusiastic willingness to interact with artificially intelligent systems. The rapid adoptions of systems like Apple’s Siri and Amazon’s Alexa are just the tip of the iceberg.
This situation makes the 2013 Spike Jones film “Her” feel especially prescient. In the movie, a lonely writer cultivates a relationship with an artificial intelligence embedded in his mobile device, much like Siri, but radically more advanced.
Samantha: “You know what’s Interesting? I used to be..so worried about not having a body,but now I.. I truly love it. You know, I’m growing in a way I couldn’t if I had a physical form. I mean, I’m not limited. I can be anywhere and everywhere simultaneously. I’m not tethered to time and space in a way that I would be if I was stuck in a body that’s inevitably gonna die.“
As artificial intelligence infiltrates more and more of our lives, we may indeed find that it has the ability to be anywhere and everywhere all at once. Surely, there will be unintended consequences. But if early indications continue to hold, the human race will adapt. And it isn’t much of a leap to imagine giving these systems even more autonomy. Imagine a Duplex-like AI system in a body like the one built by Boston Dynamics. Add in a little bit of the magic of Disney-like animatronics and imagineering and we could be in for an interesting ride…..a lot sooner than many of us expected.
But lest you think that all the Duplex stuff is just smoke and mirrors created just to provide a cool, on-stage demo for the Google developers conference, take a listen to the call below. The lead developers for the Duplex project, Yaniv Leviathan and Matan Kalman, used the system to make a real restaurant reservation for themselves. And the took a photo to prove it.
