22 December 2014

Natural Language Interaction

Andy Peart - Artificial Solutions

4Posts 10,858Views 0Comments

Innovation in Financial Services

A discussion of trends in innovation management within financial institutions, and the key processes, technology and cultural shifts driving innovation.

Mystery Shopping the Virtual Assistant

08 May 2014  |  1369 views  |  0

Virtual assistants might be transforming how banks deal with customer queries, but quantifying that the best advice and the right answer is given each time, isn’t always as simple as it seems.

With most virtual assistants it’s possible for an organisation to easily see when a problem has occurred just by looking at the unresolved queries – ones that the assistant couldn’t understand and therefore answer. These can usually be resolved by looking at the actual chat log files and adjusting the assistant’s parameters or by updating its knowledge.

The organisation knows about these errors because the assistant wasn’t able to answer them.  What happens to the ones they do answer, but incorrectly? Detecting a wrong answer can be a bit more of a challenge.

The average number of queries a virtual assistant answers each day typically runs into the thousands. Combine this with the number of different topics from current accounts to insurance cover and it’s easy to see why Mystery Shopping your virtual assistant isn’t just a simple affair if you wish to test their entire knowledge.

Mia is the Co-operative Bank’s virtual assistant that is used internally by call-centre staff. Launched a couple of years ago she generally achieves a very high accuracy rate in answering queries first time, in February for instance she achieved a 98.64% efficiency rating. Thomas Bacon, Virtual Adviser General Editor at The Co-operative Bank explains how they ensure that Mia delivers the right answer straightaway – most of the time.

“All of Mia’s information is sorted into categories, for example, all ISA-related content is grouped together. Any one category might have been accessed thousands of times during the course of a month, but when you look into the finer details it’s possible to see that maybe it was just two or three hundred different terms that users inputted that took them to a specific category,” says Thomas.

By comparing the triggers against the categories Thomas and his team are able to double check that users were given answers from the right category. This highlights if something is out of place and allows for it to be corrected. This is a technique, known as Trigger Analysis, can then be used to measure Mia’s efficiency.

“If we look at February’s data analysis for example,” continues Thomas. “You can see straight away that the results in Mobile Banking were lower than the others. In this case, just one of Mia’s responses was incorrect in terms of the natural language interaction, which is demonstrated by 9% of Triggers going to the wrong place. It still wasn’t too bad – the adviser was only ever one mouse-click away from the information they needed to see, but it is something that the editorial team can correct easily, continuing to improve Mia.”

TagsMobile & onlineRetail banking

Comments: (0)

Comment on this story (membership required)
Log in to receive notifications when someone posts a comment

Latest posts from Andy

Mystery Shopping the Virtual Assistant

08 May 2014  |  1369 views  |  0  |  Recommends 0 TagsMobile & onlineRetail bankingGroupInnovation in Financial Services

Maximizing the value of in-house data

04 July 2013  |  2569 views  |  1  |  Recommends 0 TagsRetail bankingInnovationGroupInnovation in Financial Services

Banking on smartphones

26 June 2013  |  3286 views  |  2  |  Recommends 0 TagsMobile & onlineGroupInnovation in Financial Services

Why Siri Signals a Change in Customer Communications

10 November 2011  |  3635 views  |  0  |  Recommends 0 TagsMobile & onlineRetail bankingGroupInnovation in Financial Services

Andy's profile

job title CMO
location Newbury
member since 2011
Summary profile See full profile »
CMO of Artificial Solutions, a provider of natural language technology used by organizations such as Co-Operative Banking Group, Credit Suisse and Banca Sella. Over 25 years' experience in running mar...

Andy's expertise

What Andy reads
Andy writes about
Andy's blog archive
May 2014 (1)2013 (2)2011 (1)

Who is commenting on Andy's posts