If you’ve seen Bright’s coach inbox, you know it’s a wonderful way to engage your team and provide personalized feedback that helps them grow faster.
But we’re also keenly aware that it’s more difficult to SCALE that kind of personalized feedback. That’s why one of the features we’re proudest of is Bright’s AI Coach. This brings natural language processing - or NLP - into the platform to review what learners write or say, and automatically coach their submissions.
We believe this is part of the future of learning - the ability to wield practice and personalize coaching at scale.
We’re going to walk you through this capability step by step. Before we do that, I want to help frame your expectations on this. Because our AI Coach is a very substantial feature set. NLP Rules builder and AI Coach expert are the kind of thing you can eventually put on your resume. But it’s an emerging technology, so something you’ll want to take the time to really master.
To start, let’s look at the high level process for getting your AI Coach to work in Bright.
The feature set has 5 parts:
- AI Rules- These are the rules we create on the Skills + AI tab of a moment and the node of a Conversation Builder. We recommend getting comfortable with standard moment AI rules before jumping into a Conversation Builder.
- Speech to Text- This is how audio moments are able to be graded by AI. Speech to Text takes the Learner's audio recording and changes it to text just as Siri and other programs can take the audio to create messages and type notes.
- Coach Inbox Previews- We want our AI feature set to be scalable, but we also need to ensure a high quality. We use a Training Mode with new AI rules to allow a human coach to verify the AI rating in the coach inbox before serving the score + feedback to Learners.
- AI Reports- The AI Training Report allows a quick view of all AI rated moments and whether they are still in Training or Live. You will also be able to see the acceptance rate and the total submissions run against the rule. This is the best way to determine when we are ready for the rule to go Live for Learners.
- AI Coach Report- Even after turning an AI rule live, the Bright AI in the Coach Report will show the total number of ratings done by the AI Coach. We can dive deeper into this report to view specific submissions to spot check our rules as needed to maintain a high quality across our Simulations.
And these features match the flow for how to activate the AI Coach. As mentioned previously, we recommend starting with moment AI coaching before building your first Conversation Builder. Let's take a look at how we begin to scale our training by implementing our AI Coach.
- Create either an Audio or Written Moment type. For more information on how to make these, check out this link for creating an Audio Moment and this link for creating a Written Moment.
- Navigate to the Skills + AI tab.
- All AI coaching is tied to Skills from your Skill Library. After Selecting a Skill, you will see the option to Add a Rating Rule. Click this option.
- Now comes decision time. We have several options to tailor each AI rule to the moment being coached.
- Number of Attempts before human coaching (1-4). This is how many times a Learner would submit against the AI rule without meeting expectations (3 stars) before the submission is sent to the Coach Inbox. The goal here is to avoid a Learner getting stuck in a loop, unable to progress. If the Learner is struggling this much, this gives a human coach the opportunity to see and intervene with the Learner.
- If Statement- In most Cases we will use the Learner Submission option. This option will setup the AI to grade the content of the Learner's submission. In some cases, you may want to use duration to promote a learner being concise or more in depth. Be mindful the duration should only be used when the moment type is set to Audio. If the duration was selected, you will see the next dropdown become options for less than or greater than with the option to enter a value in seconds.
- Term Match- You will add words or phrases, the AI will look for the exact match in the Learner's submission. Term match does not worry about case, but it will otherwise be a character to character match. For example if your Term Match is looking for "Have a great day". The learner will not pass if they say, "Have a really great day." Since it is an exact match, we want to save this type of AI rule for 1-2 words that are compliance based (i.e. your company name or the phrase, Patient ID if that is something required of your Learners).
- Phrase Match- You will add words or phrases, the AI will look for these in the Learner's submission. Phrase match is more forgiving than term matching. It will accept different tenses, and filler words between the words of your correct phrase. In our previous example, the learner would get credit for "Have a really great day" if phrase match is in play because the root, "Have a great day" is all present.
- Full Intent Match- You will add 3-5 sample statements, the AI will pull the intent from the given statements and use this to score Learners. This is best suited when the specific verbiage has high variation across acceptable answers- topics like empathy are better for intent due to the long list of phrasing that may be valid for your purposes- sorry, apologize, condolences, would be frustrating, we want to provide better service, etc. This is our newest option, so more information can be found in this article. Learner Submission Types- Here is where we get into the nit and gritty of our AI capabilities. We have different options for different algorithms to score. The main 3 you should use are Phrase, Term, and Full Intent. While other options are present, these three are the newest and most accurate algorithms for moments. Most commonly, phrase matching will be used, but let's hit each of these at a high level.
- It will take a little getting used to, but you should definitely play around with the submissions to start to understand how the AI is working.
- As you have likely come to learn, we strongly encourage meaningful feedback. The AI feedback allows feedback tailored to each element being graded. So far we have only made 1, so you will see a green box for what learners see if they followed the rule and an orange box for when the missed the rule. It is important to provide direct, meaningful help without giving the answer. For example the followed the rule feedback might say, "Our call flow always ends with a positive tone when we wish the customer well." Meanwhile, the missed rule feedback might look like this, "Refer to our call flow diagram to be sure you include the 3 pieces of a call wrap." This specific feedback gives the Learner specific instructions on where to find the information to review before attempting the moment again.
- Add Stars for the Learner following the AI rule. In total, at least 3 stars are required to save.
- Once you Save, the rule by default starts In Training. We will discuss this later in this conversation. For now, let's look at some additional options that are present for our AI Coach.
Now this is a good place to talk about Transcription in the platform, which is a type of speech to text. Currently, this feature is supported by Amazon. The speech to text by nature is set to strip punctuation for speech in an effort to simplify the overall process. You probably noticed this if you have tried to put punctuation within your AI rule terms on an Audio moment, since the system will not let you enter those special characters. It is important to note that this does mean it takes more time for longer transcriptions. But the overall time saved from the AI coach compared to human coaching will make up for it many times over!
Connectors
Connectors enable additional criteria for the same Skill/Star rating. Whether we are using the AND connector to look for the learner to mention both of the options we have for the caller's concern, the OR connector to allow a learner to use one of the required pieces of caller verification, or the THEN connector to check for a process being communicated in the proper order, they are all available for your customization.
Best Practices with Connectors:
- Only 1 type of connector can be used, though multiple can exist under the same Skill. For example we could use 2 AND connectors to look for the Learner to mention 3 separate pieces of information, but we cannot use an AND plus an OR connector under the same Skill.
- We should only use 1 intent algorithm per moment. Since the Full Intent AI is looking for a concept rather than specific verbiage, we recommend including all parts you are looking for Learners to include in the Intent Samples. This will result in the best experience for you and your Learners.
- Use a Phrase Match with a Full Intent AI to look for a concept being communicated with specific compliance verbiage as well. Maybe you want your Learner to provide empathy while stating a specific name of a program or policy available for the situation or you want them to educate the caller on the next steps of a process while using the caller's name. These types of situations can really benefit from the combination of Intent and Phrase Matching.
Common Errors
Common Errors are an additional feature within the Bright AI Coach. These allow us to identify the common errors existing employees struggle with and call them out specifically. The Common Errors function the same as adding terms to your existing Phrase Match rule. If a Learner would otherwise pass an AI rule, but they hit a phrase or keyword in the Common Errors, they will see the Common Error Feedback (our grey feedback box). This will also mean the moment is served up to the Learner again to give the opportunity to meet all the desired behaviors without our common errors.
An example might look like this:
Here you can see we want our Learner to ask for the reason for the call. But we have heard some cases where our existing employees have used the, I know you will cringe too, "What's your issue?" verbiage. Yikes!
However, now during the training process we can aim to curb the behavior and reinforce better phrasing. In the image, you can see our rule would give 3 stars for any of these types of phrases:
- How may I assist you today?
- What can I do for you?
- What can I help you with?
- What questions do you have today?
- What is the reason for your call today?
Meanwhile the Learners who use our problematic phrasing will see the feedback from the grey box and be asked to resubmit this moment. This is the concept behind common errors.
Training the AI Coach
In a perfect world, we would make an AI rule and it would be flawless, ready for Learners. Unfortunately, this technology is new and there is a lot of variables in play. Each of us have a diverse group of Learners which at a basic level means we cannot predict every possible verbiage we would accept on the first iteration of our rules. This is why the AI Coach starts In Training for each new rule we create.
The process of training the AI at a high level is:
- Create the first iteration of the rule
- Have at least 10-15 Learners go through the moment.
- Make needed edits to the AI rule.
- Once the Acceptance rate is 90%+, with at least 10 ideally more towards 15 submissions, turn the AI rule Live.
- Continue to listen to Learner feedback and spot-check the AI Coach using the reporting suite we mentioned at the start of this article.
We have already run through how to make the first iteration so let's pickup the training process with number 2. For this step in the process, we need to add our moment to an assignment path and have 10-15 Learners go through the moment to answer how they would. These Learners should typically be a core team such as your trainers, QA, or other leaders within the L&D umbrella. Including QA is one of the best practices to ensure each AI rule is up to standards from the team who knows those standards inside and out.
Before we get into the process, you may feel that the AI rule you are making is simple or maybe we are on a time crunch to push it live, but it is important to NEVER push an AI rule live without putting it through training. Training the AI is one of the processes that, when done, creates a smooth experience for your Learners.
The next phase of the process is to train and edit your AI rules based on the submissions you’re receiving through the Coach Inbox preview. This is SO important that we have an entire lesson to go over best practices for this in detail. For the purposes of this lesson, let’s just leave it at you’re going to make sure your AI rules are working BEFORE you make them live.
So now comes the edits. In our Coach Inbox, we can view the AI In Training ratings for our Learner submissions. This shows the AI grader suggestion. We can playback the audio sample or even correct the transcription if needed. But our main goal is to verify the AI grading matches what we would rate the submission.
In this image, we might have found that asking, "How can I be of service?" is acceptable for probing the reason for a call. If we want this to be correct, we can click the Edit AI Rule option. This will open the moment in the admin console where we could add "be of service" to our Phrase Match terms. If the rating was accurate, say we decide this is too informal for our brand, we would not need to make an edit to the AI rule for this submission. As a best practice, we recommend viewing the other sample submissions to determine all edits needed to make them all at once. Then, we can save the moment and return to our coach inbox. In the upper right, we can refresh the AI rule to re-grade the submission with the edits we made.
Once we are in agreement with the AI Grader suggested ratings, we can simply scroll down and select Save & Archive. This sends the feedback to the Learner and removes the submission from the Coach Inbox.
Now for the acceptance rates and turning the AI live. For this, we turn to the reporting suite. Under the Reporting tab, you can navigate to the AI Training Report. This report is meant for this process. Here you can see how many submissions to each AI rule have been graded. Remember to look at the Total Training Interactions and Acceptance Rate. Our goal is at least 15 and 90%+ respectively. Once you feel confident the moment is ready for live learners, you can toggle the AI rules live from this same page OR directly in the moment.
Once the rule is live, moving forward any learners who go through this moment will be rated and coached immediately. And here’s what that looks like…
Now once you make your moment live, it may feel a little bit scary, because now your learners are seeing feedback from a coach and you’re not there to monitor. Have no fear. We have report for that as well so you can look over the AI Coach’s shoulder and confirm your rules are working as planned. This is under the Coach Report -Bright AI Coach.
