Unlocking The Potential Of AI Coaching In L&D

What Is An AI Coach?

We define an “AI coach” as a digital resource that assists learners in practicing real-world skills by providing feedback as they are engaged in them. This feedback includes identifying mistakes, suggesting improvements, and applauding correct actions. It creates a feedback loop for learners while they practice within a simulated environment to master some capability.

Unlike AI tutors, which provide answers to questions and suggest personalized learning pathways, AI coaches focus on self-reflection and metacognition, fostering self-directed learning. AI tutors are forward-looking, aiming to guide learners in their pursuit of mastery, while AI coaches are backward-looking, observing a learner’s performance and afterward delivering feedback so they recognize what they did right and wrong.

What Are The Advantages Of AI Coaching?

The only way to improve soft skills is by exercising them, with feedback showing the way to improvement. AI can help: It can provide intent analysis, word choice analysis, tonal analysis, and so on to provide learners with actionable insight into their performance. While work in these areas is nascent, it promises to complement human coaching and provide insights that add to the value of the learning experience.

Additionally, AI coaches excel in areas where humans face limitations. They can provide coaching repeatedly and with infinite patience, operating 24/7 to allow learners to practice at their own pace. They never get fatigued or demand breaks, making them highly scalable. You want to stop for now and practice again later? No problem. Want to practice over winter break or at two in the morning? Sure.

Consistency is another potential strength. In theory, AI can maintain a uniform treatment without biases or stylistic idiosyncrasies muddying the feedback.

What Are The Limitations Of AI Coaching?

L&D developers must recognize that AI coaching lacks metacognition or self-awareness. Unlike humans, AI lacks a big-picture understanding of what it’s doing and never knows how it’s coming across. It operates solely on algorithms, responding to prompts without considering the broader context, as a human coach would instinctively do. An AI model’s responses are based on weighted associations and patterns it finds between words given the terabytes of text it has been fed. It doesn’t possess inherent common sense and can’t discern the bigger picture behind a prompt. This is why meticulous training of AI and vigilance regarding its application are essential to ensure that AI responses contribute toward the intended behavioral change.

AI represents a broad family of algorithms. Here, we’re focusing on a subset, namely large language models (LLMs), which excel in conversation-based domains. AI coaching, within this context, aims at helping individuals improve their communication skills, particularly in conversations. The ability to evaluate learners’ conversational prowess and provide feedback offers tremendous potential because many (most?) roles require the ability to articulate well.

What About Transparency? Should We Make Learners Aware They Are Engaging With AI Rather Than Human Coaches?

Transparency is a fundamental aspect of ethical AI coaching. It’s essential to explicitly inform learners that they’re interacting with a machine, not a human being. Transparency helps set expectations and prevents the “uncanny valley” effect, where learners become uncomfortable interacting with something they think is human but proves otherwise. As AI coaching becomes more prevalent, transparency should become the norm to gain learners’ trust and ensure their comfort. In the systems SweetRush has built that use AI, we used an image of a robot to deliver the coaching, making it clear it is coming from the machine.

What Is Required To Develop An AI Coach?

Successful implementation requires the following considerations:

• Understanding the problem to be solved – L&D developers must be able to articulate what “good” performance means and how to recognize it.
• Need for authentic skill exercise – If teaching verbal communication skills, the training solution should allow learners to speak their response, which must be converted into text.
• Verbal feedback – If you want the AI coach to deliver coaching verbally, then a text-to-speech algorithm is needed.
• Server requirements – A server is necessary to hold the AI and to store and analyze data. Often, several are involved, such as one to support the learning experience, another to host the AI, perhaps a third for speech-to-text or text-to-speech processing.
• Factors for success – Developers must also design the learner experience, manage the project, implement data security, develop an architecture, and possibly distribute and manage headsets.

What Is The Process Of Converting Human Speech To Text In The Context Of AI Coaching?

Human speech-to-text conversion is a useful component of AI coaching for learners exercising verbal skills. If you want to get learners better at responding to customers or colleagues, you should let them speak, just as they do in real life. At SweetRush, we use specialized services, including those offered by Google, to perform this task.

Here is the algorithmic process by which a learner might engage with an AI coach verbally. Let’s use the example of a learning experience enabling learners to practice problem resolution—that is, teaching them how to best resolve customer problems. In one we built, we used prerecorded avatars to act as “customers” who voice complaints. After a learner enters the learning experience and is familiarized with their role, a simulated customer comes up to say something. Learners are asked to voice a response, captured via their microphone as an audio stream and converted into text. This text, prepended with a prompt instructing how to evaluate the response, is fed into an LLM. The LLM then performs an analysis and generates feedback, which is then shared with the learner. We deliver this feedback via audio because we’ve found that having an AI coach speak a response is more palatable to learners than having to read it. But note: if that feedback is delivered by AI, transparency dictates we tell the learner the feedback is AI-generated, human-like though it sounds.

Is It Necessary To Have An Existing Behavioral Model To Add AI Coaching To A Simulation?

Absolutely. Most organizations have either adopted models for assessing skills and behaviors or defined their own because they know that some concrete notion of “correctness” is required whenever you want to provide meaningful guidance to someone doing a task. Same with an AI coach. Furthermore, categorizing or chunking behaviors into distinct groups or steps and using mnemonics can make the model easier for learners to master, potentially reducing cognitive load and making the experience more relevant and memorable.

Here’s an example. If we’re teaching employees at a retail store how to greet customers, we might adopt the “SMART” model that consists of the following steps: Smile, Make eye contact, Ask what they need, Refer them to a department, Thank them. Such a behavioral model provides a structured foundation for AI coaching and also informs learners what behavior is expected in an easy-to-remember way. It also allows an L&D developer to design scenarios that efficiently allow learners to practice the model and provides a way to tell the LLM what is right or wrong in a response.

A behavioral framework is fundamental to creating an effective AI coach.

Why Can’t An AI Coach Function Effectively Without A Behavioral Framework? Why Not Rely On Its Inherent Smarts?

Unless you can tell an AI what is good and bad behavior, you are leaving it to its own devices to do so, with no guarantee it will succeed and no way to improve it. Organizations often have unique processes and contexts, and an AI on its own likely won’t accurately reflect their nuances.

What Is The Process Of Training The AI Coach?

As mentioned earlier, to ensure that an AI coach is providing useful feedback to learners, it needs to be fed a specific model of performance “correctness.” Training AI means providing clear direction on how it needs to evaluate learner responses. One technique SweetRush uses is to first create mock conversations that cover a range of scenarios. Some reflect where the model is executed flawlessly, while others intentionally include failures. We then use these scenarios to train the AI on how to spot good and poor behaviors.

What Role Do Prompts Play In This AI Training?

Training an LLM means fine-tuning prompts, which are essentially “recipes” that guide the AI coach’s responses during mock conversations. The prompts shape the AI’s behavior and ensure it provides feedback aligned with the model.

While all aspects related to an AI coach experience need to work, we believe the training of the model and its ultimate correctness is the most important. It’s an iterative process that requires specific knowledge to do right.

Why Is Creating A “Character” For The AI Coach Useful?

Defining the character and personality of the AI coach is another consideration. For a learning experience to make an impact, learners must embrace the AI coach’s feedback. If the feedback is delivered in a dull, disembodied way, learners may disregard it. Conversely, providing an AI coach with the context, attitude, and perspective of a character (say, an expert who is deeply knowledgeable about a subject but may have a point of view and blind spots) is likely taken more to heart.

How Do You Assess The Accuracy Of AI Coaching, And What Methodologies Do You Employ To Ensure It?

Let’s agree that, for an AI coach to be successful, it must deliver feedback that is accurate. Put bluntly, its feedback must correctly appraise a learner’s performance against the behavioral model the organization has adopted as most useful to achieve its goals. SweetRush has taken a novel approach toward this. We compare the AI’s evaluations of learners’ performances to those of human coaches evaluating the same performances. The human coach’s feedback serves as the baseline of correctness that we can compare to an AI coach’s feedback to systematically improve the prompts until the AI coach’s feedback is accurate enough for general release.

Data warehousing becomes a factor here. With a multitude of AI responses possible, we need tools to coach the AI coach efficiently and at scale. We’re developing feedback tools that allow experts (and clients) to provide feedback on AI responses, including a thumbs-up/thumbs-down with an explanation, and use this data to improve the AI coach systematically.

What Are The Challenges Involved In AI Coaching?

Currently, AI coaching via an LLM works best when evaluating a learner’s verbal response at a single moment in time. For example, a customer complains and the learner replies using the SMART model. However, when interacting with a customer, one typically doesn’t make a single response. One engages in a conversation with a series of back-and-forths. It is challenging to apply AI coaching to such multi-step interactions because the AI must retain memory of everything said as the conversation progresses, not just what was said at a given point in time. SweetRush is researching how to enable AI to provide coaching on multi-step interactions, but there are many interesting challenges associated with this goal.

Exploring The Promise And Challenges Of AI Coaching

AI coaching has much potential in Learning and Development because it offers a unique approach to skill improvement, guiding learners to self-reflect, fostering metacognition, and encouraging self-directed learning, available on demand and at scale.

Learning experiences based on AI coaching can also provide valuable data for performance analysis, particularly in areas like sharpening communication skills.

But learning professionals need to be aware of the limitations of the AI they use for coaching, notably its accuracy. Without proper oversight, AI can do more harm than good. Therefore, a plan for testing and continuous evaluation is essential.

Exploration of this technology is in its infancy, and much will be learned in the months to come. Done well, it promises to become an effective learning tool. SweetRush is working with multiple client-partners willing to explore the envelope of these technologies. If you’re interested in discussing an AI project with us, please reach out; we would love to hear from you.