We continue with the third of four video excerpts of Dr. Leslie Thomas’ presentation at the Association of Test Publishers’ Conference 2019.
That presentation was called The Four Immutable Laws of Certification Success: A Psychometrician Speaks.
Today’s video focuses on Law #3: Recruit Your Subject Matter Experts (SMEs) Carefully & Optimize Their Time.
You can find a summary of Law #3 below the video.
Academic or Product Knowledge Vs. On-The-Job Knowledge
Recruiting SMEs is a challenge in the industry. There’s really a difference between people who have academic and product knowledge in the IT/software space versus people who have on-the-job knowledge.
Instead of measuring book knowledge per se, an effective credentialing program measures knowledge as it’s applied on the job. You want to measure that job knowledge for a credential in order to provide value in the marketplace. Book knowledge may not provide value to your customers, but on-the-job knowledge will. Subject matter experts who come from academia are helpful additions to the team, but you want to make sure that the team isn’t stacked with academicians with their tendency toward book learning and theory as opposed to practitioners who bring real-world knowledge and practice.
Likewise, you may have SMEs who are product-oriented and more focused on features but may not realize how they’re used on the job. Make sure that most of the SMEs on the team are actually doing the job in their job role.
Embedded SMEs
“Embedded SMEs” are helpful to the test-development team. For example, if your client is an IT/software company, embedded SMEs would be individuals working in the company’s actual product group. They can provide a valuable heads-up about future releases that will affect test content. In fact, in larger IT companies, the credentialing team often works closely with the product group, even requesting a member from that group to be an embedded SME who can alert the development team to releases that will impact test content.
This advance notice is valuable because content changes affect training, item writing and beta testing. There are multiple processes and timelines that are impacted by changing content, so the earlier you know about that, the better you can prepare all stakeholders.
The same goes for any regulatory changes that may impact the credential. Early warning facilitates item writing, for example. If SMEs are aware of coming changes to the job role, they’ll be able to write better questions that stay abreast of current practice. So it’s helpful to have SMEs who sit on regulatory boards or work in product teams. Also consider people who work in support functions like help-desk environments for IT or even medical help desks. They deal with real-world issues every day and can be valuable sources of information.
SMEs and the Minimally Qualified Candidate (MQC)
You should (bordering on must) have one SME with the same level of expertise and experience as the minimally qualified candidate. None of us remember the first time we used a mouse for a computer, but that cursor was very likely darting all over the screen. Now, we use the mouse with ease. It’s so simple. That’s the way your SMEs are.
Leslie shares a story about a technology company that wanted to develop a credential for sales people. They used their sales engineers to develop it and the resulting test was far too difficult for sales people to pass. So it became a credential for sales engineers.
So having a SME who matches the minimally qualified candidate gives you a valuable reality check. Many of your subject matter experts are experts by definition. They’re always, typically going to want to make test items more difficult than they need to be. The SME who matches the MQC can speak up and let the other SMEs know when items are written above his/her level of knowledge and experience. This SME has the effect of level setting the experts.
Pick Your SMEs For “Fit”
In pre-employment testing there’s a saying, “Hire for skills. Fire for fit.” Some SMEs can be disruptive and negatively impact the group dynamics. This should be addressed right away, otherwise, resentment can fester because others may not speak up about the disruptive SME. If allowed to continue, the result will be a lower-quality credential and the loss of your most valuable resource.
Work ethic is important for SME groups. The tendency to socialize can be distracting and impede progress.
Recruit More SMEs Than You Need
We recognize the added work in doing this. There are some SMEs who just won’t be able to learn how to write items. They’ll struggle. They can still be valuable for reviewing items and working in teams, however.
SME Games (Inspired by the Hunger Games)
The last SME wins. This is where you have 6-8 SMEs who go through a technical review. When their items are sent to someone else for review, that person declares the item(s) to be substandard, claiming, for example, that they don’t measure the right thing. Letting this last SME make the final decision effectively negates the work of the other 6-8 SMEs.
So you have to be careful handling SMEs. They’re experts and have strong opinions. You want them to have strong opinions. But each person’s opinion is just one opinion and needs to be understood in context. Technical review is an iterative process and often the SME who thinks he’s right isn’t.
The dominant SME wins. If there’s a bully in the group, others will shut down, thereby creating more weight for the bully’s opinions. It’s worthwhile taking the bully aside and letting him know that the opinions of other people need to be heard. Often calling on the others individually will provide balance.
Call on different SMEs. Calling on them, especially in a virtual environment, encourages them to participate and diminishes the impact of the same people speaking up all of the time. Calling on people in a round robin keeps them on their toes, prevents them from multi-tasking during virtual meetings and ensures that you’re getting everyone’s opinion.
SMEs Should Be Focused on the MQC
Make sure that your SMEs are focused on the minimally qualified candidate. That’s obvious. But your SMEs are experts and will make items more difficult than they need to be. The items may be at the appropriate level of difficulty, but you should check that just the same. On several occasions, beta testing stats have confirmed Leslie’s concern that the items were too difficult for the MQC.
NDAs and Prohibited SME Behavior
A very important thing to remember is to have SMEs sign Non-Disclosure Agreements (NDAs) and make sure there’s a prohibition on working on training and test-prep activities. There have been cases where SMEs, understanding the value of being privy to confidential test content, launched their own test-prep companies.
Similar to test-candidate agreements, NDAs must be very specific about prohibited activities for SMEs. If you don’t spell out those prohibited activities, SMEs may pretend that they don’t know that sharing test content isn’t permitted. The more specific your NDAs are about acceptable/unacceptable behavior, the more secure the test-development phase will be.
***
See the Videos for Laws 1 and 2!
If you missed the first two laws of Certification Success, just click the links below:
Law #1: Identify the Primary Problem the Market Will Pay You to Solve
Law #2: Document Program Goals & Major Decisions BEFORE YOU BEGIN
Contact Us! The Kryterion Psychometric Team would love to learn more about your first or next certification exam. You can start the conversation here.
