August 2007
Three Common Threats to Validity
By Leslie Thomas, Ph.D.
Is a test or an item valid? I hope that most of us know that the answer to this question is a resounding "no". Validity refers to the appropriateness of the inferences we make and the actions we take based upon the test scores.
That said, I am often asked whether a test is a "valid test". Because it is easy to forget that validity has to do with the interpretation and uses of test scores rather than a property of the test itself, we sometimes forget that even a "good" test can give us "bad" information.
1. Test Security
For instance, say you have spent a lot of time and money on developing a quality exam. What if a Test Taker goes to a "brain dump" site and memorises the answers to the exam? Would the Test Taker's scores have the same meaning as someone's scores who did not cheat? In this instance, would a "good" test be giving us "bad" information on the Test Taker's ability level?
Test security significantly impacts score validity. Not only do you need to evaluate the security of your test-development practices, but also the processes you use to administer your tests and validate the scores.
2. Inappropriate Interpretations and Uses of Test Scores
One way to be creative is to look at your program data in a new way. Creative analysis means accessing data that is relevant to you and your organisation and using it to come up with new solutions. Get your hands into your data and slice it differently. Compare it to different time intervals, analyse it by different demographics, correlate cause and effect, identify trends, forecast activity, pinpoint areas for improvement, prevent challenges, etc.
Let us say that you developed an exam to determine whether someone has the prerequisite knowledge necessary to take a technology training class. If an employer decides to use the exam to make hiring decisions, is that a valid use of the exam based on its design? No
To protect yourself from inappropriate interpretations and uses of test scores, make sure you specify - in writing - the intended interpretations and uses of test scores in your technical reports, test administration guides, and/or instructions.
3. Inadequate Test Development Practices
During the test development process, we gather validity evidence to demonstrate that the inferences and decisions we make, based upon the test scores are appropriate. Every step in the test development process, including job task analysis, item writing, passing score study and final forms construction, is an important link in the chain of validity evidence. Skipping or skimping on test development steps significantly weakens your chain of validity evidence, which you will need if the exam is ever challenged. Having a test-development expert will help to ensure that you follow the best practices and help you fully document validity evidence.
Above are just a few of the common threats to validity. Unfortunately, there are many others. When developing, delivering, and maintaining a test, it is important to have access to the expertise that will help you avoid these threats and others.
Testing Connections
Here are just some of the things you should know about in the testing industry...
www.caveon.com/resources_news.htm
www.testpublishers.org/europe.htm
Comments?
Please send us your comments or editorial ideas to euonline@kryteriononline.com
Newsletter Archive
Technology Spotlight
Q. As a Program Manager, I need to run frequent reports on my Test Taker activity. How can I streamline my reports to meet the needs of different audiences?
A. Webassessor's flexible reporting allows Program Managers to create specific, customised reports based on one of the eight pre-defined Reports template. You can save your customised reports to your reports library for quick access in the future.
The first step in creating a customised Report is selecting the applicable pre-defined Report Template. Access the Reports Templates from the Reporting, Self-Service tab. Name your Report and create a Report Title. Next, select how you'd like to group your report fields. You can then customise the report by selecting the Report Fields you'd like to include on the customised Report, change the order of the Report Fields, and change the appearance of Report Fields including labels and layout.
Once you have configured all Report Fields, use the Reports Criteria to define the specific data you'd like included on the Report. This is where as a Program Manager you can really get creative with your reports. Would you like to know all Test Takers who registered for a specific test during the last week of July, purchased using a MasterCard, have completed the test, live in Birmingham, West Midlands and have a last name that starts with the letter A? Simply select the Reports Criteria you'd like to use and the specific data you'd like to report on and you've got a real-time report with the exact information you need.
Want to Learn More?
Webassessor clients can learn more about flexible reporting by contacting the Technical Account Management Team or contact one of our Business Development Managers for a personal demo at euonline@kryteriononline.com.
That's taking control. That's Webassessor
Send to a Friend
Click here to send to a friend