NOVEMBER 2006
Are You Creative or Reactive?
By Soley Somma, Editor
If I asked you to name creative professionals you might list artists, graphic designers, writers, interior designers, architects, etc. These individuals are paid (in most cases), to think differently and to question the status quo. I'd like to think that our profession, the testing industry, is full of creative professionals, and I am not just talking about the marketing people.
I think that testing and certification program managers and administrators could be the most creative individuals in our industry. I say could be, because in most cases this group usually has the least amount of potential creative time. But why be creative anyway? Being creative gives you an advantage. It allows you to switch from dealing with what's happening now (putting out fires), to finding ways of doing things differently (fire prevention). Move the 'C' in Reactive to the front and you have a new mindset. So which are you - Creative or Reactive?
Replace Complacency with Creativity
You might think that you are too busy to be creative. But being creative doesn't mean that you have to replicate the environment of a writer's world or an artist's studio. Being creative means evaluating variables in a new way and making decisions based on new perspectives, not habit. Habit and routine make us comfortable, confident and dare I say complacent; even though deep down we might think that there should be a better way or even an easier way.
Technology and Data are Your Design Tools
One way to be creative is to look at your program data in a new way. Creative analysis means accessing data that is relevant to you and your organization and using it to come up with new solutions. Get your hands into your data and slice it differently, compare it to different time periods, analyze it by different demographics, correlate cause and effect, identify trends, forecast activity, pinpoint areas for improvement, prevent challenges, etc.
You can do this on a daily, monthly, quarterly, yearly basis or any other time period that makes sense to you. This is your way of switching from reactive to creative. In other words, examine your data when there isn't necessarily a need or a fire to put out, but a want to think differently. And don't think that you have to be a 'large organization' to make creativity work for you. Programs of all sizes have the same opportunity to manage and analyze their data.
Use Your Time Differently
I am sure you are creative everyday in numerous aspects of your life - both personal and professional on an impromptu basis. You don't have to find more time in your workday to be creative. Just take the time you used to spend chasing down your data and allocate it to reviewing your data. How do you do this? Find a way to quickly access your data on your terms and make your data work for you. Take control and let that inner creative program genius bring on progress in your organization!
Some Say it's Impossible. We Call it Progress.
Einstein at Work: How to Use Reports Like a True Program Manager Genius
By Tim Holladay, Client Services
Winston Churchill once said, "True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information." If you manage a testing or certification program, congratulations, you are a genius!
Like most program managing geniuses, you understand the importance of collecting data and reporting on your program. It's important to have your data at your fingertips, because if knowledge is power then reporting is the extension cord of your testing program. How can you power your program effectively and efficiently if you can't plug it in to the data that you are collecting on a daily basis?
Example of Real Life Genius at Work
John is the administrator for a certification program. The board was meeting the following day and they requested some last-minute information about the program. The board wanted to know where their Test Taker base was and how many in each area. They also wanted a year-to-date program revenue report. In the paper-pencil world this would have taken hours, maybe even days.
Fortunately John had real-time online secured access to his testing data. He was able to run a quick and comprehensive report. He picked his fields - test name, city, state, country, complete date, and score - and ran a report. The data was filtered to include YTD data and grouped the data by state.
He then ran a financial report based on the test, payment method, purchase price and date purchased fields. He then constrained the data to the YTD. He assembled and ran the reports in about 120 seconds, exported to and formatted in Excel in 90 seconds and sent it to the appropriate people in another 60 seconds. How long does it take you to access and analyze your data?
Making Your Data Work for You
Aspiring testing program Einsteins may ask, "Now that I have the raw data, what reports are most important?" Critical reports for your program will depend on your setup and needs. However, there are few reports that are important to most programs. These include:
Weekly
- Test Taker Registrations - identify trends in registrations and correlate them to your efforts to increase registrations.
- Assessment Results - How many passed and who, where and when?
- Financial - Monthly test program revenue
- Test Taker Feedback - End of test comments or survey data
- Marketing - Test Taker names, addresses and emails
- Item Analysis - How are the items and test form performing?
- Demographics - Where is your Test Taker base?
- Total tests taken, passing rates, financials
When creating reports remember the words of Martin Fischer, "Knowledge is a process of piling up facts; wisdom lies in their simplification. "Clear, concise and accurate reports are the "Theory of Relativity" for program managers in that all of the data can be consolidated into more basic terms (like Einstein's E=mc2).
Using tools like Webassessor, KRYTERION's online, secured test delivery system, anyone with access to the Internet and the proper administrative rights can run any of these reports at any time. The Webassessor system records detailed level data on your testing program and can quickly produce Excel, PDF and CSV exports on the data. In addition, your favorite reports can be saved and run over and over, for free.
Proactive Program Management
Effective reporting allows you to be proactive and creative. It empowers you with the data to make and justify your program management decisions.
Now that's genius.
Newsletter Archive
Technology Spotlight
Q. As a Program Manager, I need to run frequent reports on my Test Taker activity. How can I streamline my reports to meet the needs of different audiences?
A. Webassessor's flexible reporting allows Program Managers to create specific, customized reports based on one of the eight pre-defined Reports template. You can save your customized reports to your reports library for quick access in the future.
The first step in creating a customized Report is selecting the applicable pre-defined Report Template. Access the Reports Templates from the Reporting, Self-Service tab. Name your Report and create a Report Title. Next, select how you'd like to group your report fields. You can then customize the report by selecting the Report Fields you'd like to include on the customized Report, change the order of the Report Fields, and change the appearance of Report Fields including labels and layout.
Once you have configured all Report Fields, use the Reports Criteria to define the specific data you'd like included on the Report. This is where as a Program Manager you can really get creative with your reports. Would you like to know all Test Takers who registered for a specific test during the last week of October, purchased using a MasterCard, have completed the test, live in California and have a last name that starts with the letter A? Simply select the Reports Criteria you'd like to use and the specific data you'd like to report on and you've got a real-time report with the exact information you need.
Want to Learn More?
Webassessor clients can learn more about flexible reporting by contacting the Technical Account Management Team or contact one of our Business Development Managers for a personal demo at info@kryteriononline.com.
That's taking control. That's Webassessor.
Psychometric Corner
Psychometric Alchemy: Turning Poor Items into Gold
You just finished beta testing some items, conducted an item analysis, and selected your best performing items to go into production. So what do you do with the remaining items that didn't make the cut? Do you throw away all the hard work, time, and money that went into creating them?
Well, the good news is you don't have to IF you have the right information at your finger tips and know how to use it. Item statistic reports can give you valuable insight as to how to revise an item to make it perform better. For instance:
1. Item difficulty values (i.e., p-values) can tell you if an item is too easy or too difficult.
2. An option analysis shows how many examinees chose each answer option and can help you determine which option(s) need to be revised to make an item easier or more difficult.
3. An item comments reports provides the examinees' qualitative feedback on each item. Often there will be explicit suggestions as to how to improve an item.
With some training, all of this information can be used by subject matter experts to improve items that didn't initially make the cut. Once the items have been revised, they must undergo beta testing and analysis again before being scored on an operational exam. That said, it's much easier, more efficient, and cost effective to leverage what you have as much as possible than start from scratch!
Wanted: Your Ideas
Thank you to everyone who attended our last webinar. We're now looking for suggestions for future webinar topics. Please send your ideas to pr@kryteriononline.com and we'll try to incorporate it into our webinar series.
Send to a Friend
Click here to send to a friend
Comments?
Please send us your questions, comments or editorial ideas to Quinn Sutton at pr@kryteriononline.com.
Press ContactQuinn Sutton |