A rubric may be the most common way to capture feedback about an application, but not all are equal. If you’ve inherited an old version that has your head spinning or have been told to build one from scratch, these five questions will help you think through what needs to be covered.
Will each criterion help us make decisions?
Is the core criterion easy to identify?
Are the corresponding application questions or materials easy to find?
Are the rating scales appropriate?
1. Will each criterion help us make decisions?
We’ve all seen it: The rubric with 10 criteria when only 4 or 5 are really considered during the final evaluation process. If this is a reoccurring process you can review past cycles, but if the process is new review your goals for the program and remind yourself of key requirements. Go through each criterion and make sure it’s worth factoring into both the overall score and as a standalone value.
2. Is the core criterion easy to identify?Sometimes it’s tempting to write long review questions to make sure all aspects of the criterion are considered. But don’t say “How does the applicant’s scholarly pursuits align with the overall goals and strategic plans for the department?” when “Academic Fit” will suffice. Start each question with the shortest version of the criterion and save longer text and prompting questions for instructions.
3. Are the corresponding application questions or materials easy to find?Put your rubric side by side with the application and compare. Do they generally flow in the same order so that earlier rubric criteria can be evaluated using the first part of the application? If not, consider re-ordering rubric questions or adding specific instructions for reviewers on where to look in the application to answer the question. This is particularly helpful for specific attachments, such as a budget or reference letters.
4. Are the rating scales appropriate?Using rating scales that are the same for each criterion can give the cleanest scores later from a data perspective but consider adjusting scales to weight certain questions. A key question could have a 1-10 scale, with others only 1-5. If the scale is large, give reviewers a description for at least the lowest, middle, and highest scores as anchors.
If there is an established rating scare for the type of program being reviewed (i.e. NIH ratings go from 1 to 9 with one being the best score), consider mimicking the scale for your rubric.
5. Are ratings enough?Top applications are rarely determined solely by a numeric score. Reviewers often need to provide comments alongside their scores. Consider prompting them to give unique comments per criterion, or at least a separate area for strengths and weaknesses. These extra notes become critical when making final decisions on applications with similar (or even identical) scores.
Giving examples in your instructions can help your reviewers know what feedback is the most helpful to make your rubric a success.
And finally...
Test your rubric
If time allows, our best advice is to run the rubric by some of the actual reviewers so you can address any questions and clarify instructions before the review process is underway. Doing so will save time for everyone involved.
If you're looking to optimize your review process and make the most of your reviewers' time, InfoReady is here to help. Our comprehensive form-building and review workflow software is designed to streamline the evaluation process and improve decision-making. When you're ready to see how InfoReady can enhance your review processes, reach out to us and we'll show you firsthand the benefits it can bring.
