The behind-the-scenes work in research development can often be incredibly time consuming. Nowhere is this more evident than engaging reviewers, specifically recruiting, assigning, and managing reviewers. In a recent webinar, Joe Farmer, Program Manager and Data Analyst at Boston University (BU) shared the details about creating and maintaining faculty review panels at BU. Along the way, the changes made their processes more efficient and transparent for applicants, administrators, and the reviewers themselves.
Prior Process
After submission deadline passed, reviewers were assigned on ad hoc basis.
Aimed for 3-5 reviewers per proposal, which Joe noted, “...led us to relying on a very small group of fairly senior faculty members...that we had a closer relationship with as an office of research.”
Rating scale was 1-10, which was simple and straightforward, but did not provide granular detail they needed.
Standard turnaround timeline (2 weeks) was difficult to meet.
New Review Format
Created new scoring template using the "4 Comments & Ratings Combination" Routing Step in InfoReady, with scales 1-5 for each of the four new criteria being assessed:
Application strength
Specific fit to funder/likelihood of successfully obtaining funding
Benefit to University if successful
Benefit to society if successful
All review criteria helped answer the question: “What are some of the less tangible benefits to this proposal that might happen or might be more clearly defined [or identified]?”
Joe worked with prior reviewers, administrators, and faculty to develop rubric.
Two Key Successes for Reviewer Feedback
More granular scoring: Better ability to assess nuances of applications, especially mission-based opportunities (like foundations).
Encouraged reviewers to provide better detail in their feedback: “We have a system of anonymizing it and providing it to faculty whether they’re selected or not.” Sharing constructive feedback helped with their professional development, research, and improving future submissions.
New Faculty Review Panel Outcomes
Standing review panels created for 1-year terms, which allows them to reassess composition on an annual basis to add expertise as needed.
Mix of senior and junior faculty. Joe emphasized the importance for the professional development of junior faculty: “This is a really beneficial opportunity for junior faculty to kind of see another side of the grant-getting process, to learn and hone their own skills, and for service to the institution.”
The review panels became more inclusive and diverse.
More easily meet their 2-week review window: “Within an hour of a deadline passing, we can have the materials distributed to the reviewers.”
We want to thank Joe Farmer for sharing his experience in building and maintaining review panels. The combination of timesaving and more detailed feedback to applicants were the exact outcomes BU was hoping to achieve.