The importance of doing a post-match analysis.
The purpose of creating a post-match survey for your applicants is to get feedback on what your program is doing well and where there is room for improvement. For example, what did your program do well? In what ways could the program improve? Is there something your program is doing differently than other programs that you want to assess? Are you effectively messaging your program’s mission and values to the applicants?
The first step in creating a post-match survey is to analyze and review the NRMP “Match Results by Ranked Applicant” report for your program. The report is available to any program that uses ERAS & NRMP for recruitment and is usually available the week after Match. To locate this report, log into NRMP and from the top menu, select “Options” —> “Reports” where you will see a list of available reports. Available data include characteristics of Matched applicants, as well as the Match results of those applicants on your Match list than ended up at other programs. A careful and deliberate analysis of this data set may allow you to determine:
Which program(s) are your largest competitors?
Is the program attracting the “right” type of candidate?
Is the program effectively messaging its values & mission?
Do applicants Match at similar training programs in other cities, or do they seek other specialties or types of training environments?
Discussing this data with faculty, trainees, and students may lead to changes in your recruitment or candidate selection processes. Similarly, subgroup analysis and/or feedback from certain types of candidates may allow further refinement of efforts/strategies, and perhaps a separate or additional survey may be geared towards that group directly.
Examples of such subgroup analysis might include:
Non-Matched Applicant Survey
Under-represented Minority (URM) Survey
Survey for applicants that matched to your program’s largest competitor
Matched Applicant Survey
Creating an optimally designed survey requires careful consideration of a number of factors. In an effort to keep the survey as short as possible (and thus hopefully increase your response rate), give thought to whether or not to include factors that may not be within your control such as geographic location of the program, cost of living, program size, or program mission. Instead, one might focus on issues related to faculty/resident/staff interactions, areas of expertise/focus within the program, communication from the program to the applicants, applicant dinners and social events, "second look" opportunities, etc...
Similarly, it is essential to thoroughly consider the different types of questions that can be used in a survey, such as open-ended (text box; comments), dichotomous (True/False; Y/N), or rating scales (1 —> 10; Poor —> Excellent). The data that your program requires should dictate which scales are to be used.
As an example, my program prefers to use open-ended questions and general comment sections whenever possible. This gives applicants the ability to be specific with their feedback, enabling us to better understand if there are outliers in the applicant pool or if there are anomalies that happen on one specific interview day.
However, sometimes specific questions are very helpful. Some may want to investigate and probe in areas such as: Was there information that you wish was covered in the introduction to the day? Did you feel that you had enough interaction with residents on the interview day? Do you have comments or thoughts about the wrap-up session? Pay particular attention to the wording of the questions, and you might even consider enlisting the expertise of a survey expert — the feedback that you receive will only be as good as the questions that you ask!
Consider how the questions below might shape the feedback that you receive.
“Was our program website helpful?” Yes / No
“What information on our website was most helpful to you as an applicant?” [Comment Box]
“Please rate the information on our website vs other programs you interviewed with…" 1(Poor) —> 10 (Excellent)
“Did you review our program website prior to attending an interview day, and if so please use the comment box to indicate what you found helpful or unhelpful?” Yes/No + [Comment Box]
Each year in our post-match survey, we ask “What aspects drew you to the program with which you matched?” In some instances, we were shocked by the applicants' responses as we felt that the reasons that many listed were values or aspects of our own program! Learning this, we altered our interview day in subsequent years to highlight our program’s mission and values. We had assumed applicants knew our program's mission, so we had not previously put much thought or effort into highlighting this during recruitment.
Lastly, the timing of the survey depends on the specific goals or needs of your program. Some have applicants complete on the day of the interview, some send out one survey to all applicants at the conclusion of interview season, and others send surveys post-match day, or any comination! If your program plans to send surveys pre-Match Day, be very careful in your survey questions. You cannot ask anything about ranking, indicate anything about applicant ranks, and you should include statements about completion or non-completion of the survey has no bearing on your program’s applicant rankings. Again, it depends on the specific needs of your program.
Once you have the program’s feedback goals, a completed survey form, and decided on the timeline for completion of the survey(s), it’s time to send them out and start analyzing that data to determine of changes or adjustments are needed to your processes.
In conclusion, when developing a post-match survey it is best to think about the entire interview process and consider:
What specifics does your program want feedback on?
Who does your program want feedback from?
Is your program using the best types of questions to gather the most effective information?
When is the optimal time for your program to gather the information?
Best of luck with your surveys, please share in the comments any questions or comments!