AN:

A A M ( /
AAMC Supplemental ERAS® Application: Elactroeibe Rasidanncy
Key Findings From the 2022 Application Cycle Application Service®

AAMC Supplemental ERAS® Application:
Key Findings From the 2022 Application Cycle

Background

The AAMC partnered with U.S. residency programs in dermatology, general surgery, and internal
medicine to offer the supplemental ERAS® application for the 2022 application cycle. The purpose of the
supplemental ERAS application was to:

o Update the myERAS® application to reflect the current state of residency training.
o Empower applicants to share more information about themselves using a fair process.
o Drive holistic review in a high-volume application context.

The supplemental ERAS application was administered Sept. 1-30, 2022, and consisted of three sections:
past experiences, geographic preferences, and preference signals. (Analysis of a fourth, research-only
section of that application, work preferences, is ongoing.) A total of 720 unique programs participated.
Some internal medicine programs offered both categorical and preliminary tracks. Participating programs
represented all geographic regions, university- and community-affiliated programs, and programs of
different sizes:

o Dermatology (87% participated, 117/135 programs).

o General surgery - categorical (71% participated, 232/327 programs).

o Internal medicine - categorical (64% participated, 361/566 programs with
categorical tracks).

o Internal medicine - preliminary (75% participated, 184/246 programs with
preliminary tracks).

About 83% of applicants to these three specialties submitted the supplemental ERAS application
(22,900/27 ,496). (Applicants can apply to more than one specialty, so percentages may exceed 100.) The
demographic composition of the sample was similar to the population of applicants from these specialties
(as of January 2022), although slightly fewer Asian applicants to internal medicine (categorical and
preliminary) submitted the supplemental application compared with all applicants from internal medicine.

e Dermatology (93% submitted, 950/1,019 applicants)

e General surgery - categorical (87% submitted, 3,578/4,115 applicants)

e Internal medicine - categorical (82% submitted, 15,486/18,986 applicants)
e Internal medicine - preliminary (74% submitted, 7,802/10,511 applicants)
Methodology

The evaluation plan centers around fairness, psychometrics and validity, program director use and
reactions, and application and advisor reactions. In the first phase of the evaluation, we explored
applicants' responses to evaluate fairness, psychometrics, and validity. We explored how program
directors used the supplemental application and their perceptions of how it added value.

Applicant, advisor, and program director surveys were administered between October 2021 and January
2022. Although survey results should be interpreted with caution given the low response rate and small
sample sizes, the samples represented a variety of geographic regions, university- and community-
affiliated programs, and programs of different sizes:

© 2022 Association of American Medical Colleges 1 aamc.org
AAMC Supplemental ERAS® Application:
Key Findings From the 2022 Application Cycle Application Service®

Electronic Residency

Applicant survey (36% responded, 8,207/22,899).

Advisor survey (MD schools: 62% responded, 96/155; DO schools: 37%, 14/38).
Dermatology program director survey (44% responded, 52/117 programs).
General surgery program director survey (34% responded, 79/232 programs).
Internal medicine program director survey (30% responded, 108/366 programs).

This document summarizes key findings from the first round of analysis of supplemental application
responses and survey results. It describes how current findings will inform changes to the design and
administration of the supplemental application in year 2. For detailed survey results, refer to AAMC
Supplemental Application Data and Reports.

Findings

Past Experiences

e Over 90% of applicants reported four or five meaningful experiences, and most applicants
included a brief description about their experiences. More than 66% of these meaningful
experiences required medical or health care knowledge. On average, the duration of experiences
reported was 12 months. The most-reported experience types were:

3

oO
°
oO

Education/training (13%-22%).
Research (14%-23%).
Volunteer/service/advocacy (22%-27%).
Work (11%-19%).

e There were no or small differences in responses to past-experiences questions by gender and
race/ethnicity. Graduates of medical schools outside the United States (IMGs) reported more
medical/health care-related, education/training, and work experiences than did graduates of MD
and DO schools.

e Just under half of applicants (46%) thought the experiences essays empowered them to highlight
who they are to programs.

e Program directors reported mixed reactions to the five-most-meaningful-experiences section.
Most of them reported using these during their initial application review (85%) and when deciding
whom to interview (80%). However, they also reported the section was redundant with
information from the myERAS application. We will consider updating guidance to applicants to
minimize redundancies with the existing application.

e About 70% of applicants responded to the Other Impactful Life Experiences essay. Asian, Black,
and Hispanic applicants responded at higher rates than White applicants. DO and IMG applicants
were more likely to respond than MD applicants.

e Although most applicants wrote about topics related to overcoming hardship in answer to the
Other Impactful Life Experiences essay, some wrote about experiences that did not reflect
sustained or extraordinary hardship, suggesting that more guidance about the intent of this essay
is needed for applicants. The most common essay topics were:

oO
oO
oO

Financial hardships (19%).
Serious life challenge, illness, passing of a family member (18%).
First generation to higher education or in medicine (12%).

© 2022 Association of American Medical Colleges 2 aamc.org
AAMC Supplemental ERAS® Application:
Key Findings From the 2022 Application Cycle

e Over 70% of dermatology, almost 50% of general surgery, and 35% of internal medicine program

4 A
A

AK

\ MI ( J

Electronic Residency
Application Service®

directors reported the Other Impactful Life Experiences essay provided important context.

e Based on these data, the supplemental ERAS application for the 2023 cycle will include the past
experiences section. As shown in Table 1, the primary components of this section (i.e., the 5
Most Meaningful Experiences and the Other Impactful Life Experiences essay) will remain.
However, we will streamline the section by reducing the number of questions and response
options to those that provide the most value. We'll also improve guidance to applicants.

Table 1. The Past Experiences Section

 

 

 

 

 

 

 

Element Stays the Same Potentially Change
Content 5 Most Meaningful Experiences Reduce the number of follow-up
questions.
and short essays.
: : Reduce the number of response
Other Impactful Life Experiences options for some questions.
essay.

: . . Update guidance to reflect any
Guidance to Applicant guidance. :
Applicants and changes to the question content
Advisors Applicant and advisor guidance Add guidance and examples

webinars. .

about:

o Overlap between the
supplemental application and
other application information.

o What program directors are
looking for/not looking for in
the Other Impactful Life
Experiences essay.

: rr : Update guidance to reflect any
Guidance to Supplemental application guide :

Program Directors for programs. changes to the question content.
Program director guidance
webinars.

 

 

© 2022 Association of American Medical Colleges 3

aamc.org
AAMC Supplemental ERAS® Application:

Electronic Residency

Key Findings From the 2022 Application Cycle Application Service®

Geographic Information

Note: General surgery did not participate in this section of the supplementary ERAS application.

More than 60% of applicants indicated at least one geographic division preference, with most
indicating a preference for the division aligning with their home or medical school location.

About 33% of applicants indicated they had no geographic preference, suggesting that factors
beyond geography are important to them and/or they were willing to relocate to a new part of the
country. Black and Hispanic applicants (35%-53%) and IMG applicants (44%-51%) were more
likely to select "No Geographic Preference" than other applicants.

About 70%-80% of applicants provided a short essay explaining their preference (or lack thereof).

More than 60% of applicants reported having no preference for a rural setting, and about 50%
reported no preference for an urban setting. More applicants reported a strong preference for an
urban (24%-33%) than a rural (3%-10%) setting. About half the applicants provided a short essay
explaining their urban/rural preference (or lack thereof). It's possible that including separate
questions for both rural and urban preferences was confusing for some applicants, so simplifying
this to one question is under consideration.

Nearly 70% of applicants reported that their responses to the geographic preferences section
reflected their true preferences at the time of their application, and about half of applicants and
advisors thought geographic preference information may help applicants be noticed by programs
they have the most interest in.

This section was widely used and appreciated by program directors. Over 90% of dermatology
and 85% of internal medicine program directors reported using geographic preferences as part of
their holistic decision and/or as a tiebreaker when deciding whom to interview. Furthermore, 64%
of dermatology and 58% of internal medicine program directors reported that geographic
preferences information helped identify applicants they would otherwise have overlooked.

Based on these data, the supplemental ERAS application for the 2023 cycle will include the
geographic information section. As shown in Table 2, applicants will still be able to share their
geographic preferences (or no preference) for divisions of the country and for a rural or urban
setting. However, we will explore changing the geographic divisions and will create a single rural
and urban preference question.

© 2022 Association of American Medical Colleges 4 aamc.org
LA N /\ (
Vi LIVIN
® Co
AAMC Supplemental ERAS' Application: Elactronis Resldancy
Key Findings From the 2022 Application Cycle Application Service®

Table 2. The Geographic Information Section

 

 

 

 

 

 

 

 

Element Stays the Same Potentially Change
Content e Indicate a geographic preference, ° Explore changing divisional
including a "No Geographic groupings.
Preference" option. e Drop the "Do Not Wish to
e Short essays describing division Indicate' response option.
preferences (or lack thereof). © Change rural and urban
e Short essay describing rural or preference to a single question.
urban preference (or lack
thereof).
e Specialties decide whether they
want to receive this information.
: . . Update guidance to reflect any
Guidance to e Applicant guidance. ° :
Applicants and changes to question content.
Advisors e Applicant and advisor guidance
webinars.
: . . Update guidance to reflect any
Guidance to * Supplemental application guide ° :
Program Directors for programs. changes to question content.
e Program director guidance
webinars.

 

Preference Signals

e Over 85% of dermatology, general surgery, and internal medicine - categorical applicants used
the maximum number of signals available (three for dermatology, five for other participating
specialties). However, 29% of internal medicine - preliminary applicants who completed the
supplemental ERAS application did not send any signals.

e Most program signals (87%-89%) overlapped with at least one of the geographic division
preferences reported in the supplemental ERAS application. Most applicants (69%-79%) sent
preference signals to programs located in the same geographic division as their permanent
address, and between 61%-69% sent preference signals to programs located in the same
division as their medical school.

e Preference signals were not equally distributed across programs. About a quarter of the programs
in each specialty received 46%-53% of all signals. This finding suggests that preference signals
may not have equal value to all programs, especially those that receive a large or small number
of signals from their applicants. More work is needed to better understand why some programs
receive so many (or so few) signals.

© 2022 Association of American Medical Colleges 5 aamc.org
ANA:

AANA
ava MI LU
AAMC Supplemental ERAS® Application:

Key Findings From the 2022 Application Cycle

Electronic Residency
Application Service®

e More than 70% of respondents reported that preference signals reflected their true preferences at
the time of their application, and more than half believed preference signals may help applicants
be noticed by programs they have the most interest in.

e The top factors applicants considered when signaling were:

o Alignment of program strengths with career interests (67%).

oc Location of program/geographic preference (66%).

o Strength of the program's clinical training (54%).

e Program directors used and saw value in preference signals. Across specialties, most program
directors reported using preference signals as part of the holistic process to invite applicants to
interview (95%-96%) and as a tiebreaker for deciding whom to invite to interview (84%-94%).
Furthermore, 75% of program directors reported that preference signals helped them identify
applicants they would otherwise have overlooked.

e Based on these data, the supplemental ERAS application for the 2023 cycle will include the
preference signal section. As shown in Table 3, applicants will still be able to indicate their top
preferences for programs in each specialty they apply to.

Table 3. The Preference Signal Section

 

 

 

 

 

 

 

Element Stays the Same Potential Changes
Content Send signals to programs in each Explore allowing applicants to
specialty applicants apply to. signal programs affiliated with
their medical school and where
Continue to work with each they completed an in-person
specialty to determine the optimal away rotation or clinical sub-
number of signals for each internship.
specialty.
Instruct applicants to send signals
to a select number of programs
they are genuinely interested in.
: . . Update guidance to reflect any
Guidance to Applicant guidance. :
Applicants and changes to question content.
Advisors Applicant and advisor guidance
webinars.
: ar . Update guidance to reflect any
Guidance to Supplemental application guide :
Program Directors for programs. changes to question content.
Program director guidance
webinars.

 

 

© 2022 Association of American Medical Colleges 6

aamc.org
ANA:

ava MI LU
® Co
AAMC Supplemental ERAS' Application: Elactronis Resldancy
Key Findings From the 2022 Application Cycle Application Service®

Applicant and Advisor Experience

Over 80% of applicants used the AAMC Supplemental ERAS Application Guide and website, and
the majority found those resources useful. About 50% reported talking to their medical school
advisor or a program director or faculty mentor or engaging in social media discussions about the
supplemental application, and over 70% of applicants found those conversations useful.

Just over 70% of applicants prepared their responses in advance, and of those, most spent more
than three hours preparing their responses. In addition, about 80% reported spending more than
an hour filling in the operational sections of the supplemental application. Additional work should
be done to streamline questions, improve preparation resources, and simplify the process for
adding responses to questions so completing the supplemental application takes less time.

The majority (84%) of advisors used AAMC resources, and almost 90% found these resources
useful. Most advisors (75%) spent up to three hours advising their students about the
supplemental ERAS application. Only about a third of advisors felt confident advising their
applicants about the geographic preference and preference-signaling questions. The top three
questions from students about the application were about:

o The strategy for sending preference signals (66%).

co How the supplemental ERAS application will be used (64%).

o The strategy for responding to geographic preference questions (54%).

Future information and training sessions should be disseminated and/or conducted through
AAMC webinars and the Group on Student Affairs (GSA) listserv, which advisors reported as the
most effective way to share information.

The supplemental ERAS application for the 2023 cycle will continue to be administered outside
the myERAS application because question content is still being refined. As shown in Table 4, the
AAMC will make changes that will improve the user experience for applicants, such as making
information available earlier, keeping the supplemental application open longer, and simplifying
some policies.

© 2022 Association of American Medical Colleges 7 aamc.org
AAMC Supplemental ERAS® Application:
Key Findings From the 2022 Application Cycle

Electronic Residency
Application Service®

Table 4, Administration for the 2023 Cycle

 

Element

Stays the Same

Potential Changes

 

Preparation

Questions made available in
advance.

List participating specialties
on the supplemental ERAS
application in March.

Make participating programs
available to applicants in June;
close program registration
May 31 (tentative).

Post preparation materials,
including final questions, in
April-May.

Explore providing a fillable PDF
worksheet to help applicants
prepare their responses in
advance and to serve as a
copy of their responses.

Explore providing information,
including a visual, to applicants
about how responses will be
displayed to programs.

 

Invitation

Separate from myERAS.

Invite via email after saving at
least one program in a
participating specialty.

Open earlier in summer.

May send email invitations on
a different cadence.

 

Completion and
Submission

 

 

Participation is optional, and all
questions are optional.

Applicant cannot change
responses after the submission
deadline.

Applicant cannot print a copy of
their responses.

The AAMC will not provide a copy
of an applicant's responses upon
request.

 

Explore simplifying how
applicants input responses.

Send the confirmation email
immediately upon submission
of the application.

Have one close date: Sept. 16,
5 p.m. ET (tentative).

Administer the research-only work
preferences section separately
and after the supplemental
application close date.

 

 

© 2022 Association of American Medical Colleges 8

aamc.org
AAMC Supplemental ERAS® Application:

Electronic Residency

Key Findings From the 2022 Application Cycle Application Service®

Program Director Experience

Program directors accessed applicant responses using a stand-alone dashboard, not the
Program Directors Workstation (PDWS). There were three common difficulties in using the
dashboard:

o Working in two systems (dashboard and the PDWS; 81%-93%).

o Using and downloading the raw data (32%-67%).

o Navigating the dashboard generally (39%-54%).

Despite these issues, over two-thirds of program directors (64%-88%) indicated they will likely
use the supplemental ERAS application for the 2023 application cycle. However, they requested
several "must have" content or features, including:

o Preference signals (62-71%).

co Ability to search for applicants by last name (56%-94%).

co Ability to create and save filters (46%-61%).

More than two-thirds of program directors indicated (1) it was important to release supplemental
ERAS application data on the same day as the PDWS and (2) a delay in receiving the
supplemental application responses would have a significant impact on their process.

Based on this feedback, data from the supplemental ERAS application for the 2023 cycle will be
available through the PDWS to program directors at participating programs and will be available
when the PDWS opens. As shown in Table 5, AAMC will make changes that will improve the
functionality of supplemental application data during the application review process.

© 2022 Association of American Medical Colleges 9 aamc.org
v

re A N A f
AA IV} J

Electronic Residency
Application Service®

AAMC Supplemental ERAS® Application:
Key Findings From the 2022 Application Cycle

Table 5. Program Director Access for the 2023 Cycle

 

Stays the Same

Potential Changes

 

Sign Up and Use

Specialty organizations agree to
participate.

Programs opt in by agreeing to
the supplemental application
terms and conditions.

Programs cannot drop out after
they sign up.

 

Application

Data available when the PDWS
opens.

Visual "Candidate Snapshot"
available.

Data will be available in the

PDWS:

o All registered PDWS users
can access the data.

o Functionality of supplemental
application data will be the
same as other application
data in the PDWS.

Data for applicants who complete
the supplemental application and
apply to a participating program
after the PDWS opening date

will be available as soon as
applicants apply to that program.

 

Guidance and
Training

 

 

Self-guided modules.

Supplemental application guide
for programs.

Live Q&A sessions.

 

Guidance will be updated to
reflect availably of applicant
data in the PDWS.

 

0400060

21-008H_02 (02/22)

© 2022 Association of American Medical Colleges 10

aamc.org