August 3, 1971
RMPS REVIEW CRITERIA AND RATING SYSTEM

INTRODUCTION

Several important factors have contributed to the need for
and development of an RMPS rating system. Foremost among these
is the recognition that the RMPS program is a mature, complex
national activity whose processes are deserving of and accorded
scrutiny by the public at large, Congress, and others. Additional
factors include the need to assess the degree to which Regional
Programs’ strength and activities are consonant with evolving
national priorities as reflected in the Council-endorsed Mission
Statement and the growing gap in recent years between the grant

funds actually made available to the program and the dollar level

of Council approvals.

Irrespective of fluctuations in the levels of funds available,
it is important that the Review Committee and Council continue to
base their assessments and recommendations on the overall merit of
individual programs and to leave to the Director, RMPS, the
responsibility for implementing the judgments of the Council. To
assist him in discharging this responsibility, and with the

encouragement of HSMHA, a rating system has been designed and tested.

Subsequent to the May Council meeting, an RMPS staff committee
-was formed to develop and apply on a trial basis a rating system

for the applications currently under review. The Committee's
2
approach was to develop a further elaboration of the 17 criteria °
included in the RMPS Mission Statement, a weighting scale for the
criteria, a 1-5 point evaluation scheme, and associated forms and~
instructions for use by the RMPS Review Committee in applying the
criteria, The materials developed were reviewed by the Administrator,

HSMHA.

The purpose of this report is to summarize the reaction of the
Review Committee to their initial experience in using the criteria

and rating system, and*to present an analysis of the results.

THE REVIEW COMMITTEE'S REACTIONS

The Review Committee met in executive session after completing
the review of 13 triennial applications in the current cycle for the
’ purpose of providing feedback on the ‘rating system. In general,
Committee members were quite favorable in their reaction to the
criteria themselves. They apparently felt that these .comprehensively
covered the relevant and salient points that need to be taken into
account in assessing and ranking RMP's.. They also appeared to feel
that the sub-criteria that had been developed in the form of questions

were useful in clarifying the broader criteria arid helping them to

score these.
The principal drawback in applying the criteria was lack of

specific information relating to some of the individual items. ‘A

number of suggestions relating to the arrangement of the material

also were received.
1.

ANALYSIS OF RESULTS

The criteria and ratings clearly discriminated between the
best, average, and poorest regions. |

The reviewers were tough graders. The top Region received

a score of only 327 out of a possible 500. If, as expected,

they become more lenient in the future, it will be necessary

to apply weighted means to insure comparability from one review
cycle to the next.

Reviewers were asked to circle ratings about which they were
uncertain due to insufficient evidence or for other reasons.

Three criteria, Continuity of Care, Prevention, and Ambulatory Care
were the most troublesome. There was virtually no uncertainty
about Goals and Objectives, Organizational Viability and

Effectiveness. In general, criteria relating to organization,

management and objectives appeared to be least troublesome.

FOLLOW-UP

Certain minor modifications have been made in the criteria, however,.

as the result of Committee suggestions and staff analyses. Some

interim steps also have been taken to provide additional information

in selected areas (e.g., continued support, evaluation, other funding)

so that the uncertainty of reviewers with respect to their criteria

may be reduced in the future.
CONCLUSION

The trial use of the review criteria and rating system by the
Review Committee, their reactions to it, and the analysis of the
results strongly suggest that it is an’ effective and|workable basis
for assessing the quality and performance of Regional Medical

Programs.
INSTRUCTIONS: Using a one through five scoring scale (S-outstanding,
4-good, 3-satisfactory, 2-Fair, 1-Poor) rate the Region
in accordance with the criteria set forth under:

I - Performance, II - Process, and III - Program.

Subcriteria or elements in the form of questions have been included
in order to make the broad, general criteria more specific and
understandable. These are designed to be of help to the reviewer
in assigning a score to each of the criteria. Multiplication of
scores by the assigned weights and the necessary addition will be
done by staff; reviewers necd not make those computations.

Reviewers should provide their overall subjective assessment of the
Region and its application by rating on a one to five basis in
Item IV, OVERALL ASSESSMENT .

Use a check (%) in Item V, DEVELOPMENTAL COMPONENT, if in your best
judgment this Region has achieved sufficient program maturity and
status to warrant award of « developmental component.

In Item VI, BASIS FOR EVALUATION, indicate for each Region the basis
for your evaluation. When appropriate more than one item in ITEM VI

may be checked for each Region.
Mera Wee Ve ROVIGWCN

 

ADUIINISTRATIVELY , . (OPTIONAL)
COMFIDETTIAL RATINGS: ;

5-OUTSTANDING
L-GooD 2-FAIR
%-SATISFACTORY -PooR

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

= tw alol=z =z] 2/2l2lafsat=
ziti Q\2)alF] Sel Slam] S]
4] 2S) 3) 8) 2] 4 =] ol fa a 3
mn | 2} | > °) Ss; Sid 2
CRITERIA = fi : Q| GB} 2 2
< z
R
Date of Most Recent Site Visit
ob se GF TH/O 1771. 5/71 6 71112/7010/0016/71 6/71. 9/7 10 AS16/ 716/711 12/70
1, PERFORANCE:
‘1. Goals, Objectives & Priorities 7 , .
: *
2. Accomplishments § Implementation 16
3. Continued Support 12
» TL. PROCESS
1. Org. Viability & Effectiveness 12
2. Participation 5
3. Local Planning 4 “
4. Assessment of Needs §& Resources 4
5. Management § Evaluation - 5
IIT, PROSRAM PROPOSAL
1. Action Plan 5
2. Dissemination of Knowledge 2
3. Util: Manpower § Facilities 5
4, Prevention 7)
5S. Ambulatory Care - 2
6. Continuity of Care 2
7. Short-Term Payoff 5
8. Regionalization | 4
9. Other Funding 3
TV. OVEPALL ASSESSMAT
: Ie mnt (if yes,
V, DEVELOPHENTAL COMPONENT check“)
. hn 5 I il
VI. BASIS FOR EVALUATION “SBS Kcfbte
1. Current Site Visit
2. Previous Site Visit
3. Application
4. Committee Discussion
5. Other ‘
6. Primary Reviewer

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 
FACTOPS TO RE OONSIDEPED WITH THE REVIEW CRITERIA

1, PERFORMANCE
1. QQALS, OBJECTIVES, & PRIORITIES

a.

b.

Have these been developed §

explicitly stated?

Are they understood and accepted by
the health providers 4 institutions

of the Region?

Where appropriate, were community and
consumer groups also consulted in their
formulation?

Have they generally been followed in
the funding «* operational activities?.
Do they reflect short-term, specific
objectives and priorities as well as
long-range goals?

Do they reflect regional needs and
problems and realistically take into
account available resources?

2. ACCOMPLISHMENTS AND_IMPLEMENTATION

a.

Have core activities resulted in substantive
program accomplishments and stimulated
worthwhile activities?

Have successful RMP activities been replicated
and extended throughout the region? ,
Have any original and unique ideas ,. programs
or techniques been generated? >

Have activities led to a wider application of
new knowledge and techniques?

Have they had any demonstrable effect on
moderating costs?

Have they resulted in ay material increase in ,

the availability and accessibility of care
through better utilization of manpower and
the like?

Have they significantly improved the quality
of care?

Are unsuccessful or irrelevant. activities being
phased out? |

Are other health.groups aware of and using the
data, expertise, etc. available through RMP?

Do physicians and other provider groups and institu-
tions look to RMP for technical and professional

assistance, consultation and information?

If SO; does or will such assistance be concemed with
quality of care standards, peer review mechanisms,
and the like?

Is there a policy, actively pursued, aimed at
developing other sources of funding for successful
RMP activities? .

Have successful activities in fact been continued
within the regular health care financing system
after the withdrawal of RMP support?

DRAFT 6/29/71 - ADMINISTRATIVE USE ONLY

 
11, PROCESS

i.

FACTOPS TO BE OOMSIDERED WITH THE REVIEW CRITERIA

3.

ORGANIZATIONAL VIABILITY AND EFFECTIVENESS

a.

Is the coordinator effective; has he provided strong
leadership, developed program direction and cohesion
and established an effectively functioning core staff
with due regard to equal employment opportunities and °
minority group interest.
Does core staff reflect a broad range of professional
and discipline conpetence and possess adequate
a@iuinistrative and management capability?
Are most core staff essentially full-time?
Is there an adequate central core staff (as

ed to institutional components)?
Does the grantee organization (1) provide adequate
administrative and other support to the RMP and
(2) permit it sufficient freedom and flexibility,
especially insofar as the RAG's policy-making role
is concerned? ° S.
Are all key health interests, institutions, and .
groups within the region adequately represented
on the RAG (and corollary plaming committee
structure)?
Does the RAG meet as a whole at least 3 or 4 times
annually? .
Are meetings well attended?
Are minority groups and. consumers adequately
represented on the RAG and corollary committee
structure, and do they actively participate in
the deliberations?
Is the RAG playing an active role in setting program
policies and establishing objectives and priorities?
Does the RAG have an executive or steering conmittee
to provide more frequent administrative program guidance
to the coordinator?
Is that committee also fairly representative?

PARTICIPATION

a.
b.

c.

Are the key health interests, institutions, and
groups actively participating in the program?
Does it appear to have been captured or: co-opted
by a major interest?

Is the region's political and econamic power
cemplex involved?

 

LOCAL PLANNING

a. Has RMP help develop in conjunction with GiP effective
local planning groups?

Is there early involvement of these local planning groups

in the development of program proposals.

c. Are there adequate mechanisms for obtaining substantive
CHP review. and comment?

ASSESSMENT OF NEEDS AND RESOURCES

a. Is there a systematic, continuing identification of needs,
problems, and resources?

-b. Does this involve an assessment and analysis based on data?

Are identified needs and problems being translated into the
region's evolving plans and priorities?

d. Are they also reflected in the scope and nature of its
emerging core and operational activities?

MANAGEMENT AND EVALUATION .

Is there regular, systematic and adequate monitoring of

projects, contracts, and other. activities by specifically
assigned core staff?

b. Are periodic progress and financial reports required?

Is there a full-time evaluation director and staff?

a.

c.
d. Does evaluation consist of more than mere progress reporting?
e.. Is there feedback on progress and evaluation results to

“program management, RAG, and other appropriate groups?

f.° Have negative or unsatisfactory results been converted into
program decisions and modifications; specifically have
imsuccessful or ineffective activities been promptly phased —
out?

DRAFT 6/29/71 - ADMINISTRATIVE USE ONLY

 
FACTORS 70 BE CONSIDERED WITH THC PEVIEW CRITERIA

11]. PROGRAY PROPOSAL

1.

2.

ACTION PLAN 5.

a. Have priorities been established?

b. Are they congruent with national goals and objectives?

c. Do the activities proposed by the region relate to its

stated priorities, objectives and needs?

Are the plan and the proposed activities realistic in

view of resources available § Region's past performance?

e. Can the intended results be quantified to any .
significant degree?

£. Have methods for reporting accomplishments and
assessing results been proposed? .

g. Are priorities periodically reviewed and updated?

DISSEMINATION OF KNOWLEDGE

a. Have ‘provider groups or institutions that will

benefit been targeted?

Have the knowledge, skills, and techniques to be
disseminated been identified; are they. ready for
widespread implementation? .

Are the health education and research institutions of
the Region actively invclved? 7,
Is better care to more people likely to result?

Are they likely to moderate the costs of care?

Are they directed to widely applicable and currently
practical techniques rather than care of rare *
conditions or highly specialized, low volume services?

d.

b.

9

ho

UTILIZATION MANPOWER AND FACILITIES

Have areas or populations been specifically targeted?
Will presently underserved areas or populations
benefit significantly as a result? 8.
Will existing community health facilities be more
fully or effectively utilized? ,
ad. Is it likely productivity of physicians and other
health manpower will be increased? .
e. Is utilization of allied health personnel, either new
kinds or combinations of existing kinds, anticipated?

a.
b.

c.

f. Is this an identified priority area; if so, is it
proportionately reflected in this aspect of their
overall program?

PREVENTION

a, Have specific diseases, areas, or populations been
targeted?

b. Are health maintenance and disease prevention components 9,
included in current or proposed activities?

c. If so, are they realistic in view of present knowledge,
state-of-the-art, and other factors?

d. Is early detection included?

e. If so, has adequate provision been made for follow-

through treatment?
f. Is health related education of the public or patients
included, where appropriate?

a DRAFT 6/29/71 - ADMINISTRATIVE USE ONLY

AMPULATORY CARE

a.

Have RMP or other studies (1) indicated the extent to
which ambulatory care might be expanded or (2) identified
problem areas (e.g., geographic, institutional) in this
regard? |

Will current or proposed activities expand it?

Are commmications, transportation services and the like

. being exploited so that diagnosis and treatment on an

out-patient basis is possible?

CONTINUITY OF CARE

 

 

a. Have problems of access to care and continuity of care
. been identified by*RMP or others?

b. Will-current or proposed activities strengthen primary
care and relationships between specialized and primary .
care?

c. Will they lead to improved access to primary care and
‘health services for persons residing in areas presently
underserved?

a. Is it reasonable to expect that the operational activities
proposed will increase the availability of and access to
services, enhance the quality of care and/or moderate its
costs, within the next 2-3 years?

b. Is the feedback needed to document actual or prospective

. pay-offs provided? ,

c. Is it reasonable to expect that RMP support can be withdrawn
successfully within 3 years?

REGIONAL IZATION

a. Are the plan and activities proposed aimed at assisting
multiple previder groups and institutions (as opposed to
groups or institutions singly)?

b. Is greater sharing of facilities, manpower and other
resources envisaged?

c.. Will existing resources and services that are especially
scarce and/or expensive, be extended and made available
to a larger area and population than presently?

d. Will new linkages be established (or existing ones strengthened)
among health providers and institutions?

e. Is the concept of progressive patient care (e.g., OP clinics,
hospitals, ECF's, home health services), reflected?

F. NG
a. Is there evidence the region has or will attract funds other than RMP?
b. If not, has it attempted to do so? .
c. Will other funds (private, local, state, other Federal) be
available for the activities proposed? ;
d. Conversely, will the activities contribute financially or otherwise

to other significant Federally-funded or locally-supported health

programs? °