evaluation workshop ‘72 lakes area regional medical program inc. faAIUALIVUED worksho November 2, 1972 Treadway Inn, ~. Niagara Falls, New York Sponsored by Lakes Area Regional Medical Program, Inc. Title The Process of Program Evaluation Purpose To acquaint those persons associated with the Lakes.Area Regional Medical Program with the evaluation mechanism, its complexity, and the need for overt commitment to its implementation. The Lakes Area Regional Medical Program gratefully acknowledges the support of the Division of Regional Medical Programs Service, Health Services and Mental Health Administration, U.S. Department of Health, Education and Welfare. The findings and conclusions in this publication do not necessarily represent the views of the sponsoring agencies. | contents. Foreword by John R. F. Ingall, M.D. Welcome by Irwin Felsen, M.D. The Agenda Executive Director’s Address 9 Highlights of Secretary Richardson’s Remarks 11 Keynote Address 14 Model of the Process of Program Evaluation 16 Discussion Groups 17. Interaction 19 Reaction “N or fh W 21. Participants foreword The Workshop vividly portrayed evaluation as an integral process of all Lakes Area Regional Medical Program activities. It provided participants with a keen understanding of the evaluation process, its methods and procedures, and its overwhelming contribution toward the improvement of health care delivery systems. In addition to assessing program and project effectiveness, participants were shown how the evaluation proéess provides opportunity for learning, how it identifies strengths, and how it pinpoints weaknesses — all of which serves to improve program and project operations. The information contained on the following pages expresses the pride of Lakes Area Regional Medical Program staff in conducting this Workshop as well as the sincere interest shown by its many participants. . ¥ John R. F. Ingall, M.D. Executive Director... or ae welcome Irwin: Felsen, 7 D., President, Lakes Area Regional Medical Program - Regional Advisory Group, | Inc. _“t would like to welcome you to our Second Annual Regional Medical Program Workshop. The topic of _ this conference is ‘The Process of Program Evaluation.’ At our Regional Medical Program in Western New York ‘and Northwestern Pennsylvania, we are pledged to plan and to coordinate programs to strengthen the health care system capable of responding to the needs of all our people. That is our major objective. At last year’s workshop we established our goals and objectives. During the year we formed the Program Committee to serve as a long range planning group. This committee is responsible for today’s program. _ | would like to give special thanks to Patricia Hoff, Joseph Reynolds, Tony Zerbo and Dr. James McCormack ‘for helping in the mechanics necessary for the conference. We make policy and when we make policy we are betting on the future. The great danger of any policy is to project the present into the future. Today we seek some objective yardsticks for measuring our Regional Medical Program’s actual impact on society. We are interested mainly in estimating the program's ultimate health benefits. We are 3 exploring ways to assess the health impact of our projects, and to develop a meaningful method of evaluation. We seek ‘workable’ machinery. Can we do anything about it? | will confess that | am an optimist; one who draws confidence from facts as well as from hopes. | am sure we will find means of dealing with the problem, and be able to narrow the gap between promise and performance. We seek participation from all of you present. We are not too proud, too rigid, too blind, or too complacent to change. | am sure that we are all looking forward to a valuable learning experience. the agenda Thursday, November 2, 1972 9:00-10:00 Registration - Ballroom coffee 10:00-10:10 Welcome © Irwin Felsen, M.D. President, Lakes Area Regional Medical Program Regional Advisory Group, Inc. 10:10-10:30 |. “Program Evaluation and the Mission of Lakes Area Regional Medical Program” John R. F. Ingall, M.D. “—< Executive Director Lakes Area Regional Medical Program ~, 10:30-12:00 “A Design for Program Evaluation” John C. Cassel, B.Sc., M.P.H., M.D., B., Ch., Int. Med. Department of Epidemiology University of North Carolina Chapel! Hill,.North Carolina 12:00-1:30 Luncheon Thursday Afternoon 1:30-2:35 Introduction of HEW Secretary Elliot L. Richardson John R..F. Ingall, M.D. Elliot L. Richardson Secretary , United States Department of Health, Education and Welfare , Washington, D. Cc. ~ Buffalo, New York - Robert O'Shea, Ph. D. Associate Professor ~ tae! Department of Sociology fe State. University of New York at Buff : John M. ‘Glasgow, Ph. D. - Assistant F Professor: ;. University of Connecticut - Health Cent 2:40-3:15 . Small Group Discussions, Niagara Room, Ballroom Discussions will be based upon the objectives and activities proposed in local funded grant applications. A skillful leader will be assigned to direct the discussion of the group around Doctor Cassel’s presentation of his evaluation model. The participants will use Doctor Cassel’s model to work toward the creation of an evaluation approach for the proposal. - Participants will be given an opportunity - to become directly involved in'the process © . of program evaluation. * os 3:15-4:15 Group Reports 4:15-5:15 Panel Reactors Moderator J. Warren Perry, Ph.D. Dean, School of Health Related Professions State University of New York at Buffalo. Panel Harry A. Sultz, D.D.S., M.P.H. Professor, Department of Social and Preventive Medicine State University of New York at Buffalo Buffalo, New York Elsa R. Kellberg, M.A. Assistant Director for Evaluation ' Lakes Area Regional Medical Program * Buffalo, New York .::-, a Hartford, Connecticut - executive director's address John R. F. Ingall, M.D., Executive Director, Lakes Area Regional Medical Program Program Evaluation and the Mission of Lakes Area Regional Medical Program “The Lakes Area Regional Medical Program is responsive to a large constituency. We have the capacity to catalyze efforts towards their stated goals and objectives, which in time become ours. In serving this body of people we want to enable them to attack the deficits in our nine county health market area. To accomplish this, we need to measure respectively what we are doing and what we have achieved. It is an obligation for those in receipt of our funds to appreciate what evaluation means. It is important for programs to clearly under- stand where they are; where they are going; and how they are going to get there. It is important for project directors to establish what measures they can apply, in the time frame they have set for themselves, to portray ° achievement — particularly the ability to become self-supporting. 7 “ We have developed and will continue to construct an evaluation compo: nent into everything we do. Evaluation is essential for diagnosis; crucial to wise clinical judgements; and essential in measuring outcome. This process is without punitive overtones; it is a mechanism which will allow a diagnosis of performance and impact. We can, and must, refine this mechanism in order to 3 produce an effective effort toward correcting well-stated — gs deficits in the health market area to which we are ~ responsible. The Regional Medical Programs of this country have, by industrious involvement with, the grassroots community, made things happen. People of various disciplines are meeting for the first time. An informed, representive constituency of responsible individuals are sensitive to community deficits in the’ health services. The involvement of this informed community must be augmented. The fact that it is being developed, that it is underway and has momentum, is a major achieve- ment and must be recognized. Involving intelligent people in the validation . and interpretation of reliable data leads to intelligent decisions. These decisions are the first steps in regionalization which is implied in our title. Regional Medical Programs are ‘enabling’. agencies. They recognize — the industry required in bringing groups together and convening them for — specific purposes with specific challenges. Their unique role in maintaining | momentum by judicious staff support, enhances the ability of established | “agencies to deliver and to improve upon previous performance.” They have facilitated many agencies now carrying out their mandated functions. They have the ability to meet challenges whether they be in the professional. arena or in the consumer population. ee er A recurring theme is the ability to deliver and to measure the amount of goods to the maximum amount of people.. The funds to this purpose are’ tax dollars as venture capital which are invested in economically viable projects. We have sought to involve all health professionals, particularly he private and voluntary groups. -- Regional Medical Programs had dramatic visibility in developing categorical “disease components, such as the coronary care training program for nurses and physicians. The categorical killers are not in any way negated by our - broader approach to the health problems. The first of our goals is prevention. - It is an area where the biggest impact on human morbidity can be anticipated. “In stimulating efforts towards prevention, the factor of dramatic visibility, ‘which in turn releases tears and money, is absent. A crippled child will release more funds by being exhibited, than a normal child, who is so by virtue of preventive methods which are hard to render glamourous. — ~. Education of the consumer is important. The question of its feasibility, “the economy of doing so, and the method of evaluating these efforts is a challenge to the Regional Medical Programs for the future. We must try to teach the difficult problems which will have impact on the categorical _diseases. A problem well stated is half solved. It seems to me we have problems. !t seems to me these can be given dimensions. In doing so, we can remeasure these programs. That we do so -in an objective, clear, and responsible fashion is an essential ingredient in - our evaluation. | anticipate today will be a learning experience and an exercise in working towards a lucid understanding of measures of achieve- mentsand their validity.” it Highlights of Secretary Richardson's remarks We perceive very clearly that we cannot effectively improve the quality of health care delivery in the United States; we cannot improve access to good care; we cannot overcome the gaps and deficiencies in our delivery system by Washington decree. We cannot { build from the top down without displacing in the process, not only the contribution of local resources, but the contribution of | local concern. If we are genuinely interested in bringing about the kind of coalition of resources at the local level that is involved in putting the existing pieces together. with optimal efficiency, we have to rely on people like yourselves to carry out that process. There is just no way in which federal bureaucrats can know enough about the capabilities of existing components of the health delivery gystem in any community to know how to fit them together in a more coherent whole. The initiative for this and the exercise of carrying it out must be in local community and regional hands. Awareness of need A very valuable quality of American life is the devotion of community concern to meeting the problems of people. How- ever, local response to problems can lead to fragmentation because so many well meaning citizens may be motivated to create community agencies or organizations that often do not communicate sufficiently with each other or do not participate in joint efforts to create a more total effort. But it is that kind of response, that kind of ‘awareness of. need’ within a community, that is part of what we are as a people. The very idea that you can rely on the government to solve these prob- lems and to meet these needs can only have the consequence of drying up the springs of local concern and compassion, There are two values at stake here. One is = the capacity of local service to provide > agencies that can fit their efforts together. The second is the preservation of community L/ from the top down through a structure that would 10 : necessarily displace state and local initiatives. The planning capa- ‘bility at the Federal level is so enormous, that, in order to get anything done at all, the solutions prescribed would have to be uniform, and they would have to displace local responsiveness and local concern. Initiatives Applauded We look to the kind of effort that you represent and the kinds of things that are being done in response to the problems of sparsely populated rural communities in this nine county area. These include your efforts to make more accessible improved care for coronary and respiratory patients; to promote regional linkage among health care providers through your Telephone Lecture Network which enables them to keep their skills up to date; and to develop a data base that can contribute to the planning capabilities that are required in order to fit the pieces together. These are all the kinds of initiatives that we in HEW applaud and seek to encourage. Develop Planning Capabilities In my view, if we are to achieve the development of an adequate network of services through building from the community level up, rather than imposing solutions from the top down, we can only do it through substantial improvement in our planning capacity. Under the leadership of Dr. Merlin DuVal, Assistant Secretary for Health Affairs, through whose general aegis the Regional Medical Program comes, we in HEW are now reviewing all of the present legislative authorities that would impinge in any way upon the planning process. We are seeking to strengthen the capacity as well as the authority and responsibility of planners at the local and regional levels. There is a limited amount of time and unless we move more quickly now to develop our planning capabilities, we may get left behind or be overwhelmed by the pressures which insist that you just are not going to get there from here, that you have got to let Washington do it. That, to me, would be profoundly wrong. soN keynote address Highlights ~ of Dr. John Cassel’s Speech While there is a great deal of activity in the field of evaluation, most attempts have ended in rather dismal failures, producing very little in the way of data or results; and those that have produced results have usually led more to anger than to action. There seems to be three crucial reasons why current attempts at evaluation have been less than successful. The first is a confusion about the purposes of evaluation. Evaluation is often confused with inspection, whereby it is seen as a method of forcing program people to account for their actions. Results of such accounting would be a change in budget, elimination of jobs, and downgrading or upgrading personnel. Such an unfortunate set of purposes leads to a stand-off in the personnel involved in the program and those involved in evaluation, with one trying to defend their position and the other trying to attack it, There has to be some method of public accountability for the use of public funds, but | choose for the purposes of clarity not to call that evaluation. It would be more precise to consider evaluation as a type of diagnosis. It is an integral part of practice, concerned with providing such information for program personnel as how to change, improve, or modify their actions. If we regard evaluation as a diagnostic probe rather than an inspection, we would be ina lot better shape. {n addition to confusion as to purposes, there is confusion about the types of information needed for evaluation. Many evaluation programs restrict themselves to gathering data about activities. How many tons of literature are disseminated? How many blood counts did you do? How many patients did you see? | agree, there is a place for counting activities, but it produces no indication whatsoever of program outcomes, and program outcomes should be the heart and soul of any evaluative exercise. ee ON . Conga “ wt The third major deficit concerns the methods used to accomplish evaluation. It is too often thought that if you want to have an evaluation, you must go out, hire an evaluator and say to him or her, ‘Evaluate please, and give mea report.” This is obviously an oversimplification, but | think you recognize how pervasive and insidious it is. There isa place for professional evaluators; but is a very specific place and it is not something one can turn over to an ‘outside’ agency or individual. 12 Keynote address ‘The argument is made that self-evaluation is very difficult because of personal bias. | “counter that by saying that the most -- important principle of any evaluation is that > there be an on-going partnership between those people organizing and providing the service, and those people with the technical skills to conduct evaluations. If evaluation is to be successful at all, there not only has to be an acceptance that evaluation is desirable and necessary, but a commitment on the " part of those who have the responsibility for providing the service to be involved in this evaluation process from the beginning through to the end, in partnership. Too frequently, people state their goals in “for-mother and against-sin’’ terms, such as “This program is supposed to improve the health of all the people’. This is relatively useless, because the questions that follow must be: “‘What aspect of health, what evidence would you accept that this program will make any difference in anyone’s health?”’ Such vague, global statements are desirable, but unpractical or unrealistic in terms of the action taken, Another error often made in defining goal statements is to state them in terms of ‘services provided’. These are not goals. They are the means to accomplish something. What they aim to accomplish and what evidence they can produce to measure that accomplishment are the direct concerns of the program and its evaluation. It is necessary for a dialogue to take place between all those people who are going to contribute to the evaluation, in order to specify what could be the feasible and acceptable goals of such an activity. Having specified the goals, what information would you accept as evidence that you have arrived there? Having stated the goals, unless there is clear understanding about the nature of the evidence that would be acceptable, conducting an evaluation exercise that produces unacceptable evidence is not going to lead to any change. It takes time. There is no instant solution. . It takes a lot of dialogue because people have been unaccustomed to think in these terms. And it takes professional judgement. A very difficult and important question follows: “Given the objectives, and given that a certain set of information would be acceptable evidence, what needs to be changed in order to accomplish these objectives?” What should be done that is not being done now; what sorts of procedures and techniques are available; and what will be used in order to accomplish these changes and lead to the desirable program goals? Evaluations will tell us how accurate and useful our original formulations have been and enable us to reformulate them in a second go around, or a third if need be. This becomes a feedback situation. It is the evaluator’s responsibility to pick up the dialogue, to take the ideas that have been developed by the program people and to translate them into measurable quantitative data. This step of translating information into data requires the skill of a professional evaluator. There are three general types of data that are needed in evaluation. First, data must show that the program is being successful; that there has been a change in health status or change in health behavior. The question is, ‘What sort of data do we need that will measure sick people in need of care?” Objective and subjective data have to be presented back to the people responsible for the program, who must then determine what would be appropriate sorts of information labeling people as sick and in need of care. Homogenous groups of patients should be specified so that it would be possible to develop a basis for describing the problem that has to be changed in order to affect the health status or health behavior of such people. The second type of data needed pertains to the changes that have been postulated as necessary. {f you talk about preventing heart disease by changing diet, you not only must know how to measure heart disease, but how to measure the diet. You should have some information about the occurrence of the phenomena that are going to be changed and whether one can see if they have changed. The third type of data concerns the services or techniques that have been used. It does become necessary in this whole picture to measure whether the activities have been carried out, how frequently etc., in order to find out if they have been successful in bringing about the change. There are three general types of strategy that can be used in gathering and analyzing data: to compare the situation before and after the program has been introduced; to compare one program with another or with the absence of another program; and to engage in some sort of randomized trial. Common to all three approaches is that similar types of subjects are to be compared. All factors likely to affect the outcome should be similar as far as possible. The before and after comparisons involve finding out what the situation is, introducing the program or service, and measuring the same phenomenon at the second point in time. The advantage of this approach is that it is simple and usually feasible. If there is no difference between the before and after comparison, the program has been unsuccessful. If you do find a difference, however, it is not certain that it is due to the program or whether there were other factors not measured. A control or situation would make this method more rigorous. Another alternative is to collect as much data as possible to provide the program with influential factors. Or there is the option to replicate, to repeat the program in other surrounding regions | thas avhihit eimiler noade Ay The second strategy for program evaluation is a comparison of two or more different programs operating under the same needs or toward the same goals. If the programs are operating in the same or adjacent localities, it is reasonable to compare just the after data on the assumption that the sort of people and conditions being treated are relatively similar. The third method is the randomized control trial. If there is the option of two programs or treatments to be introduced and evaluated, eligible patients can be identified and randomly assigned to program A and program B. The hope is that by random assignment the other factors that influence outcome are equalized. The outcome of the treatment should remain unknown to the provider and the patient, in order to avoid bias. All of this sounds awfully formidable and = 13 long-term. It certainly has its complex problems. | feel, however, that there would | be a lot more progress made in the area of evaluation if, in addition to this formalized _ approach that involves a lot of dialogue and a a lot of cooperation, we made it desirable, possible, and expected for every professional : person working within a program to ask himself, “What would | accept as evidence that | am doing my job well?” We can make | un . it less formal, more personalized, and more =... involved, We can involve ourselves so that we,’ as professional people providing services, can : begin to get some sort of feedback as to how . a effective our actions are and where we need - to modify these activities. Again, evaluation is an integral part of practice, not an extraneous © ; set of activities for inspection. It can lead to improvement in our knowledge and in our. ability to deliver services. We can learn from our failures as well as our successes, and we can begin to make knowledge cumulative If we can do this, then | think we have a. fighting chance of getting something . useful done. Keynote address the process of program evaluation-a model by Dr. John Cassel he Statement of objectives - in operational terms. \ Change in techniques, procedures or services necessary to accomplish the objective. 2 Selection of characteristics or variables measures of problem; measures of process; measures of outcome. Strategy of data collection; choice of method, design of study. . 3 1. Before and after study * 2. Program of comparison > 3. Controlled trial Interpretations 1. Have changes occurred 2. Have objectives been met 5 3. What are implications for service q | | | ! ‘ ’ evaluation workshop '72 16 discussion groups 5 “Group ve 1. Telephone Lecture Network . Comprehensive Continuing Care * for Chronic IIIness Pulmonary Rehabilitation . Pulmonary Home Care (The above two projects are a part of the Chronic Respiratory Disease Program) . Model Program for Comprehensive Family Health ‘ . Community Health Information Profile (CHIP) _ Model Stroke Rehabilitation Proposal for Regional Community Hospitals The following projects were discussed in separate group discussions Project Directors Joseph Reynolds, TLN Coordinator Evan Calkins, MD, Chairman, Department of Medicine, State University of New York at Buffalo; E. J. Meyer Memorial Hospital John Vance, MD, Clinical Associate Professor of Medicine, State University of New York at Buffalo; Millard Fillmore Hospital Ernest Haynes, MD, Director, Family Practice Center, Deaconess Hospital, Buffalo Harry Sultz, DDS, MPH, Professor, Department of Social and Preventive Medicine, State University of New York at Buffalo William Kinkel, MD, Clinical Associate, Professor Neurology, Anatomy, State University of New York at Buffalo; Millard Fillmore Hospital noon, interaction Seven groups met to evaluate individual LARMP projects, and found themselves QUESTIONING... We had better decide first what we are going to define as SUCCESS and if we do arrive at this success, PW WHAT ARE WE GOING TO DO ABOUT IT? HOW FAR DO YOU GO ina study? WHEN do you say it is SIGNIFICANT? What measures are VALID ina particular case that i could really measure CHANGE? We want to Know if this kind of program will help people to BETTER COPE with the problems of living in contemporary society? WHY DO THIS? oe e and questioning We question the SCOPE of evaluation, the BOUNDARIES of evaluation...WHERE DO YOU STOP? What caused the drop in USE of this program; Was it CONTENT or COST or both? . —— Do other PEOPLE in the community ever FIND OUT about these programs? How many times do you have to . WORK THROUGH these steps po : and then WORK BACK? Are the MEANS getting mixed . with the ENDS; are ACTION A physician must question, “Are there functions programs getting mixed in with physicians have traditionally performed that the OBJECTIVES? would better be performed by others?” Is ita VIABLE program or is it NOT? — Could we turn success into USEFUL ACTION; “and, if not, we QUESTION the whole worth of the whole evaluation. ‘ | ? ~ and concluding We are concerned with the TOTAL well-being of the patient, If you break not strictly a particular health problem. an objective down to a specific point, The most important thing is to determine whether there is ~ it starts a NEED for a program or an ADVANTAGE in having one. to MAKE SENSE. difficult to ACCESS TOTAL IMPACT There are parts of health care that cannot be measured. Evaluation is like salt: every meal IMPROVES with salt; some foods NEED MORE than others; but there can be TOO MUCH salt! Are the objectives WORKABLE? Periodically there should be an assessment of data. | Based upon the insights of the professionals, you could make sound decisions to BUILD ON. There are many VARIABLES to consider: TIMING; PROGRAM; HOW and WHEN PEOPLE ARE INVOLVED; WHO they are trying to reach...you must CLARIFY your thinking. Data collection can be a valid INDICATION of CHANGE. This is all based on the assumption that we BELIEVE it will WORK! en ena 8 >. a workable way to accomplish, that understanding, for both project oe and Program people. WHAT DID THE WORKSHOP ACCOMPLISH? What was indeed accomplished was an attitudinal change toward | ~~ evaluation. Those attending realized the importance for evaluation, * a! and demonstrated an increased appreciation of its relevance to program in goals and objectives. The Workshop produced rationales for ‘why’: evaluation is necessary in terms of planning and outcome. It forced basic and direct questions about the need, motive and direction * behind projects. It lead to the clarification of individual goals, demonstrating the need for relevant, workable, and specific objectives. ; In an attempt to evaluate particular projects of LARMP, participants were confronted evaluation problems. They found that as they shaped evaluation, they re-shaped the project. . Evaluation began as an idea and emerged as a functional process that required work, understanding, and unqualified commitment from every . facet of every program. oe participants Project Directors Virginia Barker, R.N., Ph.D. Director, Mobile Health Unit Alfred, New York Michael Carey, Director Lake Area Health Education Center Erie, Pennsylvania Roger Fenton Emergency Medical Services Erie County Health Department Buffalo, New York Phyllis Higley, Ph.D., Director Plan and Articulation for Allied Health Buffalo, New York Regional Advisory Group Members Irwin Felsen, M.D., President Lakes Area Regional Medical Program Wellsville, New York Father Cosmos Girard, OFM, Ph.D. Vice-President, . Lakes Area Regional Medical Program St. Bonaventure University Olean, New York Theodore Bronk, M.D., Secretary Lakes Area Regional Medical Program Mt. St. Mary’s Hospital Niagara Falls, New York John C. Patterson, M.D., Treasurer Lakes Area Regional Medical Program Director, Tumor Registry Buffalo, New York Virginia Barker, R.N., Ph.D. Alfred University Alfred, New York Buffalo, New York Jean Miller Information Dissemination Service State University of New York at Buffalo Buffalo, New York John C. Patterson, M.D., Director Tumor Registry Buffalo, New York Joseph Reynolds, Director Telephone Lecture Network Buffalo, New York Harry Sultz, D.D.S., M.P.H., Director Community Health Information Profile (CHIP) Buffalo, New York ‘ 21 LaVerne Campbell, M.D. New York State Health Department Buffalo, New York Herbert E. Joyce, M.D., Past President Lakes Area Regional Medical Program Buffalo, New York Bert S. Klein, D.P.M. Jamestown, New York J. Warren Perry, Ph.D., Dean School of Health Related Professions State University of New York at Buffalo Buffalo, New York Harry A. Sultz, D.D.S., Professor - School of Medicine fone State University of New York at Buffalo ee 22 Professional Staff Members John R. F. Ingall, M.D. “. Executive Director “© James McCormack, Ph.D. Associate Director for Planning “and Evaluation Robert J. Beebee “" Staff Associate for Niagara, Genesee, Allegany and Wyoming Counties Helen Brown .» Hospital Library Consultant David Buck Staff Associate for Grant Development John Cady Staff Associate for Evaluation Floyd Cogley, Jr. Staff Associate for Grant Development Jean Hanna, R.N. Staff Associate for Erie (Pa.}, Chautauqua, Cattaraugus and McKean, Pa. Counties Patricia Hoff, R.N. Assistant Director for Nursing and Allied Health: Affairs Elsa Kellberg Staff Associate for Evaluation Michael Miller Staff Associate for Inner City Affairs Robert J. Miller Staff Associate Joan Philipps Staff Assistant for Evaluation Joseph Reynolds Director, Telephone Lecture Network Mrs. Marion Sumner Consultant for Fiscal Affairs Anthony Zerbo, Jr. Assistant Director for Communications Participants Helen Applebaum Occupational Therapy State University of New York at Buffalo Sister Mary Alberta (for Sister Gerard, Admin.) St. Jerome Hospital Batavia, New York Nancy L. Benson (for Sister Armiger) Dean, School of Nursing Niagara University Niagara Falls, New York Donald A. Bradley, V.P. Millard Fillmore Hospital Buffalo, New York Carolyn Brayley Occupational Therapy State University of New York at Buffalo Raymond E. Brewster Mt. View Hospital Lockport, New York Thyra Charles, R.N. Visiting Nursing Association Buffalo, New York Sara Marie Cicarelli Medical Technology State University of New York at Buffalo Frances Cirbus Pulmonary Study Unit Millard Fillmore Hospital Buffalo, New York Louis Cohen Health Association of Niagara County, Inc. Niagara Falls, New York Kay Collard, R.N. Pulmonary Study Unit Millard Fillmore Hospital Buffalo, New York Margaret Connelly, R.N. Director, Allegany County Public Health Nursing Service Angelica, New York Mary Lou Considine Niagara Coalition HANCI Niagara Falls, New York Mrs. Joan Cookfair Family Practice Buffalo, New York Robert M. Cooper Department of Pharmacy State University of New York at Buffalo Buffalo, New York John A. Coulter School of Pharmacy State University of New York at Buffalo William D. Crage Niagara Coalition Health Association Niagara County Inc. Niagara Falls, New York Bette Dale, Executive Director Lockport Senior Citizens Lockport, New York Carolyn Daughtry Department of Mental Health Buffalo, New York John Donnelly Erie, Pennsylvania Alan J. Drinnan, D.D.S., M.D. School of Dentistry State University of New York at Buffalo Barbara Edelman, R.N. Pulmonary Study Unit Millard Fillmore Hospital Buffalo, New York Ruth Elder, Ph.D., M.S.N. Associate Professor of Nursing State University of New York at Buffalo Roy Fabry, Rehabilitation Director Meyer Memorial Hospital Buffalo, New York Gerald Farmer, Director Comprehensive Health Planning Erie, Pennsylvania Dominic F. Falsetti, M.D. Vice-President Health Association Niagara County , Niagara Falls, New York Stanley Grenn United Health Foundation Buffalo, New York Jerrold S. Greenberg, Ed.D. Assistant Professor Department of Instruction State University of New York at Buffalo Sister Margaret Ann Hardner, S.S.J. Erie, Pennsylvania Robert W. Harris Executive Director Western New York Hospital Association Buffalo, New York Mrs. Elizabeth Harvey Family Practice Buffalo, New York A. A. Hosinski, Director Veterans Administration Hospital ,. Batavia, New York . Phyllis Higley, Ph.D. Allied Health Planning State University of New York at Buffalo Mrs. Sylvia Helbert, R.D. Extension Home Economist Erie County Cooperative Extension Buffalo, New York Mrs. Fleeta Hill (for Mr. William Gaitor) “Build” Buffalo, New York Rick Horanburg Aide - Congressman Henry P. Smith tf Rev. Edgar L. Huff Niagara Falls, New York Art Hundsburger Comprehensive Health Planning of Northwestern Pennsylvania, Inc. Erie, Pennsylvania - Doris Jones Niagara Community Action Program Inc. Niagara Falis, New York Robert Joss Community Psychiatry E. J. Meyer Memoria! Hospital Buffalo, New York 23 eT. J. Kittle "|. Family Practice 24 ’ Buffalo, New York ice: Hal Farrier: . Comprehensive Health Planning of - Western New York, Inc. Erie, Pennsy!vania “> James Felsen, M.D. . Indian Health Service Wellsville, New York Roger Fenton Emergency Medical Services Erie County Health Department Buffalo, New York John F. Fortune Erie County Health Department Buffalo, New York John R. Foster Martin Luther King Center Erie, Pennsylvania Cc. M. Fuller American Cancer Society Erie, Pennsylvania Charles Garverick, Ph.D. Behavioral and Related Sciences State University of New York at Buffalo Martin Gerowitz Comprehensive Health Planning of Western New York, Inc. Buffalo, New York Geoffrey Gibson, Ph.D. Sociology Department State University of New York at Buffalo John Glasgow University of Connecticut Rev. Allen H. Goss Niagara County Migrant and Ministry Committee Wilson, New York John C. Grabau Comprehensive Health Planning of Western New York, Inc. Buffalo, New York Larry J. Green, D.D.S. School of Dentistry State University of New York at Buffalo A. Burton Kline Regional Medical Program Service Rockville, Maryland Ruth E. Kocher Regional Nutritionist - New York State Health Department George Lawn, D.P.M. WCA Hospital Jamestown, New York Ralph Levy, retired business man Niagara Falls, New York Victor Lapuma Comprehensive Health Planning of Northwestern Pennsylvania, Inc. Erie, Pennsy!fvania Janet Lindner Visiting Nurse Association Buffalo, New York Robert A. Ludwig Comprehensive Health Planning of Western New York, Inc. Buffalo, New York Kathleen D. Mack, R.N. Genesee Community College Batavia, New York Suzanne Manson Family Practice Buffalo, New York Emanuel Luper, R.N. ’ West Seneca State School Andrea McKown Comprehensive Care E. J. Meyer Memorial Hospital Buffalo, New York Carol McPhee Division of Community Psychiatry Buffalo, New York Robert M. Mercurio, Ph.D. Gowanda State Hospital Gowanda, New York Jean Miller Health Sciences Library State University of New York at Buffalo Mary M. Mogan, R.N. New York State Helath Department Sister Mary Alberta (for Sister Gerard, Administrator} St. Jerome Hospital Batavia, New York Alice Moordian, Executive Director Golden Age Club Niagara Falls, New York Michael Parsnick, Clinic Coordinator _ Niagara County Migrant and Rural Congressman Henry P. Smith Ut Dona d E. “Sayder, D.V.M. EJ: ‘Meyer Memorial Hospital “ Buffalo, "New York | Marguerite van O. K chner - Tuberculosis: and. Respiratory Disease Td wate Peters Niagara County Migrant and. Aural Ministry, Commission “Dorothy Reardon, R. N. pases ‘of Nurses me a Elise f Richardson or ~. Secretary of Health; ‘Education “and Welfare Kany" Sawiné?, B. 3. , era tealth Related Professions qore” “eState University of New York at Bottalo