RADIX and PENGUIN Projects 5P41-RRO0785-15

"Approximate Fuzzy C-means (AFCM) Cluster Analysis of Medical
Magnetic Resonance Image (MRI) Data" Submitted to the International
Conference on Pattern Recognition, Special Session on Diagnostic
Medical Image Processing, Beijing, China, October 1988.

28. DeZegher-Geets, I|., Freeman, A.G., Walker, M.G., Blum, A.L., and
Wiederhold, G. "Intelligent Summarization and Display ‘of On-Line Medical
Records" Accepted for publication in M.D. Computing.

29. DeZegher-Geets, 1.M., Freeman, A.G., Walker, M.G., Blum, R.L., and
Wiederhold, G.C.M.: “Computer-aided Summarization of a Time-oriented
Medical Data Base"; in Proceedings of the Third Annual Conference on
Computerization of Medical Records. Institute for Medical Record
Economics, Chicago, April 1987. Also Stanford Knowledge Systems
Laboratory (KSL) Report KSL-87-18.

30. DeZegher-Geets, _.: Intelligent Summarization of a Time-Oriented
Medical Data Base. Master's thesis, Stanford University, 1987.

31. Downs, S., Walker, M.G., and Blum, R.L.: Automated Summarization of
On-line Medical Records. In Proceedings of the Fifth World Congress
on Medical Informatics (Medinfo), pages 800-804. Elsevier Science
Publishers, 1986.

32. Downs, S.M.: A Program for Automated Summarization of On-line
Medical Records, Master's thesis, Stanford University, 1986.

33. Kuhn, I., Wiederhold, G., Rodnick, J.E.. Ramsey-Klee, D.M. Benett, S.,
Beck, D.D.: Automated Ambulatory Medical Record Systems in the US.,
to be published by Springer-Verlag, 1983, in Information Systems for
Patient Care, B. Blum (ed.), Section Ill, Chapter 14.

34. Shortliffe, E.H., Wiederhold, G., Fagan, L., and Perrault,L. : An
Introduction To Medical Computer Science. Addison-Wesley, 1987. In
preparation, several chapters completed.

35. Walker, M.G. "Expert Systems in Geologic Exploration: Can They Be
Cost Effective?" Accepted for publication in Geobyte.

36. Walker, M.G. "Automated Recognition of Brain Anatomy and Pathology in
MRI and CT: a Review". In preparation for IEEE Transactions on Medical
Image Processing.

37. Walker, M.G., Blum, R.L., and Fagan, L.M. “Minimycin: A Miniature Rule-
Based System" in "M.D.Computing Tutorials", C.J.McDonald (editor),
Springer-Verlag, 1988. Reprinted from M.D.Computing, V.2,No.4, 1985.

38. Walker, M.G., and Blum, R.L. “An Introduction to LISP" in "M.0.Computing
Tutorials", C.J.McDonald (editor), Springer-Verlag, 1988. Reprinted from
M.D.Computing, V.2, No.1, 1985.

39. Walker, M.G. "How Feasible is Automated Discovery?” The Computer
and Philosophy Newsletter, Vol.3, No.1, pp. 1-37, Feb 1988, Center for
Design of Educational Computing, Carnegie Mellon University. Reprinted
from IEEE Expert, Vol.2, No.1, pp. 70-82, Spring 1987.

40. Walker, MG. Clauer, R.C., Samadani, R. Craven, J., and Frank,
L. "Automated Analysis of Auroral Images"; Abstract. EOS, Vol. 68, No.

E. H. Shortliffe 176
5P41-RROO785-15 RADIX and PENGUIN Projects

44, page 1436, November 1987. Poster session paper presented at .
American Geophysical Union Annual Meeting, San Francisco, December
1987.

41. Walker, MG. "Expert Systems in Exploration: Can They Be Cost
Effective?" American Association of Petroleum Geologists, Annual
Convention, Los Angeles, California, June 1987.

42. Walker, M.G., and Blum, R.L:: Towards Automated Discovery from
Clinical Databases: the RADIX Project. In Proceedings of the Fifth World
Congress on Medical Informatics (Medinfo), pages 32-36. Elsevier
Science Publishers, 1986.

43. Walker, M.G., Blum, R.L., and Fagan, L.M.: Minimycin: A Miniature Rule-
Based System. M.D.Computing, Vol. 2, No. 4., 1985.

44. Walker, M.G., and Blum, R.L.: An Introduction to LISP. M.D. Computing,
Vol. 2, No. 1., 1985.

45. Wiederhold, Gio: "Hospital Information Systems": in Encyclopedia of
Medical Devices and Instrumentation, vol.3, Webster,John G.(ed), Wiley
1988, pp.1517--1542

46. Wiederhold, Gio and Paul D. Clayton: "Processing Biological Data in
Real Time"; M.D. Computing, Springer Verlag, Vol.2 No.6, November
1985, pages 16-25, republished in ‘Images, Signals, and Devices’, C.J.
McDonald (editor), Springer Verlag, 1987, pp.107--116.

47. Wiederhold, G.: File Organization for Database Design. McGraw-Hill Book
Company, 1987.

48. Wiederhold, Gio CM, Michael G. Walker, Robert L. Blum, Stephen
M. Downs, and Isabelle deZegher-Geets: "Acquisition of Knowledge from
Medical Records" abstract: in Namen, Daten Themen,
Benutzer- gruppenseminar Medizin |, Systems 87 conference, Munich,
FRG, Oct.1987, pp.213-214.

49. Wiederhold, Gio CM, Michael G. Walker, Waqar Hasan, Surajit Chaudhuri,
Arun Swami, Sang K. Cha, XiaoLei Qian, Marianne Winslett, Linda
DeMichiel, and Peter K. Rathmann: "KSYS: An Architecture for Integrating
Databases and Knowledge Bases"; Computer Science Department,
Stanford University, May 1987; in Amar Gupta and Stuart Madnick
(editors) ‘Technical Opinions Regarding Knowledge- Based Integrated
Information Systems Engineering’, MIT, 1987,

50. Wiederhold, Gio: "Knowledge versus Data"; Chapter 8 of ‘On Knowledge
Base Management Systems: Integrating Artificial Intelligence and
Database Technologies’ (Brodie, Mylopoulos, and Schmidt, eds.),
Springer Verlag, June 1986, pages 77 to 82.

51. Wiederhold, Gio, Robert L. Blum, and Michael Walker: "An Integration of
Knowledge and Data Representation": Proc. of Islamorada Workshop,
Feb.1985, Computer Corporation of America, Cambridge MA; Chapter 29
of 'On Knowledge Base Management Systems: Integrating Artificial
intelligence and Database Technologies' (Brodie, Mylopoulos, and
Schmidt, eds.), Springer Verlag, June 1986, pages 431 to 444.

177 E. H. Shortliffe
RADIX and PENGUIN Projects 5P41-RROO785-15

52. Wiederhold, G.: Views, Objects, and Databases. IEEE Computer
19(12):37-44, December, 1986.

53. Wiederhold, G.C.M, Walker, M.G., Blum, RL. and Downs, S.M::
Acquisition of Knowledge from Data. in Ras, Z.W., and Zemankova,
M. (editors), Proceedings of the International Symposium on
Methodologies for Intelligent Systems, ACM, Oct.1986, Knoxville TN.

54. Missikoff, Michele and Gio Wiederhold: Towards a Unified Approach for
Expert and Database Systems. In ‘Expert Database Systems’, Larry
Kerschberg (editor), Benjamin/Cummings, 1986, pages 383-399; also in
Proceedings of First Workshop on Expert Database Systems, Kiawah
Island, South Carolina, Oct. 1984, vol.1, pp.186-206,

55. Wiederhold, Gio: Knowledge Bases; Future Generations Computer
Systems, North-Holland, vol.1 no.4; April 1985, pp.223--235.

56. Wiederhold, Gio: Disease Registers: Use of Databases to Generate New
Medical Knowledge; in K.Abt, W.Giere and B.Leiber: 'Krankendaten,
Krankheitsregister, Datenschutz’, vol.58, Medizinische Informatik und
Statistik, Springer Verlag, 1985, pp.39-55.

57. Wiederhold, G.: Networking of Data Information, National Cancer Institute
Workshop on the Role of Computers in Cancer Clinical Trials, National
Institutes of Health, June 1983, pp.113-119.

58. Wiederhold, G.: Database Design (in the Computer Science Series);
McGraw-Hill Book Company, New York, NY, May 1977, 678 pp. Second
edition, Jan. 1983, 768 pp.

59. Wiederhold, G.: In D.A.B. Lindberg and P.L. Reichertz (Eds.), Databases
for Health Care, Lecture Notes in Medical Informatics, Springer-Verlag,
1981.

60. Wiederhold, G.: “Database technology in health care"; J. Medical
Systems 5(3):175-196, 1981.

E. Funding Support Status

1987-1988 Principal Investigator: Gio Wiederhold 5/20%
(6 months) DADAISM, Databases for Ada Information System Modules
(SRI//NRL/DoD Stars: $176,931)

1987-1991 Principal Investigator: Gio Wiederhold 10%
Support for Parallel Design in an Engineering Information
System (NSF DTMP 8619595 $153,766, $146,360 prop.,
$156,044 prop.)

1987-1988 Principal Investigator: Gio Wiederhold 5%
Validation of Knowledge from a Database (IBM KBS Menlo
Park, $50,051)

1986-1990 Principal Investigator: Gio Wiederhold 50% effort

Knowledge Base Management Structures (ONR/SPAWAR/ARPA,
N39-84-C-211, task 7: $1,756,410)

E. H. Shortliffe 178
5P41-RROO785-15 RADIX and PENGUIN Projects

1987-1988 Principal Investigator: Gio Wiederhold 5%
Reasoning about R1ME
(Digital Equipment Corp: $131,337)

Requests in process:

1988 (6 mo) Principal Investigator: Gio Wiederhold 20% summer,
10% academic year
DADAISM: DBMS in and for ADA-supported Information
System Management (SRI/NRL/STARS/DoD: $336,719)

1987-1990 Principal Investigator: Gio Wiederhold 25% effort
FACS~Penguin: An Expert Workstation for Flow Cytometry
(NLM/NIH : $856,449 direct costs, approved at high priority)

1987-1990 Associate Investigator: Gio Wiederhold 10% effort
Integrating Knowledge and DBMS (SRI Al/ONR/SPAWAR/AFPA,
to be funded: $75,000 Stanford Subcontract)

1988-1990 Principal Investigator: Gio Wiederhold 10% effort
PARADATA: Databases on Parallel Computers (NSF/IRI/K&DSP,
approved, not yet funded, $499,385 requested)

1986-1989 Principal Investigator: Gio Wiederhold 5% effort
RADIX Project: Software for Automated Peer Review (NCHSR/
DHHS HS5632, approved, unlikely to be funded: $231,996 direct)

I. INTERACTIONS WITH THE SUMEX-AIM RESOURCE

A. Collaborations

SUMEX AIM provides the central communication node for our research. While
individual experiments tend to take place on workstations, the availability of SUMEX
to load and edit files, communicate with colleagues and peers is absolutely critical.

B. Interactions with Other SUMEX-AIM Projects

During the current reporting year we have had frequent interaction with members of
other SUMEX projects; for example, development of algorithms for transforming
INTERNIST data to Bayes form, presentation of research results at Stanford Medical
Information Science Colloquia, discussions of automated discovery and automated
summarization, practical programming issues, and training of Medical Computer
Science Students in the use of KEE, Lisp workstations, and so on. The SUMEX
community is an invaluable resource for providing such interaction.

C. Critique of Resource Management

The DECSystem 20 continues to provide acceptable performance, but it is frequently
over loaded at peak hours. The archiving facility (SAFE) seems to be very loaded,
so that we are forced to make decisions on retention of past material more
frequently than optimal. As users of large databases this is one of the resources
upon we must draw frequently.

The SUMEX resource management continues to be accessible and quite helpful.

New networking links, for instance to the Genetics Department equipment, make new
collaborations viable.

179 E. H. Shortliffe
RADIX and PENGUIN Projects 5P41-RROO785-15

lll. RESEARCH PLANS

A. Project Goals and Plans

The the RADIX Project is now inactive. We plan to keep the results and software
produced accessible for educational and sharing objectives within the MIS program.

~The long-range goal of the PENGUIN project is to integrate our experiment design
advisory system with the set of programs developed in the Genetics department for
FACS operation and control, thereby defining a comprehensive "FACS Workstation”
that should considerably facilitate effective utilization of FACS technology. In the
shorter term, we will:

1. complete the development and testing of the object interface. The
interface should then be made a fully domain-independent module that
could be applied in different contexts. -

2. validate the structure of the database and make it available to the
scientists in the Genetics department, for further refinement as well as
for routine data storage and retrieval.

3. move gradually toward a distributed environment with Macintoshes as
local workstations connected to a central VAX computer, acting as a
database server.

4. develop the expert system module which will be coupled to the
database through the object interface. The expert system module is
aimed at assisting people in an experimental design task. This can
actually be done by taking two different approaches, one being the
actual automatic design of the experiment (i.e., a planning task), the
other being the critique of a protocol proposed by the user. Although
we recognize both directions as important, we will focus on the first
approach because knowing how to properly plan new experiments is a
necessary step prior to any critiquing process. Although the problem
solving strategies of our experts are not yet fully determined, we plan to
investigate the general concept of hierarchical planning.

B. Justification and Requirements for Use of SUMEX
The PENGUIN project is dependent on new methods of communication among clinical
and academic researchers.

C. Recommendations for Resource Development

The movement towards work-stations is obviously the direction of the future. We
hope that the communication capability, now supported via the central SUMEX facility,
the DEC 2060, will continue to be supported. We continue to make constant use of
this communication with our colleagues within and outside of Stanford, preparation of
large documents, file and database handling, and program demonstrations.

E. H. Shortliffe 180
5P41-RROO785-15 National AIM Projects

IV.B. National AIM Projects

The following group of projects is formally approved for access to the AIM aliquot of
the SUMEX-AIM resource. Their access is based on review by the AIM Advisory
Group and approval by the AIM Executive Committee.

In addition to the progress reports presented here, abstracts for each project and its
individual users are submitted on a separate Scientific Subproject Form.

181 E. H. Shortliffe
ATTENDING Project 5P41-RROO785-15

IV.B.1. ATTENDING Project

ATTENDING Project -- Expert Critiquing Systems

Perry L. Miller, M.D. Ph.D.
Department of Anesthesiology
Yale University School of Medicine
New Haven, CT 06510

Il. SUMMARY OF RESEARCH PROGRAM

A. Project rationale

Our project is exploring the “critiquing” approach to bringing computer-based advice
to the practicing physician.

Critiquing is a different approach to the design of artificial intelligence based expert
systems. Most medical expert systems attempt to simulate a physician's decision-
making process. As a result, they have the clinical effect of trying to tell a physician
what to do: how to practice medicine. In contrast, a critiquing system first asks the
physician how he contemplates approaching his patient’s care, and then critiques that
plan. In the critique, the system discusses any risks or benefits of the proposed
approach, and of any other approaches which might be preferred. It is anticipated
that the critiquing approach may be particularly well suited for domains, like
medicine, where decisions involve a great deal of subjective judgment.

To date, several prototype critiquing systems have been developed in different
medical domains, including:

1. ATTENDING, the first system to implement the critiquing approach,
critiques anesthetic management.

2. HT-ATTENDING critiques the pharmacologic management of essential
hypertension.

3. VQ-ATTENDING critiques aspects of ventilator management.

4. PHEO-ATTENDING critiques the laboratory and radiologic workup of a
patient for a suspected pheochromocytoma.

In addition, a domain-independent system, ESSENTIAL-ATTENDING, has been
developed to facilitate the implementation of critiquing systems in other domains.

C. Highlights of Research Progress

Current projects include the following:

HT-ATTENDING -- The original prototype version of HT-ATTENDING has been
converted to the ESSENTIAL-ATTENDING format, and updated to reflect current
thinking in the field of hypertension management. A major priority is to subject this
system to validation and clinical evaluation. In addition, we are currently exploring
concretely how best to disseminate the system as a practical consultation tool.

E. H. Shoriliffe 182
5P41-RROO785-15 ATTENDING Project

ICON: Critiquing Radiological Differential Diagnosis -- Most existing diagnostic
computer systems produce a ranked differential diagnosis as their output. In this
process, the rich structure of the knowledge that went into developing the diagnoses
may be lost to the user. ICON explores a different approach to diagnostic advice in
the domain of radiology. To use ICON, a radiologist describes a set of findings seen
on chest x-ray, together with a proposed diagnosis. ICON then produces a detailed
analysis of why the observed findings serve to support or to rule out the diagnosis.
It may also suggest further findings that might help refine the diagnosis, again
explaining why the findings are important. In addition, we are currently exploring how
images might be incorporated into ICON’s knowledge base to enhance its
consultative effectiveness. We see this project as an exploration of the question of
how prose and images might best be indexed and organized for purposes of
explanation in the context of a particular clinical case.

D. Publications

1. Miller, P.L. (Ed.): Selected Topics in Medicai Artificial Intelligence. New
York: Springer-Verlag (in press).

2. Rennels, G.D., Miller, P.L.: Artificial intelligence research in anesthesia
and intensive care. Journal of Clinical Monitoring (in press).

3. Miller, P.L., Rennels, G.D.: Prose generation from expert systems: An
applied computational linguistics approach. Al Magazine (in press).

4. Rennels, G.D., Shortliffe, E.H., Stockdale, F.E., Miller, P.L.: A
computational model of reasoning from the clinical literature. Computer
Methods and Programs in Biomedicine 24:139-149, 1987.

5. Rennels, G.D., Shortliffe, E.H., Stockdale, F.E., Miller, P.L.: A structured
representation of the clinical literature and its use in a medical
management advice system. Bulletin du Cancer 74:215-220, 1987.

6. Miller, P.L., Barwick, K.W., Morrow, J.S., Powsner, S.M., Riely, C.A.:
Semantic relationships and medical bibliographic retrieval: A preliminary
assessment. Computers and Biomedical Research 21:64-77, 1988.

7. Miller, P.L., Morrow, J.S., Powsner, S.M., Riely, C.A.: Semantically
assisted medical bibliographic retrieval: An experimental computer
system. Bulletin of the Medical Library Association 76:131-136, 1988.

8. Miller, P.L.: Exploring the critiquing approach: Clinical practice-based
feedback by computer. Biomedical Measurement, Informatics and Control
(in press).

9. Rennels, G.D., Shortliffe, £.H., Stockdale, F.E., Miller, P.L.: A
computational model of reasoning from the clinical literature. The Al
Magazine (accepted pending revision).

10. Miller, P.L., Fisher, P.R.: Causal models in medical artificial intelligence.
Proceedings of the Eleventh Symposium on Computer Applications in
Medical Care, Washington, D.C., November 1987, pp. 17-22.

11. Powsner, S.M., Barwick, K.W., Morrow, J.S., Riely, C.A., Miller, P.L.:
Coding semantic relationships for medical bibliographic retrieval: A
preliminary study. Proceedings of the Eleventh Symposium on Computer
Applications in Medical Care, Washington, D.C., November 1987, pp.
108-112.

183 E. H. Shortliffe
ATTENDING Project 5P41-RROO785-15

E. Funding Support

EXPERT COMPUTER SYSTEMS WHICH CRITIQUE PHYSICIAN PLANS
NIH Grant RO1 LM04336

Principal Investigator: Perry L. Miller, M.D., Ph.D.

Annual Direct Costs: approximately $95,000

Period of Support: 3/1/88-2/28/91

This grant supports the exploration of the critiquing approach to bringing
computer-based advice to the physician, focusing especially on refining and
evaluating the HT-ATTENDING system which critiques hypertension management,
and on developing tools for knowledge base maintenance and updating.

SUPPORT OF THE UNIFIED MEDICAL LANGUAGE PROGRAM
NLM Contract NO1 LM63524

Principal Investigator: Perry L. Miller, M.D., Ph.D.

Annual Direct Costs: approximately $100,000

Period of Support: 8/22/86-8/21/88

This research contract is part of the NLM Unified Medical Language (UML)
program. We are defining a set of semantic relationships which could be used

to augment the UML, to facilitate such functions as medical bibliographic
retrieval.

SUPPORT FOR MEDICAL INFORMATICS AND ARTIFICIAL INTELLIGENCE
Ira DeCamp Foundation
Co-Principal Investigators: Henry A. Swett, M.D.
Perry L. Miller, M.D., Ph.D.
Annual Costs: $75,000
Period of Support: 7/1/86-6/30/90

This grant supports activities of our Medical Informatics program.

MEDICAL INFORMATICS RESEARCH TRAINING AT YALE
Principal Investigator: Perry L. Miller, M.D., Ph.D.

NLM Training Grant T15 LM07056

Period of Support: 7/1/87-6/30/92

This grant currently supports 3 postdoctoral trainees and three predoctorai
trainees in Medical Informatics.

Pending Support
BIOTECHNOLOGY COMPUTER RESEARCH AT YALE
Perry L. Miller, M.D., Ph.D. (Pl)
RFA 88-LM-01
This grant proposes computer-related research in molecular biology.
ll. INTERACTIONS WITH THE SUMEX-AIM RESOURCE

Until last year we were using the RUTGERS-AIM Resource. We used that facility to

E. H. Shortliffe 184
5P41-RROO785-15 ATTENDING Project

implement all of our early critiquing systems. This year we moved the HT-
ATTENDING project. to the SUMEX-AIM facility. Very recently, we have moved HT-
ATTENDING to a local VAX station, and we are currently migrating entirely to local

workstations for our research. This past year, our main use of SUMEX-AIM has been
the following:

1. We used SUMEX-AIM to refine and to demonstrate HT-ATTENDING.

2. We used SUMEX-AIM for communication access to the national AIM
community.

We have found our use of the RUTGERS-AIM and SUMEX-AIM facilities to be
extremely valuable. They provided us the resources needed to initiate our research
and to continue several projects which are still active. They provided a natural
vehicle to allow us to demonstrate the various systems easily, both in the United
States and in Europe. Also, they enabled us to collaborate very closely with Dr.
Glenn Rennels in his Stanford Medical Information Science thesis project on the
Roundsman system. Via SUMEX-AIM and RUTGERS-AIM, Dr. Rennels and Dr. Miller
maintained very close contact, typically with multiple messages each week, and
sometimes within a single day.

ili. FUTURE PLANS

We plan to continue our critiquing research as outlined above. As discussed
previously, we have now moved all our research onto local workstations, and will
continue the research using these resources.

185 E. H. Shortliffe
INTERNIST-I/QMR Project 5P41-RROO785-15

IV.B.2. INTERNIST-I/QMR Project

INTERNIST-I/QMR (Quick Medical Reference)

Jack. D. Myers, M.D.
University Professor Emeritus (Medicine)

Randolph A. Miller, M.D.
Associate Professor of Medicine
Chief, Section of Medical Informatics

University of Pittsburgh
1291 Scaife Hall
Pittsburgh, Pa., 15261

|. SUMMARY OF RESEARCH PROGRAM

A. Project rationale

The principal objective of this project is the development of a high-level computer
diagnostic program in the broad field of internal medicine as an aid in the solution of
complex and complicated diagnostic problems. To be effective, the program must be
capable of multiple diagnoses (related or independent) in a given patient.

A major achievement of this research undertaking has been the design of a program
called INTERNIST-1, along with an extensive medical knowledge base. This program
has been used over the past decade to analyze many hundreds of difficult diagnostic
problems in the field of internal medicine. These problem cases have included
cases published in medical journals (particularly Case Records of the Massachusetts
General Hospital, in the New England Journal of Medicine), CPCs, and unusual
problems of patients in our Medical Center. In most instances, but by no means all,
INTERNIST-I has performed at the level of the skilled internist, but the experience
has highlighted several areas for improvement.

B. Medical Relevance and Collaboration
The program inherently has direct and substantial medical relevance.

The development of the QUICK MEDICAL REFERENCE (QMR) under the leadership of
Dr. Randolph A. Miller has allowed us to distribute the INTERNIST-I knowledge base
in a modified format to over twenty other academic medical institutions. The
knowledge base can thereby be used as an “electronic textbook" in medical
education at all levels -- by medical students, residents and fellows, and faculty and
staff physicians. This distribution is continuing to expand.

The INTERNIST-I program has been used in recent years to develop patient
management problems for the American College of Physician's Medical Knowledge
Self-assessment Program.

E. H. Shortliffe 186
5P41-RROO785-15 INTERNIST-I/QMR Project

C. Highlights of Research Progress

C.1 Accomplishments this past year

For the record, it should be noted that grant support for the QMR project has come
solely from the CAMDAT Foundation of Farmington, Conn., from the Department of
Medicine of the University of Pittsburgh, and from Dr. Miller's NLM RCDA grant and
NLM RO1 grant.

In the past year, the University of Pittsburgh was named recipient of a National
Library of Medicine Medical Informatics Training Grant Award.

The group of us (Myers, Miller and Masarie) together with assigned residents in
internal medicine and fellows in medical informatics are continuing to expand the
knowledge base and to incorporate the diagnostic consultative program into QMR.
The computer program for the interrogative part of the diagnostic program is the
main remaining task. An editor for the QMR knowledge base, as modified from the
INTERNIST-I knowledge base, has been written from scratch in Turbo Pascal by Dr.
Masarie. The entire QMR program can be accommodated in, maintained (particularly
edited) and operated on individual IBM PC-AT computers.

Our group has incorporated into the QMR diagnostic consultant program modifications
and embeilishments of the INTERNIST-! knowledge base, and will continue to. do so
over the next year by adding "facets" of diseases or syndromes. This addition and
modification is expected to improve the performance of the diagnostic consultant
program.

The medical knowledge base has continued to grow both in the incorporation of new
diseases and the modification of diseases already profiled so as to include recent
advances in medical knowledge. Several dozen new diseases have been profiled
during the past year. The current number of diseases in the QMR knowledge base is
589, and 4237 possible patient findings are included.

C.2 Research in progress
There are four major components to the continuation of this research project:

1. The enlargement, continued updating, refinement and testing of the
extensive medical knowledge base required for the operation of
INTERNIST-I and the QMR modification.

2. Institution of field trials of QMR on the clinical services in internal
medicine at the Health Center of the University of Pittsburgh. This has
been accomplished in a limited fashion, which began in 1987: a
“computer-based diagnostic consultation service" has been made
available to attending physicians and house staff on the medical services
of our two main teaching hospitals. Institutional Review Board (IRB)
approval was granted to the service before it was initiated.

3. Expansion of the clinical field trials to other university health centers
which have expressed interest in working with the system.

4. Adaptation of the diagnostic program and data base of INTERNIST-I and

the QMR modification to subserve educational purposes and the
evaluation of clinical performance and competence.

187 E. H. Shortliffe
INTERNIST-!/QMR Project

Current activity is devoted mainly to the first two of these, namely, the continued
development of the medical knowledge base, and the implementation of the improved
diagnostic consulting program, and preliminary evaluation of the diagnostic

consultation service.

D. List of relevant publications

1.

10.

Myers JD. The Background of INTERNIST-I and QMR. In: Proceedings
of the History of Medical informatics Conference. National Library of
Medicine. pp. 195-197, November 1987.

. Masarie, F.E., Miller, R.A. Medical Subject Headings and Medical

Terminology: An analysis of terminology used in hospital charts. Bulletin
of the Medical Library Association, 1987; 75:89-94.

. Masarie FE, Miller RA. INTERNIST-I to Quick Medical Reference (QMR):

Transition from a mainframe to a microcomputer. Proceedings Ninth
Annual IEEE/Engineering in Biology and Biology Society. IEEE Press.
pp. 1521- 1522, November 1987.

. Parker, RC, Miller RA. Using causal knowledge to create simulated

patient cases: The CPCS project as an extension of INTERNIST-I.
Proceedings of the Eleventh Annual Symposium on Computer
Applications in Medical Care. pp. 473-480, November 1987.

. Bankowitz RA, Blumenfeld BH, Miller RA, et al. User variability in

abstracting and entering printed case histories with Quick Medical
Reference (QMR). Proceedings of the Eleventh Annual Symposium on
Computer Applications in Medical Care. pp. 68-73, November 1987.

. Masarie FE, Miller RA. Quick Medical Reference (QMR): An information

management tool for clinical diagnosis. IN: Critical Reviews in Medical
Informatics. Boca Raton, Florida, CRC Press, 1988.

. Parker RC, Miller RA. Using causal knowledge to created simulated

patient cases: The CPCS project as an extension of INTERNIST-I. IN:
Miller PL (ed) Topics in Medical Artificial Intelligence. Computers and
Medicine Series, Springer-Verlag, New York, 1988.

. Challinor SM, McNeil MA, Bankowitz RA, et. al. Evaluation of a

Computer-Assisted General Medicine Diagnostic Consultation Service.
Abstract presented at the 11th Annual SGIM Meeting, Washington, D.C.,
April 27-29, 1988.

. Miller RA. From Automated Medical Records to Expert System

Knowledge Bases: Common Problems in Representing and Processing
Patient Data. Topics in Health Record Management. 75(3):23-36, 1987

Miller RA. Computer-based Diagnostic Decision-making. Medical Care.
25(12):S148-S152, 1987.

E. Funding support

1. Diagnostic-internist: A Computerized Medical Consultant
Randolph A. Miller, M.D.

E. H. Shortliffe 188

5P41-RROO785-15
5P41-RROO785-15 INTERNIST-|/QMR Project

Associate Professor of Medicine

Chief, Section of Medical Informatics

University of Pittsburgh Department of Medicine

National Library of Medicine - Development Award Research
Career

National Institutes of Health

5 KO4 LM00084-03

09/30/85 - 09/29/86 - $55,296

09/30/86 - 09/29/87 - $55,296

09/30/87 - 09/29/88 - $54,648

Support recommended for 2 additional years ending 09/29/90,
The Amounts to be determined annually.

2. Developing INTERNIST-I Knowledge Base into a Resource
Randolph A. Miller, M.D.
Associate Professor of Medicine
Chief, Section of Medical Informatics
University of Pittsburgh Department of Medicine
National Library of Medicine
National Institutes of Health

1 RO1 LM04622-01

08/30/87 through 09/29/90
09/30/87 - 09/29/88 - $71,892
09/30/88 - 09/29/89 - $112,938
09/30/89 - 09/29/90 - $112,580

3. Pittsburgh Medical Information Sciences Training Program
Randolph A. Miller, M.D.
Associate Professor of Medicine
Chief, Section of Medical Informatics
University of Pittsburgh Department of Medicine
National Library of Medicine
National Institute of Health
07/01/87 through 06/30/92
07/01/87 - 06/30/88 - $153,454
07/01/88 - 06/30/89 - $199,038
07/01/89 - 06/30/90 - $217,572
07/01/90 - 06/30/91 - $278,092
07/01/91 - 06/30/92 - $276,596

ll. INTERACTIONS WITH THE SUMEX-AIM RESOURCE

A,B. Medical Collaborations and Program Dissemination Via SUMEX

INTERNIST-I and QMR remain in a stage of research and particularly development. As
noted above, we are continuing to develop better computer programs to operate the
diagnostic system, and the knowledge base cannot be used very effectively for
collaborative purposes until it has reached a critical stage of completion. These
factors have stifled collaboration via SUMEX up to this point and will continue to do
so for the next year or two. In the meanwhile, through the SUMEX community there
continues to be an exchange of information and states of progress. Such
interactions particularly take place at the annual AIM Workshop.

189 E. H. Shortliffe
INTERNIST-I/QMR Project §P41-RROO785-15

C. Critique of Resource Management

SUMEX has been an excellent resource for the development of INTERNIST-I. Our
large program is handled efficiently, effectively and accurately. The staff at SUMEX
have been uniformly supportive, cooperative, and innovative in connection with our
project's needs.

lil. RESEARCH PLANS

A. Project Goals and Plans

Continued effort to complete the medical knowledge base in internal medicine will be
pursued including the incorporation of newly described diseases and new or altered
medical information on "old" diseases. The latter two activities have proven to be
more formidable than originally conceived.

B. Justification and Requirements for Continued SUMEX Use

Our use of SUMEX has declined with the adaptation of our programs to the IBM PC-
AT. Nevertheless, the excellent facilities of SUMEX are expected to be used for
certain developmental work. It is intended for the present to keep INTERNIST-1 at
SUMEX for comparative use as QMR is developed here. We will not need the DEC
2060 beyond its anticipated phase-out in early 1989, but will require access to its
replacement for mailing purposes and to maintain contact with the national medical
informatics community.

C. Needs and Plans for Other Computing Resources Beyond SUMEX-AIM

Our predictabie needs in this area will be met by our recently acquired personal
work stations.

E. H. Shortliffe 180
5P41-RROO785-15 MENTOR Project

IV.B.3. MENTOR Project

MENTOR Project -- Medical Evaluation of Therapeutic Orders

Stuart M. Speedie, Ph.D.
School of Pharmacy
University of Maryland

Terrence F. Blaschke, M.D.
Department of Medicine
Division of Clinical Pharmacology
Stanford University

1. SUMMARY OF RESEARCH PROGRAM

A. Project Rationale

The goal of the MENTOR (Medical EvaluatioN of Therapeutic ORders) project is to
design and develop an expert system for monitoring drug therapy for hospitalized
patients that will provide appropriate advice to physicians concerning the existence
and management of adverse drug reactions. The computer as a record-keeping
device is becoming increasingly common in hospital-based health care, but much of
its potential remains unrealized. Furthermore, this information is provided to the
physician in the form of raw data which is often difficult to interpret. The wealth of
raw data may effectively hide important information about the patient from the
physician. This is particularly true with respect to adverse reactions to drugs which
can only be detected by simultaneous examinations of several different types of data
including drug data, laboratory tests and clinical signs.

In order to detect and appropriately manage adverse drug reactions, sophisticated
medical knowledge and problem solving is required. Expert systems offer the
possibility of embedding this expertise in a computer system. Such a system could
automatically gather the appropriate information from existing record-keeping systems
and continually monitor for the occurrence of adverse drug reactions. Based on a
knowledge base of relevant data, it could analyze incoming data and inform
physicians when adverse reactions are likely to occur or when they have occurred.
The MENTOR project is an attempt to explore the problems associated with the
development and implementation of such a system and to implement a prototype of a
drug monitoring system in a hospital setting.

B. Medical Relevance and Collaboration

A number of independent studies have confirmed that the incidence of adverse
reactions to drugs in hospitalized patients is significant and that they are for the
most part preventable. Moreover, such statistics do not include instances of
suboptimal drug therapy which may resuit in increased costs, extended length-of-
stay, or ineffective therapy. Data in these areas are sparse, though medical care
evaluations carried out as part of hospital quality assurance programs suggest that
suboptimal therapy is common.

Other computer systems have been developed to influence physician decision
making by monitoring patient data and providing feedback. However, most of these

191 E. H. Shortliffe
MENTOR Project 5P41-RROO785-15

systems suffer from a significant structural shortcoming. This shortcoming involves
the evaluation rules that are used to generate feedback. In all cases, these criteria
consist of discrete, independent rules, yet medical decision making is a complex
process in which many factors are interrelated. Thus, attempting to represent
medical decision-making as a discrete set of independent rules, no matter how
complex, is a task that can, at best, result in a first-order approximation of the
process. This places an inherent limitation on the quality of feedback that can be
provided. As a consequence it is extremely difficult to develop feedback that
explicitly takes into account all information available on the patient. One might
speculate that the lack of widespread acceptance of such systems may be due to
the fact that their recommendations are often rejected by physicians. These
systems must be made more valid if they are to enjoy widespread acceptance
among physicians.

The MENTOR system is designed to address the significant problem of adverse drug
reactions by means of a computer-based monitoring and feedback system to
influence physician decision-making. It employs principles of artificial intelligence to
create a more valid system for evaluating therapeutic decision-making.

The work in the MENTOR project is a collaboration between Dr. Blaschke at
Stanford University, Dr. Speedie at the University of Maryland, and Dr. Charles
Friedman at the University of North Carolina. Dr. Speedie provides the expertise in
the area of artificial intelligence programming. Dr. Blaschke provides the medical
expertise. Dr. Friedman contributes expertise in the area of physician feedback
design and system impact evaluation. The blend of previous experience, medical
knowledge, computer science knowledge and evaluation design expertise they
represent is vital to the successful completion of the activities in the MENTOR
project.

C. Highlights of Research Progress

The MENTOR project was initiated in December, 1983. The project has been funded
by the National Center for Health Services Research since January 1, 1985. Initial
effort focused on exploration of the problem of designing the MENTOR system. As
of June 1, 1988, a working prototype system has been developed and is undergoing
evaluation. The prototype consists of a Patient Data Base, an Inference Engine, an
Advisory Module and a Medical Knowledge Base. The Medical Knowledge Base
currently contains information related to aminoglycoside therapy, digoxin therapy,
potassium supplementation, surgical prophylaxis, and microbiology lab reports. The
system is currently implemented on a Xerox 1186 Al Workstation. Another version of
the Patient Data Base has been developed for a VAX 750 that is connected via an
asynchronous line to the 1186 running the inference engine. The project has
received additional funding from the National Center for Health Services Research to
install and evaluate the MENTOR system in a Veterans Administration Hospital. This
effort began in June of 1988 and will continue for two additional years. The VA
system will reside on an 1186 and a VAX Station Il connected directly to the VA's
Ethernet LAN, and accessing hospital data through the FILEMAN software.

E. Funding Support
Title: MENTOR: Monitoring Drug Therapy for Hospitalized Patients
Principal Investigators:

Terrence F. Blaschke, M.D.
Division of Clinical Pharmacology

E. H. Shortliffe 192
5P41-RROO785-15 MENTOR Project

Department of Medicine
Stanford University

Stuart M. Speedie, Ph.D.
School of Pharmacy
University of Maryland

Funding Agency: National Center for Health Services Research
Grant Identification Number: 1 R18 HS05263

Total Award: January 1, 1985 - May 31, 1990 $1,056,201 Total
Direct Costs

Current Period: June 1, 1988 - May 31, 1989 $396,569 Total
Direct Costs

ll. INTERACTIONS WITH THE SUMEX~AIM RESOURCE

A. Medical Collaborations and Program Dissemination via SUMEX

This project represents a collaboration between faculty at Stanford University
Medical Center, the University of Maryland School of Pharmacy, and the University of
North Carolina in exploring computer-based monitoring of drug therapy. SUMEX,
through its communications capabilities, facilitates this collaboration of geographically

separated project participants by providing electronic mail and file exchange between
sites.

B. Sharing and Interactions with Other SUMEX-AIM Projects

Interactions with other SUMEX-AIM projects has been on an informal basis. Personal
contacts have been made with individuals working on the ONCOCIN project
concerning system development issues. Dr. Perry Miller has also been of assistance
by providing software for advisory generation. Given the geographic separation of
the investigators, the ability to exchange mail and programs via the SUMEX system
as well as communicate with other SUMEX-AIM projects is vital to the success of
the project.

C. Critique of Resource Management

To date, the resources of SUMEX have been fully adequate for the needs of this
project. The staff have been most helpful with any problems we have had and we
are quite satisfied with the current resource management.

ill. RESEARCH PLANS

A. Project Goals and Plans

The MENTOR project has the following goals:

1. Implement a prototype computer system to continuously monitor patient
drug therapy in a hospital setting. This will be an expert system that
will use a modular, frame-oriented form of medical knowledge, a
separate inference engine for applying the knowledge to specific
situations, and automated collection of data from hospital information
systems to produce therapeutic advisories.

193 E. H. Shortliffe
MENTOR Project 5P41-RRO0785-15

2. Select a small. number of important and frequently occurring medical
settings (e.g., combination therapy with cardiac glycosides and diuretics)
that can lead to therapeutic misadventures, construct a comprehensive
medical knowledge base necessary to detect these situations using the
information typically found in a computerized hospital information system
and generate timely advisories intended to alter behavior and avoid
preventable drug reactions.

3. Design and begin to implement an evaluation of the impact of the
prototype MENTOR system on physicians’ therapeutic decision-making
as well as on outcome measures related to patient health and costs of
care.

1988 will be spent on continued prototype development in six content areas,
refinement of the inference mechanisms, and installation of the system at the Palo
Alto Veterans Administration Hospital.

B. Justification and Requirements for Continued SUMEX Use

This project needs continued use of the SUMEX facilities for one primary reason.
Access to SUMEX is necessary to support the collaborative efforts of geographically
separated development teams at Stanford and the University of Maryland.

Furthermore, the MENTOR project is predicated on the access to the SUMEX
resource free of charge over the next two years. Given the current restrictions on
funding, the scope of the project would have to be greatly reduced if there were
charges for use of SUMEX.

C. Needs and Plans for Other Computing Resources Beyond SUMEX-AIM

A major long-range goal of the MENTOR project is to implement this system on a
independent hardware system of suitable architecture. It is recognized that the full
monitoring system will require a large patient data base as well as a sizeable
medical knowledge base and must operate on a close to real-time basis. Ultimately,
the SUMEX facilities will not be suitable for these applications. Thus, we have
transported the prototype system to a dedicated hardware system that can fully
support the the planned system and which can be integrated into a Hospital
Information System. For this purpose a VAX 750 and three Xerox 1186 workstations
have been acquired and our development efforts have been transferred to them.

D. Recommendations for Future Community and Resource Development

In the time we have been associated with SUMEX, we have been generally pleased
with the facilities and services. However, it is clearly evident that the users' almost
insatiable demands for CPU cycles and disk space cannot be met by a single central
machine. The best strategy would appear to be one of emphasizing powerful
workstations or relatively small, multi-user machines linked together in a nationwide
network with SUMEX serving as the its central hub. This would give the individual
users much more control over the resources available for their needs, yet at the
same time allow for the communications among users that have been one of
SUMEX's strong points.

For such a network to be successful, further work needs to be done in improving the
network capabilities of SUMEX to encourage users at sites other than Stanford.
Further work is also needed in the area of personal workstations to link them to
such a network. Given the successful completion of this work, it would be

E. H. Shortliffe 194
5P41-RROO785-15 MENTOR Project

reasonable to consider the phase-out of the central SUMEX machine years and its
replacement by an efficient, high-speed communications server.

195 E. H. Shortliffe
Pilot Stanford Projects 5P41-RROO785-15

IV.C. Pilot Stanford Projects

Following are descriptions of the informal pilot projects currently using the Stanford
portion of the SUMEX-AIM resource, pending funding, full review, and authorization.

In addition to the progress reports presented here, abstracts for each project are
submitted on a separate Scientific Subproject Form.

E. H. Shortliffe 196
5P41-RROO785-15 REFEREE Project

IV.C.1. REFEREE Project

REFEREE Project

Principal Investigator: Bruce G. Buchanan, Ph.D.
Computer Science Department
Stanford University

Co-Principal Investigator: Byron W. Brown, Ph.D.
Department of Medicine
Stanford University

Associate Investigator: Daniel E. Feldman, Ph.D., M.D.
Department of Medicine
Stanford University

|. SUMMARY OF RESEARCH PROGRAM

A. Project Rationale

The goals of this project are related both to medical science and _ artificial
intelligence: (a) use Al methods to allow the informed but non-expert reader of the
medical literature to evaluate a randomized clinical trial, and (b) use the
interpretation of the medical literature as a test problem for studies of knowledge
acquisition and fusion of information from disparate sources. REFEREE and
REVIEWER, a planned extension, will be used to evaluate the medical literature of
Clinical trials to determine the quality of a clinical trial, make judgments on the
efficacy of the treatment proposed, and synthesize rules of clinical practice. The
research is an initial step toward a more general goal - building computer systems to
help the clinician and medical scientist read the medical literature more critically and
more rapidly for use in making clinical decisions.

B. Medical Relevance

The explosive growth of the medical literature has created a severe information gap
for the busy clinician. Most physicians can afford neither the time required to study
all the pertinent journal articles in their field, nor the risk of ignoring potentially
significant discoveries. The majority of clinicians, in fact, have little sophistication in
epidemiology and statistics; they must nonetheless base their pragmatic decisions on
a combination of clinical experience and published fiterature. The clinician's
computerized assistant must ferret out useful maxims of clinical practice from the
medical literature, pass judgment on the quality of medical reports, evaluate the
efficacy of proposed treatments, and adjudicate the interpretation of conflicting and
even contradictory studies.

C. Highlights of Progress

REFEREE presently encodes the methodological knowledge of a highly regarded
biostatistician at Stanford (Dr. Bill Brown). The system allows the informed but non-
expert reader of the medical literature to evaluate the credibility of a randomized
clinical trial.

In the future, REFEREE and its extensions will alleviate the knowledge-acquisition

197 E. H. Shortliffe
REFEREE Project 5P41-RROO785-15

bottleneck for an automated medical decision-maker: the program will evaluate the
quality of a clinical trial, judge the efficacy of the treatment proposed therein, and
synthesize rules of clinical practice. For the present, however, the fusion of
knowledge from disparate sources remains a problem in pure Al. The efforts of the
REFEREE team have instead focused their efforts on the refinement and deepening
of REFEREE'’s biostatistical knowledge by applying effective knowledge acquisition
and knowledge engineering techniques. Dr. Diana Forsythe and Dr. Harold Lehmann
are developing and using interview methods to acquire this knowledge from Dr.
Brown, and R. Martin Chavez is implementing this in the prototype REFEREE expert
system.

The REFEREE prototype is a consultant that evaluates the design and reporting of a
single conclusion from randomized control trial for its credibility. It contains, in
preliminary form, Professor Brown's expert knowledge of biostatistics. REFEREE
evaluates each statistical procedure described by the authors of the paper. The
automated consultant then determines the most appropriate method for the problem
at hand, based on the design of the trial and the hypotheses to be tested. REFEREE
checks critical assumptions, looks for possible statistical abuses, and verifies
adjustments.

The Knowledge Base: Randomized controlled trials are used to test hypotheses
regarding the effectiveness of various kinds of medical interventions. Dr. Brown
classifies studies on the basis of three major attributes: the type of intervention
tested (e.g. drug, surgery, health process change, etc.); the type of endpoint against
which that intervention was tested (e.g. mortality, objective morbidity, subjective
morbidity, etc.); and the type of conclusion drawn by the investigator/author on the
basis of the research (e.g. that different treatments do or do not produce different
outcomes, that a particular treatment is or is not cost-effective, etc.). Following this
classificatory scheme, we decided to begin by producing a prototype REFEREE
system that would help the reader to evaluate a single published conclusion
concerning the effect of a given drug treatment on mortality.

Knowledge Acquisition: Having defined the scope of the initial knowledge base, we
turned to the problem of collecting the information from Dr. Brown for inclusion in the
system, i.e. knowledge acquisition. This task generally involves a relatively long-
term process of face-to-face information gathering during sessions between the
expert and one or more knowledge engineers. Dr. Diana Forsythe has noted a
parailel between the communicative and analytical tasks involved in knowledge
acquisition and those undertaken in ethnographic research. For this reason, we
included an anthropologist in the research team and make use of ethnographic
techniques in order to maximize the efficiency and quality of the data collection
process.

Dr. Lehmann and Dr. Forsythe have carried out several months of systematic
interviews with Dr. Brown in order to begin the process of constructing and refining
the knowledge base for the current REFEREE prototype. We have combined a case-
based approach that allows us actively to observe Dr. Brown as he reads papers,
with semi-directed interviewing oriented toward understanding his terminology and
category system. We find that these techniques work very well: Dr. Brown's
interest in the knowledge acquisition process has been sustained, and indeed has
increased over time as the system based on his expertise has evolved. He is
clearly comfortable with this approach, and notes that it has actually afforded him
additional insight into the way he interprets the literature.

Over the past year we have altered our knowledge representation from that of rules
to that of an influence diagram. This is an acyclic directed graph of propositions or
variables connected by links, where the absence of a link indicates conditional

E. H. Shortliffe 198
5P41-RROO785-15 REFEREE Project

independence of the two variables. This formalism has been used in decision
analysis to enable experts to convey their knowledge of a domain, and, more
recently, has been used in Al to represent that knowledge in expert systems. This
shift in formalism significantly altered the knowledge acquisition process and the
implementation of that knowledge in our program.

Based on information from our expert, we have taken credibility as ‘the goal
parameter of the present system. This goal is defined operationally by Dr. Brown as
“my odds that the conclusion of the paper would be replicated in an experiment
based on the methods reported in the paper but without any of the flaws". In
assessing credibility, for instance, Dr. Brown considers the blindedness of the
randomization, the blindedness of the execution, the equivalence of the two groups
at baseline, the equivalence in treatment of the two groups, the completeness of
results reporting, and the propriety of the statistical analysis. We recognize that
these variables are not all conditionally independent on credibility, work is in
progress to assess as accurately as possible just what the conditional relationships
are. Our use of influence diagrams has numerous advantages: the approach is
acceptable to Dr. Brown, it is flexible, it can represent several aspects of the
structure of the knowledge used by the expert, and the resultant data can be
entered easily into the computer.

Inference in REFEREE: REFEREE was originally built within EMYCIN, a backward-
chaining rule-based Al environment developed from MYCIN at Stanford. This
environment is ideally suited for ordered collection of evidence and a diagnosis of
the goal state at the end of that process. The state of belief or knowledge in
parameters not directly between the evidence and the goal state is irrelevant. Our
present system focuses on maintaining consistency over the entire knowledge base
as new evidence is incorporated into the system. The constraints implied by the
new data and Dr. Brown's prior knowledge are propagated throughout the system by
Judea Pearl's message-passing algorithm for belief networks. During the consultation
with the program, questions are chosen by the user and answered at his or her
discretion, and the state of belief in any parameter can be requested at any time.
The odds of replicating the study, then, can be viewed at any point during evidence
collection.

The User Interface: REFEREE was initially run entirely on the SUMEX resource. Mr.
Chavez reimplemented the program on a stand-alone workstation, the Xerox 1186 in
the KEE commercial expert system shell. The availability of bit-mapped screens
made us more sensitive to issues of the user interface, but the shell could not deal
easily with the uncertainty inherent in our domain. Mr. Chavez then ported the
system to a Texas Instrument Explorer work-station, for which he designed an
entirely new knowledge engineering shell which integrated EMYCIN and _ influence
diagrams. It was apparent, however, that to accommodate the multiple interface
needs of our potential user community, we needed a graphics environment that
would allow frequent changes and customization. Thus, we turned to a final
environment custom-made for influence-diagram-based expert systems. The KNET
system, also developed by Mr. Chavez, separates the inferencing capabilities and
graphical manipulation of the knowledge base into MPW Object Pascal from the
textual part of the knowledge base and the the evidence collection in HyperCard.
This system runs on the Macintosh II with 4 MB of RAM.

The program code is now entirely independent of the knowledge required for reading
papers. REFEREE has a new interface that is intuitive and consistent. There is an
innovative consultation mode in which questions are presented in free-format menus.
The dialogues are mixed-initiative and of mixed levels, allowing the user such
options as requesting more detailed questions or cutting off apparently fruitless lines
of questioning. With the new REFEREE prototype, the user interacts with the

199 E. H. Shortiliffe
REFEREE Project 5P41-RROO785-15

machine using a mouse-pointing device Finally, the screen enables the user to orient
himself at all times, obviating the need for special commands to help the user
"navigate" through the knowledge base. Our expert recently provided the best
indication of the usability of this new system. After only a brief introduction to the
new machine and interface, he was able - for the first time - to run an entire
consultation by himself.

Current Status: At this point, REFEREE is a prototype that enables the clinician to
read clinical trials more critically. A number ot computational issues remain, such as
the optimal representation of Dr. Brown's knowledge in our current formalism.
Furthermore, REFEREE represents only the first step in a larger research plan, the
automation of knowledge acquisition (see section on Research Plans, below).
Current work in the restricted domain of clinical trials will, we hope, illustrate general
principles in the design of decision makers that gather expertise from written text
and multiple knowledge sources.

D. Relevant Publications

1. Haggerty, J: REFEREE and RULECRITIC: Two prototypes for assessing
the quality of a medical paper. REPORT KSL-84-49. Master's Thesis,
Stanford University, May 1984.

2. *Chavez, R. Martin and Cooper, G. F.: KNET: Integration Hypermedia and
Normative Bayesian Modeling. REPORT KSL. Stanford University, March
1988.

3. “Lehmann, H. Knowledge Acquisition for Probabilistic Expert Systems.
Submitted to Symposium on Computer Applications in Medical Care,
1988.

E, Funding Support

REFEREE currently receives only a small amount of funding. Most of the research is
performed in time contributed by the researchers to this project.

Title: Knowledge-Based Systems Research

Pl: Edward A. Feigenbaum

Agency: Defense Advanced Projects Research Agency

Grant identification number: NOO039-86-0033

Total award period and amount: 10/1/85 - 9/30/88 $4,130,230
(direct and indirect)

Current award period and amount: 10/1/87 - 9/30/88 $1,467,300

(direct and indirect)
REFEREE component is $27,706, or 1.9 % of grant total.

i. INTERACTIONS WITH THE SUMEX-AIM RESOURCE

A. Medical Collaborations
Dr. Brown and Dr. Feldman of the Stanford University School of Medicine are actively

involved in the REFEREE project and are the primary domain experts and critics for
this project.

E. H. Shortliffe 200