Patient-Focused Drug Development: Collecting Comprehensive and Representative Input Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER) June 2020 Procedural Patient-Focused Drug Development: Collecting Comprehensive and Representative Input Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders Additional copies are available from: Office of Communications, Division of Drug Information Center for Drug Evaluation and Research Food and Drug Administration 10001 New Hampshire Ave., Hillandale Bldg., 4th Floor Silver Spring, MD 20993-0002 Phone: 855-543-3784 or 301-796-3400; Fax: 301-431-6353 Email: druginfo@fda.hhs.gov https://www.fda.gov/drugs/guidance-compliance-regulatory-information/guidances-drugs and/or Office of Communication, Outreach and Development Center for Biologics Evaluation and Research Food and Drug Administration 10903 New Hampshire Ave., Bldg. 71, Room 3128 Silver Spring, MD 20993-0002 Phone: 800-835-4709 or 240-402-8010 Email: ocod@fda.hhs.gov https://www.fda.gov/vaccines-blood-biologics/guidance-compliance-regulatory-information-biologics/biologics-guidances U.S. Department of Health and Human Services Food and Drug Administration Center for Drug Evaluation and Research (CDER) Center for Biologics Evaluation and Research (CBER) June 2020 Procedural Contains Nonbinding Recommendations TABLE OF CONTENTS INTRODUCTION .......................................................................................................... 1 Overview of Guidance 1 ................................................................................................. 3 Patient Experience Data ................................................................................................ 4 GENERAL CONSIDERATIONS FOR COLLECTING PATIENT EXPERIENCE DATA .............................................................................................................................. 7 Overview ......................................................................................................................... 7 Defining the Research Objectives and Questions ........................................................ 8 Who to Collect Information From ................................................................................ 9 Defining the Target Population ....................................................................................... 9 Determining Who Will Be Providing Patient Experience Data....................................... 9 Determining the Study Design and Research Setting ............................................... 10 Sampling Methods .......................................................................................................... 10 Representativeness ......................................................................................................... 15 Sample Size .................................................................................................................... 16 Constructing a Sampling Frame ................................................................................. 17 Additional Considerations ........................................................................................... 18 Missing Data/Non-Response.......................................................................................... 18 Leveraging Existing Data .............................................................................................. 19 OPERATIONALIZING AND STANDARDIZING DATA COLLECTION AND DATA MANAGEMENT ............................................................................................. 19 Locating Patients/Sites ................................................................................................. 19 Human Subjects Protection ......................................................................................... 19 Sampling Strategy ........................................................................................................ 20 Collecting Data ............................................................................................................. 20 Interviews, Focus Groups, and Facilitated Discussions ............................................... 20 Observational Methods .................................................................................................. 21 Documents...................................................................................................................... 21 Survey Instruments ......................................................................................................... 21 Audiovisual Materials .................................................................................................... 21 Social Media and Verified Patient Communities ........................................................... 22 Recording Information ................................................................................................ 25 Resolving Site/Field Issues ........................................................................................... 25 Data Management ........................................................................................................ 26 Data Standards ............................................................................................................. 26 Monitoring and Quality Assurance ............................................................................ 27 Storing Data .................................................................................................................. 27 i Contains Nonbinding Recommendations Confidentiality .............................................................................................................. 27 Data Analysis ................................................................................................................ 27 CONCLUSIONS........................................................................................................... 28 REFERENCES ............................................................................................................. 29 APPENDICES .............................................................................................................. 33 Appendix 1. Standards and Requirements Pertaining to Submission of Data .................. 33 Appendix 2. Glossary ............................................................................................................... 34 ii Contains Nonbinding Recommendations TABLE OF TABLES Table 1. Methodological Approaches for Collecting Patient Experience Data .............................. 5 Table 2. Types of Sampling, Examples and Some Potential Limitations ..................................... 12 Table 3. Common Methods for Gathering Patient Input Using Online Methods (Including Social Media) ........................................................................................................................................... 22 TABLE OF FIGURES Figure 1. General Steps for Conducting Studies About Patient Experience................................... 8 Figure 2. Factors to Consider To Achieve Sufficient Representation .......................................... 16 iii Contains Nonbinding Recommendations Patient-Focused Drug Development: Collecting Comprehensive and Representative Input Guidance for Industry, Food and Drug Administration Staff, and Other Stakeholders1 This guidance represents the current thinking of the Food and Drug Administration (FDA or Agency) on this topic. It does not establish any rights for any person and is not binding on FDA or the public. You can use an alternative approach if it satisfies the requirements of the applicable statutes and regulations. To discuss an alternative approach, contact the FDA staff responsible for this guidance as listed on the title page. INTRODUCTION This guidance is intended to address both a statutory requirement under the 21st Century Cures Act of 2016 (hereafter referred to as “Cures Act”) Section 3002 (c) and a commitment made under the Prescription Drug User Fee Act (PDUFA) VI (authorized under the FDA Reauthorization Act of 2017 (FDARA), Title I) to issue methodological guidance to support patient-focused drug development. The guidance to be issued under Cures Act section 3002(c)(1) shall address: “Methodological approaches that a person seeking to collect patient experience data for submission to, and proposed use by, the Secretary in regulatory decision making may use, that are relevant and objective and ensure that such data are accurate and representative of the intended population, including methods to collect meaningful patient input throughout the drug development process and methodological considerations for data collection, reporting, management, and analysis.” In addition to this Cures Act provision, FDA committed to publish a series of guidances under Section I.J of the PDUFA VI Reauthorization Performance Goals for “Enhancing the Incorporation of the Patient’s Voice in Drug Development and Decision-Making,” and the first of the series (referred to as “Guidance 1”) covers the same methodological topics as Cures Act sections 3002(c)(1), with the additional commitment to include standardized nomenclature and terminologies. The Cures Act2 defines the term “patient experience data” to include data that: 1 This guidance has been prepared by the Office of New Drugs and the Office of Translational Sciences in the Center for Drug Evaluation and Research (CDER), in cooperation with the Center for Biologics Evaluation and Research (CBER), at the Food and Drug Administration. 2 Patient experience data is defined for purposes of this guidance in Title III, Section 3001 of the 21st Century Cures Act, as amended by section 605 of the Food and Drug Administration Reauthorization Act (FDARA) of 2017. 1 Contains Nonbinding Recommendations (1) are collected by any persons (including patients, family members and caregivers of patients, patient advocacy organizations, disease research foundations, researchers and drug manufacturers); and (2) are intended to provide information about patients’ experiences with a disease or condition, including (A) the ‘impact (including physical and psychosocial impacts) of such disease or condition or a related therapy or clinical investigation; and (B) patient preferences with respect to treatment of the disease or condition. This expansive definition of “patient experience data” includes a wide range of opportunities for the collection of information that might be used to inform and provide a greater patient focus in medical product development. The range of patient experience data that would fit within the Cures Act statutory definition includes: patient registry data, natural history study data, patient focus group or meeting reports, patient survey data, clinical outcome assessment (COA) data collected during clinical trials, and elicited patient preference data. For the purposes of this guidance, some methodological considerations apply across the diversity of potential study objectives, but others will apply to only a subset of these. This guidance (Guidance 1) presents a general overview of methods and approaches for collecting patient experience data rather than focusing on methods for a specific, single purpose, e.g., to support collection of COA data or patient preference information. Specific issues related to COAs and patient preference information are addressed in other published FDA guidances.3 FDA is publishing a series of guidances intended to facilitate the advancement and use of systematic approaches to collect and use robust and meaningful patient and caregiver input that can better inform medical product development and regulatory decision-making. The topics and questions that each guidance document will address are described below. Guidance 1: From whom do you get input, and why? How do you collect the information? Guidance 1 discusses sampling methods that could be used when planning a study to collect patient input. It also provides a general overview of the relationship between potential research question(s) and method(s) when deciding from whom to get input (including defining the target population and development of the sampling strategy). Guidance 2: What do you ask, and why? How do you ask non-leading questions that are well- understood by a wide range of patients and other stakeholders? Guidance 2 will discuss methods for eliciting information from individuals identified in Guidance 1, gathering information about what aspects of symptoms, impacts of their disease, and 3 Issues related to COAs and patient preference information are addressed in the following guidance for industry: Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims and Patient Preference Information—Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling. 2 Contains Nonbinding Recommendations other issues are important to patients. It will discuss best practices in conducting qualitative research and reference-related resources. Guidance 3: How do you decide what to measure in a clinical trial and select or develop fit-for- purpose COAs ? Guidance 3 will address refining the list of concepts of interest important to patients for measurement. Given that not everything identified as important by patients, caregivers, and clinicians can be addressed by an investigational treatment or be measured, this guidance will address issues related to selecting what to measure in a medical product development program and identification or development of fit-for-purpose COAs to assess outcomes of importance to patients. Guidance 4: How do you incorporate a given COA tool or set of measures into a defined clinical study endpoint? How would you define a meaningful change in that endpoint? Guidance 4 will address topics related to incorporating COAs into endpoints for regulatory decision-making including COA-related endpoint development, defining meaningful within- patient score changes, and collection, analysis, interpretation, and submission of data. In general, FDA’s guidance documents do not establish legally enforceable responsibilities. Instead, guidances describe the Agency’s current thinking on a topic and should be viewed only as recommendations, unless specific regulatory or statutory requirements are cited. The use of the word should in Agency guidances means that something is suggested or recommended, but not required. Overview of Guidance 1 Section I.B discusses patient experience data, how they are collected, and the types of methods that may apply. Section II discusses general considerations for collecting patient experience data. It presents a logical sequence of questions for stakeholders to address to define the research questions of interest to them, the relevant study population, and considerations for the design of the study. Both quantitative and qualitative methods are presented to cover the range and diversity of study aims and approaches for collecting patient experience data. To satisfy the statutory requirement to address representative data collection in the context of both quantitative and qualitative methods, Guidance 1 also provides an overview of a range of potentially relevant sampling approaches of varying complexity. Section III discusses considerations related to collection and management of the data. Depending on the type of patient experience data and how the data will be used in medical product development, different content and formats may be appropriate for submission. When patient experience data are submitted to the Agency, the submitter should usually include the intended purpose of the patient experience data being submitted (i.e., how the data are intended for use in supporting medical product development and regulatory decision-making), and a study report 3 Contains Nonbinding Recommendations and protocol from the research study, as well as additional information including the mode of primary data capture (see Appendix 1). Finally, some of the topics covered in the main body of Guidance 1 are further discussed in a series of appendices, including standards and requirements related to submission of data (Appendix 1) and the glossary of terms (Appendix 2). Hypothetical case examples are also included. To illustrate important concepts presented in this guidance, they are also posted on the CDER PFDD webpage.4 Patient Experience Data Patient experience data include the experiences, perspectives, needs, and priorities of patients related to: • The signs and symptoms patients experience and how these signs and symptoms affect their day-to-day functioning and quality of life • The course of their disease over time, including the effect the disease has on patients’ day-to-day function and quality of life over time, and the changes that patients experience in their symptoms over time • Patients’ experience with the treatments for their disease: the symptoms and burdens related to treatment • Patients’ views on potential disease or treatment outcomes and how they weigh the importance of different possible outcomes • How patients view the impact of the disease, treatment, and outcomes, and their view of potential tradeoffs between disease outcomes and treatment benefits and risks The patient experience in a medical product development context incorporates their journey throughout the course of their disease or condition, including patient views, feelings, needs, actions, preferences, and interactions (e.g., clinical trials, home life, social life) with respect to their disease and its treatment (Wolf et al. 2014; McCarthy et al. 2016). There are different parts of the patient experience to collect and/or measure. These may include: • Impact of the disease and its treatment on the patient - Signs/symptoms of disease or condition - Chief complaints (most bothersome signs/symptoms) - Burden of living with or managing a disease or condition (including effect of the disease or condition on activities of daily living and functioning) 4https://www.fda.gov/drugs/development-approval-process-drugs/fda-patient-focused-drug-development-guidance- series-enhancing-incorporation-patients-voice-medical. 4 Contains Nonbinding Recommendations - Burden of treatment (including the effect of treatment on activities of daily living and functioning) - Burden of participating in clinical studies • Patients’ perspectives about potential and current treatments - Expectations of benefits - Tolerance for harms or risks - Acceptable tradeoffs of benefits and risks (i.e., patient preference) - Attitudes towards uncertainty • Views on unmet medical needs and available treatment options • Enhanced understanding of the natural history of the disease or condition, including progression, severity, and chronicity Patient experience data may be collected throughout medical product development, beginning at the launch of a discovery program, or may be independent of any specific medical product development program. Data can be collected in a variety of settings, including clinical trials, observational studies, advisory boards, public meetings (such as Patient-Focused Drug Development (PFDD) meetings), and other novel settings (e.g., social media, online patient communities). Depending on study or research goals and the research questions, qualitative, quantitative, or mixed methods may be appropriate for collection of patient experience data. Table 1 presents some basic distinguishing features of qualitative, quantitative, and mixed methods research approaches. As noted earlier, Section II provides an overview of these methodological approaches and discusses factors to consider when selecting an appropriate methodological approach. Table 1. Methodological Approaches for Collecting Patient Experience Data Research Approaches Study Qualitative Research Quantitative Research Mixed Methods Research Element Common Description, Numerical • Multiple objectives; research understanding and description, causal provide complex and objectives exploration/confirm explanation and fuller explanation and ation prediction understanding; understand multiple perspectives 5 Contains Nonbinding Recommendations Research Approaches Study Qualitative Research Quantitative Research Mixed Methods Research Element Common Attempt to Study behavior • Study multiple contexts, study understand under controlled perspectives or characteristics participant views, conditions; isolate conditions; study perspectives and the causal effect of multiple factors as they meanings of single variables operate together concepts; study groups and individuals in natural or controlled settings Data Qualitative data Quantitative data • Both qualitative and collection such as in-depth generated using quantitative data interviews, structured data participant collection observations, field instruments notes and open- ended questions Data analysis Use descriptive Identify statistical • Quantitative and analysis to identify relationships qualitative analysis, patterns, themes among variables used separately and in and holistic features combination of qualitative data Examples A group of patients A group of • A group of patients are are interviewed to patients are given a survey describe their surveyed about instrument with both experience with the their experience open-ended and closed- disease or condition with the disease or ended questions to condition with a describe their experience survey instrument with the disease or that uses closed- condition ended questions with distinct • A group of patients are response options first surveyed about to quantify their experience with the information disease or condition with a survey instrument and then are later interviewed to obtain additional information 6 Contains Nonbinding Recommendations GENERAL CONSIDERATIONS FOR COLLECTING PATIENT EXPERIENCE DATA Overview The selection of people from whom to collect input depends upon the specific questions and issues to be addressed (Figure 1). Thus, the selection process starts by considering the research question: what are the specific objectives to be addressed by collecting patient input? Are the objectives focused on understanding the most burdensome symptoms, the impact of current therapies, patients’ tolerance for risk, or the course of the disease over time? Each of these may require different approaches to patient selection and input collection. Some factors that are important to consider when selecting a research approach include: • Research goals or questions to be addressed • Target population and availability of people in that population • Most valuable information that should be generated through the study to achieve your research goals or answer your research questions • Feasibility of leveraging existing literature and data (See Section II.F.2) • Expected short-term and long-term impacts of the information you intend to gather through the study • Amount of time to conduct your studies • Study budget (including staffing, travel time, facilities costs, remuneration, data storage, management, and analysis) 7 Contains Nonbinding Recommendations Figure 1. General Steps for Conducting Studies About Patient Experience 1. Define the research objective(s) and questions 2. Determine the target patient population from whom to collect information 3. Determine the study design and research setting, including instruments 4. Determine which analyses are required to achieve the research objectives 5. Construct the study sample 6. Collect the data and perform data management tasks 7. Analyze and interpret the data 8. Report study results Defining the Research Objectives and Questions When formulating your research objectives, you should be specific. A broader research goal can be outlined by a set of specific research objectives, each reflecting a component of the broader goal. For example, the broader goal may be to evaluate how a treatment affects a particular symptom as well as the burden the administration of that treatment places on the patient. In this example each of these (the impact on the symptom, the burden on the patient) may be individual objectives of the research. Your research questions will further refine the research objectives into answerable questions. Your research objectives and questions should inform which methodological approaches you use in your research. You should examine previously conducted studies and other relevant research literature and consult subject matter experts (e.g., clinicians, social scientists, patients, advocates, caregivers) to help determine the most appropriate questions and to decide: • Which methods are better suited to meet your research goals and provide evidence to support your research questions • The design of study materials (e.g., study protocol, interview guides, coding dictionary; refer to Guidance 2 for more details). 8 Contains Nonbinding Recommendations Who to Collect Information From Defining the Target Population The group of patients whose experience you wish to learn about is the target population. It is important that the study enrollment criteria (inclusion and exclusion criteria) are carefully designed to properly select the target patient population. For example, if there is interest in understanding the perspective of patients newly diagnosed with a particular disease, defining this population carefully (e.g., what are the criteria for “newly diagnosed?”, what diagnostic criteria are necessary to ensure patients have the target condition?) is important. If the target population is intended to represent the broad range of patients with a particular disease, the enrollment criteria and sampling plan should ensure that study enrollment is not limited to patients with recent onset, or longer-duration of disease, but is a sample of the full spectrum of patients with the disease. Example: If you wish to understand the views and preferences of all individuals with Parkinson’s disease (PD) in the world, the target population could be defined as the set of all individuals who have been diagnosed with PD. If you are interested in a subset of PD patients, such as patients diagnosed within the last 5 years, the target population could be restricted accordingly. Note, however, that patient experience in a restricted target population might be different from that in a broader target population. For example, PD patients diagnosed within the last 5 years may have different views and preferences than the set of all PD patients, which includes those living with PD for over 10 years. Determining Who Will Be Providing Patient Experience Data For the collection of patient experience data, FDA recommends direct reports from patients, unless they are unable to reliably report on the concept of interest (e.g., young children, individuals with cognitive problems). The ability to provide self-report also depends on the methods used to elicit the input and the complexity of the concepts. Methods of data collection can be tailored to specific situations and patient groups, for example by eliciting patient experience through play and drawings from young children. When collection of direct patient experience is limited, valuable, but distinct, information may still be obtained from caregivers, patient advocates, clinicians, and others. In situations where direct report from the patient is limited (e.g., patients with cognitive limitations), alternatives to patient-reported outcome (PRO) measures may be needed. Such measures can provide useful information on observable patient experience, such as signs of disease or condition and functioning (FDA, 2015). However, research has repeatedly demonstrated lack of agreement, often markedly so, on the severity and frequency of signs and symptoms when patient-reported and non patient-reported measures are compared. Factors to consider when deciding whether and how patient self-report may be used include: • Level of cognitive development, function, or mental status 9 Contains Nonbinding Recommendations • Language skills • Numeracy skills (e.g., these are needed if a quantitative scale is to be assessed) • Health literacy (including basic literacy) • Health state • Co-morbidities Prior to study initiation, it is important to set the criteria for determining who the reporter is. For example: • To what extent can patients reliably and validly self-report? • What are the scenarios under which multiple reporters may be required? • How might this change over the duration of the study? The reporter should be recorded for each individual in your study and, if necessary, at each time point of the report. When there is the potential for patient report to be limited or compromised by any of the factors noted above, the researcher should consider engaging with subject matter experts (e.g., clinicians, social scientists, patients, advocates, caregivers) in the specific disease area of interest to determine the appropriateness of self-report in the target population. Determining the Study Design and Research Setting The design of the patient experience study is largely determined by the study objectives (as well as resources and setting) and includes determination of a number of key features including: • Study type (e.g., a clinical trial/study, observational study, survey study) • Methodological approach (e.g., qualitative, quantitative, or mixed research methods) • Patient selection/sampling method • Sample size • Methods for diagnosis determination (e.g. self-report, clinician report, other source) • Subgroups – whether special considerations are needed for subgroups of interest Sampling Methods An initial step is the approach to selecting the patient population that will participate in the study, referred to as the sampling scheme. As discussed previously, the research objective defines the appropriate population to be included (factors such as disease, stage of disease, treatment and treatment complications, disease status, disease complications), and selection of the appropriate population is essential to getting information relevant to addressing the research objectives. There are many sampling approaches, and they vary in complexity. Which approach you use may depend on your research objectives and resource constraints. You should engage with subject matter experts (e.g., statisticians, psychometricians) when determining which sampling methods to use. 10 Contains Nonbinding Recommendations Table 2 provides a non-exhaustive listing of possible sampling approaches that can be used to obtain patient experience data. They can be classified under two broad categories: probability and non-probability methods. Studies using probability sampling use some version of random sampling (see Table 2) to select from a larger population to create a sample, for example to select who will participate in a survey. When probability sampling is used, the patients who participate in the study and the results of the study are more likely to reflect those in the target population. Non-probability sampling uses non-random processes to select the study sample, such that the selected sample may not be representative of the target population. When choosing a method, it is important to consider resource or other constraints and carefully consider how the sample selected may impact the interpretation of the patient experience data. Probability sampling includes the following important components: • A well-defined target population • Listing of individuals within the target population • Use of a random number generator The listing of individuals in the target population is referred to as the sampling frame. Ideally, the sampling frame should enumerate all individuals in the target population. Where this is not feasible, the sampling frame should be representative of the target population. A random number generator can be used to randomly sample individuals from the sampling frame, which in principle produces a sample of patients whose experiences can be interpreted as being representative of the target population. Example: Suppose the target population consists of 100,000 patients with Parkinson’s Disease (PD) alive in the United States, and each individual is enumerated in a sampling frame with a label of 1 to 100,000. Using a random number generator, a sample of 2,000 patients is randomly selected from among the 100,000 patients, and their experiences are ascertained. Random sampling provides a mechanism for extending statements made about patient experience based on the individuals in the sample to the entire PD population. In practice, additional steps such as stratification may be needed to induce a sample having the desired characteristics. It is beyond the scope of this document to discuss sampling schemes in greater detail. However, more in-depth discussions with respect to advantages and disadvantages can be found in the literature (Rothenberg 1995; Heckathorn 1997; Korn and Graubard 1999; Fricker 2008; Groves et al. 2009; Levy and Lemeshow 2013; Valliant et al. 2013; Johnson and Christensen 2014; Johnson 2015). As noted, the appropriate sampling scheme is that which enables you to address your research objectives and can be implemented within the scope of your resource constraints. 11 Contains Nonbinding Recommendations Table 2. Types of Sampling, Examples and Some Potential Limitations Potential Types of Sampling Selection Strategy Examples Limitations Probability Sampling Simple random A sample drawn by a A simple random sample is • Can be expensive sampling procedure in which every taken from a population of or infeasible to member of the patients admitted to a conduct. population has an equal hospital in the first six chance of being selected. months of 2015. • Samples can fail to reflect the heterogeneity in the target population. Stratified random A sample drawn by Population of patients with • Requires the sampling dividing the population a disease are stratified by stratification factors into mutually exclusive disease subtype and sex, to be known for groups and then selecting and a random sample is each individual in a random sample from taken within each the population. within each group. combination (stratum). Multiplicity A sample drawn by first Approach used to • The initial sampling taking a probability oversample those with rare probability sample from the target diseases when conducting a sampling phase population followed by household survey by may not be feasible. drawing a sample from including a question about the set of individuals who people with a rare disease • Relies on the belong to the network of who previously lived in the initial respondents those initially sampled. household. to identify members in their network. Cluster sampling A sample drawn by A probability sample of • Often requires which clusters (i.e., a hospitals in a state is taken, information about collective type of unit from which a probability cluster size as that includes multiple sample of patients from selection elements, such as clinical each hospital is taken. probabilities can sites in different depend on such geographic areas) are information. randomly selected and either complete- or sub- • Units within sampling of individuals cluster tend to be within the selected homogeneous. clusters are taken. 12 Contains Nonbinding Recommendations Potential Types of Sampling Selection Strategy Examples Limitations Generalization of cluster CDC Medical Monitoring • Often requires Multistage sampling to include Project. information about probability multiple levels/stages of • Stage 1, a probability cluster size as sampling cluster sampling. sample of states. selection • Stage 2, a probability probabilities can sample of facilities within depend on such each sampled state. information. • State 3, a probability sample of HIV patients • Units within a from each sampled facility. cluster tend to be homogeneous. Non-Probability Sampling Clinical trials A sample that consists of Patients with iron • Trial results may patients who volunteer to deficiency anemia are not be generalizable participate in a clinical recruited to participate in a to the population of trial. clinical study that all iron deficiency compares the efficacy of an anemia patients for experimental therapy whom the therapy against a standard of care. is indicated. Convenience A sample drawn by Patients who can travel to • Study results may sampling including people who are attend PFDD meetings. not be generalizable available, volunteer, or to the target can be easily recruited. population. Purposive A sample drawn by Researcher is interested in • Study results may sampling which the researcher studying adult females with not be generalizable specifies the acne and selects their to the target characteristics of the sample based on these population. target population and characteristics (i.e., adult locates individuals with females with acne). those characteristics. 13 Contains Nonbinding Recommendations Potential Types of Sampling Selection Strategy Examples Limitations Quota sampling A non-random sample Researcher chooses their • Study results may drawn by which the sample to consist of 45% not be generalizable researcher determines the females and 55% males to to the target appropriate sample sizes maintain the correct population. or quotas for the groups proportions representative identified as important. of the target population. Respondent-driven Similar to snowball A convenience sample of • Requires long sampling sampling (see the next individuals with substance recruitment chain type in this list). The use disorders is recruited. and socially- chain of referrals is often Each individual in this networked longer than snowball initial sample is provided a population. sampling and, under fixed number of coupons • Study results may certain conditions, that he/she uses to recruit not be generalizable estimates can be others in his/her network. to the target generalizable to target The second set of population unless population. individuals recruited via assumptions, which coupons by the first set of are not verifiable, individuals are also given a are valid. fixed number of coupons that they use to recruit individuals in their network. This is repeated for a fixed number of cycles after which recruitment terminates. The coupons serve as financial incentives for the recruited to recruit others in the network. Snowball sampling A sample drawn by Patients with sickle cell • Study results may which each research disease participate in focus not be generalizable participant is asked to groups to discuss to the target identify other potential symptoms of the disease population. research participants. The and impacts of the initial sample of medications taken. Focus individuals is often group participants are obtained via non- asked to identify other probability sampling; people they know with subsequent samples are sickle cell disease who obtained by chained may be potential research referrals from the participants so study staff previous sample. can invite them to join the research study. 14 Contains Nonbinding Recommendations Potential Types of Sampling Selection Strategy Examples Limitations Web-based A sample drawn by using Researcher selects patients • Limited by pre- sampling the web as the contact from a web-panelist (e.g., registered panelists. mode; can involve online polling panel) to multiple sampling include in study. • Study results may strategies (e.g., not be generalizable systematic sampling, to the target multiplicity sampling, population. list-based, entertainment polls, un-restricted self- • Potential response selected surveys, bias* (e.g., volunteer (opt-in) panel). measurement error, misclassification). * Although response bias such as measurement error or misclassification is listed as a limitation under web-based sampling, it applies to all types of study. Representativeness When studying patient experience, it is important to obtain patient experience data that are not only relevant, objective, and accurate, but also representative of the target population. Sufficient representation may depend on the characteristics of the target population, the disease or condition under study, and the intended use of study results. In this document, the term representative or representativeness can be interpreted in the following ways. (1) A sample is representative of the target population if statements made about patient experience based on data from the sample of patients are generalizable to the target population. In principle, probability sampling schemes enable you to obtain such representative samples and often arise in the context of quantitative studies. However, if there are subgroups of patients from the target population that are not adequately represented in your study sample, your ability to generalize your research findings to the target population may be limited, even if you use a probability sampling scheme. (2) A sample is representative of the target population to the extent that patients in the study sample consist of individuals with various characteristics that approximate the heterogeneity of characteristics in the target population. If your sample does not reflect the broad range of patient characteristics of the target population, patient experience results may not be representative of the target population. Thus, statements made about patient experience based on data from the study are not necessarily generalizable to the target population. Whether this is acceptable depends on the research objectives. If your research objective is to generate hypotheses or tools to collect patient experience data, this extent of representativeness may be sufficient. It also may be possible to attain generalizability through weighting to account for the over-sampling (or under-sampling) of certain subpopulations if the sample is obtained with known sampling probabilities. Regardless of how individuals are selected for the study, it is important to ensure that patients in the study sample represent the target population, particularly with respect to the attributes that 15 Contains Nonbinding Recommendations are associated with the endpoints of interest. Figure 2 provides some guidance regarding factors to consider to achieve sufficient representation. Figure 2. Factors to Consider to Achieve Sufficient Representation Sample Size Important to the design of any study is sample size determination. Sample size estimates are driven by: • Research objectives • Type of outcomes/endpoints under consideration • Study design, including whether the study is quantitative or qualitative in nature • Planned methods of analysis Insufficient sample size may produce unreliable or imprecise results. FDA recommends that if the sample size is limited due to practical considerations (e.g., rare diseases), the research objectives and/or methods should be adjusted accordingly, and any limitations should be noted in the study report. Some practical considerations when determining sample size include: • Drop-out rates or non-response. Sample size may need to be adjusted upward to account for drop-outs or non-response. • Subpopulations. In cases where there is also interest in one or more subpopulations within the target population, sample size should be determined to ensure there is 16 Contains Nonbinding Recommendations sufficient information to also learn about those subpopulations. Subgroup5 analyses of interest should be pre-specified at the study design stage whenever possible. Care should be taken with the number of subgroups being proposed for analysis and inference. Subgroups of interest may be based on reporter type (e.g., patients versus primary caregivers) and/or socioeconomic, demographic, cultural, linguistic, clinical or other factors pertinent to the disease/condition of interest. For qualitative studies, sample size determination is often based on concept saturation, which means the point at which no new important concepts relevant to the research question are emerging from iterative rounds of interviews and the group of patients thus far recruited appears to be representative (Sandelowski 1995; Francis et al. 2010; Dworkin 2012). For quantitative studies, sample size calculations for different sampling types, study types and data types can be found in the literature (Thompson 1987; Chow et al. 2008; Levy and Lemeshow 2013). For complex designs where sample size formulae are intractable to obtain, simulation could be used. Constructing a Sampling Frame The existence of a sampling frame facilitates probability sampling. Without a sampling frame, it is difficult to randomly sample from the target population. To the extent that disease registries are inclusive and regularly updated, they may provide a natural sampling frame. The scope of registries may vary, with some defined at the regional level, sub-regional (e.g., state, province) level, or some level local to an organization such as a hospital or a chain of hospitals owned by a particular organization or part of a network. For an example of patient registries and related information, see https://ncats.nih.gov/radar. The following example discusses potential use of a clinician registry as a sampling frame. Example: In the United States, physician listings such as the AMA Masterfile or state licensing board files have the potential to be used to create a sampling frame for the target population in the sense that a sample of physicians from these sources may be used to elicit members of the target population, for example, patients who have been diagnosed with diabetes. Physicians who treat patients with diabetes may be sampled, and for each sampled physician, his or her patients are sampled. It is important to note that unless all relevant physicians are sampled, and all patients under the care of each sampled physician are identified, the resulting sampling frame may exhibit undercoverage in the sense that not every member of the target population is listed in the frame. 5 A subgroup is a part of the sample (e.g., in a clinical trial you look at the results only in the women in the trial, that is a subgroup analysis). A subpopulation is part of the whole population (e.g., if you only sample from women you are sampling from a subpopulation). We draw an inference from a subgroup to make generalizations about the subpopulation. 17 Contains Nonbinding Recommendations Undercoverage, mentioned in the example above, occurs when a proportion of members of the target population is not included in the sampling frame. Undercoverage may not be problematic if members excluded from the frame could be reasonably viewed as not being substantially different from those within the frame. Problems occur when we do not know if patients in the frame are similar to those not in the frame, for example where patient experience data is collected using only an online community. Attempts to augment the sample from other patient populations found using some of the sampling methods mentioned in the table can help reduce undercoverage. Additional Considerations Missing Data/Non-Response Missing data are common in most types of studies. With respect to representativeness, individuals who were selected for the study may decline to participate in the study (unit non- response), stop participating during the study (dropout), or may decline to answer some questions (item non-response) after consenting to participate in the study. Investigators should anticipate the types of missing data that are likely to occur given the study design, logistics, and the particular patient experience data that are being collected. The March 2019 Clinical Trials Transformation Initiative workshop on “Enhancing the Incorporation of Patient Perspectives in Clinical Trials”6 included discussion of how to collaboratively establish plans (in the study protocols) delineating strategies to minimize missing data, and where missingness cannot be avoided, to collect or determine the reasons for the missingness. Brick (2013), Calinescu et al. (2012), Levy and Lemeshow (2013), and Schouten et al. (2013) discuss design strategies for improving the response rate in the context of surveys. O'Neill and Temple (2012) and The National Research Council (2010) discuss design strategies for the prevention of missing data in clinical trials. To help better understand the extent and impact of missing data FDA recommends the following: • Provide a table summary of missing data; useful information includes frequencies, percentages, stratification by important subgroups, and reasons for missingness. • For longitudinal data, summarize missingness stratified by assessment visits or time points. In addition, methods for handling missing data in analysis should be addressed in the protocol. See The National Research Council (2010) and International Council on Harmonisation (ICH) E9(R1) for discussion of those methods. 6 https://www.ctti-clinicaltrials.org/news/recording-now-available-ctti-and-fda-hold-workshop-incorporating- patient-perspectives-clinical 18 Contains Nonbinding Recommendations Leveraging Existing Data FDA encourages collaboration among multiple stakeholders and the use of methods to combine and leverage existing data (e.g., national registry data, archival databases, published literature) to fit the specific needs of the research questions and study goals. It is important to note that if you decide to explore the use of existing data, you should demonstrate the representativeness, methodological rigor of the data collection method and data integrity as outlined in Section III of this guidance. OPERATIONALIZING AND STANDARDIZING DATA COLLECTION AND DATA MANAGEMENT Research to collect patient experience data that can be used in regulatory decision-making not only requires the study to be well-designed but also that the methods and standards for data collection and analysis meet Agency expectations for quality. Section III of this guidance provides an overview of standard approaches to consider regarding data collection and data management. You should standardize data collection activities and methods to support the best possible data quality. Locating Patients/Sites A critical step in the process of data collection is to identify the appropriate sample and/or sites (e.g., medical practice/center, academic institution, research consortia) to study. Including patients from diverse sites helps to provide a representative patient sample. FDA acknowledges that for some conditions (e.g., rare diseases) a limited number of sites may be available for clinical trials and additional research. Human Subjects Protection Research involving access to patient information or directly engaging with patients requires careful consideration of federal, state, and local laws, and institutional polices for the protection of human subjects. Because this guidance focuses on sampling methods for collecting patient experience data through a variety of research contexts (including, but not limited to, clinical trials, observational studies, advisory boards, public meetings) a full discussion of the laws that may apply to these collection methods is beyond its scope. Research subject to FDA regulations must satisfy the requirements for informed consent at 21 CFR part 50 and the Institutional Review Board (IRB) requirements at 21 CFR part 56.7,8 Research supported or conducted by the 7 Details on 21 CFR part 50 can be found at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=50 8 Details on 21 CFR part 56 can be found at https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=56 19 Contains Nonbinding Recommendations Department of Health and Human Services must satisfy the requirements at 45 CFR part 46.9 FDA recommends that researchers work with their IRBs and Health Insurance Portability and Accountability Act Privacy Boards to determine which laws may apply. FDA also recommends that research involving patient information be conducted in accordance with the principles of good clinical practice, including the ICH Guidelines. Sampling Strategy Of importance to the data collection process is the determination of a strategy for the sampling of patients or sites. Refer to Section II.D.1 on the different types of sampling. Collecting Data You should consider which data collection approach is most appropriate for their research objective. Data collection methods can include: • Interviews • Focus groups • Facilitated discussions at meetings • Observational methods • Documents (e.g., medical charts) • Survey instruments • Audiovisual materials • Social media and verified patient communities • Digital health technologies Each of these data collection methods generates different types of data, each of which has its own advantages and limitations. Additional details, including potential advantages and limitations of each method, are discussed below. Interviews, Focus Groups, and Facilitated Discussions Different interview types are used to collect patient experience data, including one-on-one interviews (semi-structured, structured, or unstructured) or group interviews (focus groups, facilitated discussions at patient meetings). The method of interviewing (e.g., in person, telephone, or video chat) may vary depending on the goals of the interview. For example, if visual cues are important for the context of the research objective, an appropriate data collection method may be face-to-face interviews either in-person or by video chat instead of telephone interviews. Further details and considerations regarding the different interview methods will be provided in future guidance in this series. 9 Details on 45 CFR part 46 can be found at https://www.hhs.gov/ohrp/regulations-and-policy/regulations/45-cfr- 46/index.html 20 Contains Nonbinding Recommendations Observational Methods Observation can be a tool to collect patient experience data, and can include the observation of the interactions of a participant in particular setting, activity, or behavior (Creswell 2013). Observations are helpful in situations for individuals who have barriers to communicating their thoughts orally or in writing. Additionally, observations of individuals or groups often can be made to supplement interviews (individual or group) by documenting cues from the environment and behaviors such as facial expressions, gestures, tone of voice, and other non-verbal indicators. Documents Various types of documents can be used to collect patient and/or caregiver input on burden of disease and treatment. These include: • Scientific publications • Public documents • Medical records, chart audits • Personal documents (e.g., patient journal) • Other printed materials (e.g., literature) • Graphics • Photo elicitation (participants take photographs or videotapes) • Archival records and physical artifacts. Survey Instruments Survey instruments generally consist of a standard set of questions or items administered in the same order to each participant (Johnson and Christensen, 2017). They can be administered in both observational studies and clinical trials, for example, to collect PRO measures that are associated with study treatments. Data can be collected by survey instruments throughout the study or at the end of the study (e.g., exit surveys). Refer to Guidance 2 for details regarding survey instruments. If data obtained via survey instruments are intended to be a study endpoint in a clinical trial, FDA recommends that stakeholders adopt good measurement principles. Refer to subsequent guidance documents in this series for discussion of factors to consider when administering survey instruments in clinical trials. Refer to Appendix 1. Standards and Requirements Pertaining to Submission of Data regarding standards and requirements pertaining data submission to FDA. Audiovisual Materials Audiovisual materials can be used to collect data in characterizing the patient experience. These may include recordings or photographs of individuals or groups, sounds (laughter or other vocalized expressions), email or discussion board messages with audiovisual attachments, and video chat/conferencing (e.g., Skype). 21 Contains Nonbinding Recommendations Steps to consider when collecting audiovisual materials as data include: • Standardizing data collection (e.g., information that can be derived from photographs can vary based on lighting and camera settings) • Obtaining the required permissions needed to use materials, including informed consent • Obtaining permission to extract information from web content, if necessary (e.g., request permission to join online forums and inquire whether there are restrictions on use of information for research purposes) Social Media and Verified Patient Communities Social media tools (e.g., medical community blogs, crowdsourcing, social media pages) may include information on patients’ perspectives regarding symptoms and impacts of a disease or condition. Targeted social media searches may be useful during the preliminary stages of a study to complement literature review findings, inform the development of research tools (e.g., qualitative study discussion guides), or as a supplement to traditional research approaches (e.g., literature, one-on-one interviews, focus groups, or expert opinion). Common methods for generating patient input using online methods, including social media, and their potential strengths and limitations are summarized in Table 3. Table 3. Common Methods for Gathering Patient Input Using Online Methods (Including Social Media) Social Media Qualitative Strengths Limitations Research Methodology All social methodologies • May allow access to • Self-selection bias (e.g., social hard-to-reach populations media participants may include • Cost and time saving for a narrow band of patients with researchers regard to clinical or • Relatively easy to demographic characteristics or implement willingness to participate in • Accurate and automatic social media platforms) capture of data • Regulations and laws • Participant convenience surrounding privacy and use of and comfort public and private data • Might result in greater self-disclosure 22 Contains Nonbinding Recommendations Social Media Qualitative Strengths Limitations Research Methodology Asynchronous online • Can be conducted using • Lack of visual cues focus groups or interviews email, discussion forums, • Underlying selection process (occur at different places, and other forms of social might be difficult, if not different times) media impossible, to quantify • Provide flexibility and • Representativeness might be convenience of logging questionable without strong in at time and place of assumptions participant’s choosing • Lack of time pressure and greater reflection Synchronous online focus • Data captured in real- • Scheduling can be difficult; groups or interviews time (synchronous) must find a common meeting (among younger • Can be conducted using time (for focus groups) participants; occur at the phone (SMS/text • Requires a fast internet different places, same time messages), chat methods, connection, webcam/ [for focus groups]) video messaging audio/video capabilities that • Interaction is often some participants may not have dynamic, immediate, readily available conversational (similar to • Technology-rich interface can every-day interactions) present more technical • Assessment of visual difficulties cues (through video or • Moderation can be difficult with emotions conveyed too many participants; through emoticon use) sometimes participants have • Less threatening trouble taking turns (for focus methodology for younger groups) participants • Faster typing speed gives participants an advantage, and these participants can dominate the conversation (for focus groups) • Increased likelihood of passive participation (e.g., a participant logging on and observing but not participating) • Groups with more than five participants require two moderators (for focus groups) • Self-selection bias (social media participants may include a narrow band of patients with regard to clinical or demographic characteristics) 23 Contains Nonbinding Recommendations Social Media Qualitative Strengths Limitations Research Methodology Verified patient • Generated through • Must have authorization to communities10 and social platforms such as online obtain identifiable information media data collection support groups and (e.g., Protected Health online educational groups Information (PHI)) • Groups include identifiable patients and identifiable reporters • Helpful for: - gathering information on health conditions (Prieto et al., 2014) - sharing treatments and experiences of care (McGregor et al., 2014) - recruiting research participants (O'Connor, Jackson, Goldsmith, & Skirton, 2014) Social media data • Generated through easily • Participants are unknown; collection (unelicited data) accessible platforms Respondent identification not • Low burden for people verifiable providing data • PHI not verifiable • Helpful for gathering • Underlying selection process is information on health difficult if not impossible to conditions (Prieto et al. quantify 2014) • Representativeness is highly questionable without strong assumptions • Predominantly retrospective data Although social media tools can provide useful data, limitations related to sampling should be considered. With most social media sources, there is no mechanism for verifying patient identity or clinical and demographic characteristics; you are limited to relying on patient self- identification and diagnosis, which can be inaccurate. Additionally, different demographic groups tend to use different types of social media. Based on this variability, it may be important 10 Communities that provide personal information to allow verification of personal characteristics, such as identity, diagnosis, or other patient characteristics. 24 Contains Nonbinding Recommendations for you to use different social media tools to gather information from the demographic group(s) you are targeting. Likewise, when submitting information for regulatory review, you should demonstrate how the data collection methods used to generate data addresses these limitations and to ensure rigor in methodology and data integrity. Concerns around the lack of ability to confirm patient characteristics (e.g., diagnosis) can be mitigated in various ways. For example, to have a verified patient, there should be enough information to indicate the existence of a specific patient, including age (or age category, e.g., adolescent, adult, elderly), sex, gender, initials, date of birth, name, or patient identification number. In the most ideal case, clinical information would also be available, by permission, through a central database (e.g., for patients who are members of patient advocacy group message boards, social networking groups, or medical community blogs). An identifiable reporter can be a family member, doctor, other health care practitioner, or other individual who has sufficient information to indicate that they are an identifiable person who has knowledge about the patient. Refer to Guidance 2 for considerations for use of social media to collect patient experience data. Recording Information You should develop, as a part of study protocols, written forms to collect patient experience data (e.g., a discussion guide or observational data collection form). A discussion guide or observation data collection form is a pre-designed form used to record information collected during an interview or observation (e.g., an interviewer may take notes on the discussion guide or observational protocol). Patient experience data can also be recorded through various forms, such as interview summaries and audio- and video-recordings. Resolving Site/Field Issues Standardized training should be provided to the members of the research team to improve consistency of research. The roles and responsibilities of the team should be outlined in the research protocol. This will help to prevent many site issues. FDA encourages stakeholders to also have a troubleshooting guide. Researchers should anticipate and address site/field issues that might arise during data collection. Some issues to consider are listed below (Creswell (2013): Access to patients/sites • Patients’ willingness to participate in research • Patient responsiveness • Appropriateness of a site • Building of trust and credibility at the field site • IRB unfamiliar with certain methodologies Interviews, focus groups, facilitated discussions • Mechanics of conducting interviews and discussions (unexpected participant behaviors, sensitive issues, inexperienced researchers) 25 Contains Nonbinding Recommendations Observational methods • Consistency in the role of observer • Mechanics of observing (remembering to take site notes) • Recording accurate quotes/notes • Managing information sufficiently at site Documents and audiovisual materials • Locating materials • Obtaining permission from the participant to use materials (e.g., audio/video-recorder) • Minimal noise disturbance • Best location for video recorder/camera Survey instruments • Paper-based administration: Quality control at the visit (e.g., administering correct version of the survey instrument, looking for non-response patterns such as not completing a particular section) • Electronic-based administration: Consistency in data monitoring procedures and follow-up (e.g., monitoring for timely completion and attrition) Ethical issues • Informed consent, if required • IRB oversight, if required • Conflicts of interest • Dishonest or hidden (secret) activities • Confidentiality and privacy considerations • Benefits and risks of research to participants Data Management Data management considerations should be addressed in the early stages of a research study. Before initiating data collection, you should formulate a data management plan (DMP) — a written document that describes the data you expect to acquire or generate during your research study; how you intend to manage, describe, analyze, and store said data; and what mechanisms you will use at the end of your study to preserve and share your data (Stanford University Libraries n.d.). Creating a written DMP helps formalize the data management process, identify potential weaknesses in the DMP, and provide a record of what you intend to do. Data Standards External stakeholders should use appropriate data standards when collecting, managing, and reporting patient experience data. When planning a study (including the design of case report 26 Contains Nonbinding Recommendations forms, data management systems, and data analysis plans), you should determine which FDA- supported standards to use. See Appendix 1. Standards and Requirements Pertaining to Submission of Data for some data standards resources.11 While compliance with these standards may not be required for studies other than those conducted to support a regulatory medical product application (e.g., an Investigational New Drug (IND), New Drug Application (NDA) or Biologics License Application (BLA)) or medical product labeling language, we encourage researchers to, at a minimum, bear these standards in mind, because patient experience data that are ultimately intended for use in clinical trials would be subject to the applicable standards. Monitoring and Quality Assurance FDA expects that external stakeholders will be responsible for monitoring the study, ensuring data integrity, and performing the data analysis. This includes assessing the quality of the design and performance of the studies. Storing Data External stakeholders should plan how to store their data in advance of starting their study. Researchers should decide how data will best be stored so that the data can be easily retrieved and protected from any type of damage or loss. The approach to data storage should reflect the type of data collected. Regarding duration of data retention, researchers should comply with their IRB and applicable regulations. Principles to consider regarding data storage and handling data include the following (Creswell 2013): • Create back-up copies of computer files • Use high-quality equipment for audio-recording information during interviews • Protect the anonymity of participants by de-identification • Create a data collection table or database to track and identify data • Maintain a list of types of data collected Confidentiality All personal participant data collected and processed for research should be managed by the research team with adequate precautions to ensure confidentiality of the data in accordance with applicable national and/or local laws and regulations on personal data protection. Data Analysis The approach taken to data analysis depends on the type of data, study conducted, and research questions and objectives. Sampling information should be taken in to consideration. Later 11 https://www.fda.gov/industry/fda-resources-data-standards/study-data-standards-resources 27 Contains Nonbinding Recommendations guidance in this series will provide more discussion of potential data analysis methods used with different types of patient experience data. CONCLUSIONS This document has provided an overview of methods to collect robust, meaningful, and sufficiently representative patient input to inform medical product development and regulatory decision-making. The proposed methods presented serve only as a basis for dialogue in the evolving and growing discipline of the science of patient input. If you are considering collecting patient experience data, FDA encourages you to have early interactions with FDA and obtain feedback from the relevant FDA review division on appropriate research design and any applicable regulatory requirements. 28 Contains Nonbinding Recommendations REFERENCES American National Standards Institute, 2016, ANSI/AAMI/ISO 14971:2007 (R2016): Medical devices - Application of risk management to medical devices. Brick, JM, 2013, Unit nonresponse and weighting adjustments: A critical review, Journal of Official Statistics, 29(3):329-353. Calinescu, M, B Schouten, and S Bhulai, 2012, Adaptive survey designs that minimize nonresponse and measurement risk, Statistics Netherlands. Chow, SC, H Wang, and J Shao, 2008, Sample Size Calculations in Clinical Research, Boca Raton, FL: CRC Press. Creswell, JW, 2013, Qualitative Inquiry and Research Design: Choosing Among Five Approaches, Los Angeles, CA: SAGE Publications. Dworkin, SL, 2012, Sample size policy for qualitative studies using in-depth interviews, Arch Sex Behav, 41(6):1319-1320. FDA, 2009, Patient-Reported Outcome Measures: Use in Medical Product Development to Support Labeling Claims, accessed March 15, 2018, https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Gui dances/UCM193282.pdf. FDA, 2013, Electronic Source Data in Clinical Investigations., accessed March 11, 2018, https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UC M328691. FDA, 2014a, Providing Regulatory Submissions in Electronic Format -- Submissions Under Section 745A(a) of the Federal Food, Drug, and Cosmetic Act, accessed March 11, 2018, https://www.fda.gov/downloads/drugs/guidances/ucm384686.pdf. FDA, 2014b, Providing Regulatory Submissions In Electronic Format — Standardized Study Data, accessed March 11, 2018, https://www.fda.gov/Drugs/GuidanceComplianceRegulatoryInformation/Guidances/UC M292334. FDA, 2015, Clinical Outcome Assessment (COA): Glossary of Terms., accessed March 11, 2018, https://www.fda.gov/drugs/developmentapprovalprocess/drugdevelopmenttoolsqualifica tionprogram/ucm370262.htm. FDA, 2016, Patient Preference Information - Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling, accessed August 22, 2019, https://www.fda.gov/regulatory-information/search-fda-guidance- documents/patient-preference-information-voluntary-submission-review-premarket- approval-applications. 29 Contains Nonbinding Recommendations FDA, 2017a, Expedited Programs for Serious Conditions: Drugs and Biologics, accessed, https://www.fda.gov/regulatory-information/search-fda-guidance-documents. FDA, 2017b, Providing Regulatory Submissions in Electronic Format — Certain Human Pharmaceutical Product Applications and Related Submissions Using the eCTD Specifications., accessed March 11, 2018, https://www.fda.gov/downloads/Drugs/GuidanceComplianceRegulatoryInformation/Gui dances/UCM333969.pdf. FDA, 2017c, Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices accessed August 22, 2019, https://www.fda.gov/regulatory-information/search- fda-guidance-documents/use-real-world-evidence-support-regulatory-decision-making- medical-devices. Francis, JJ, M Johnston, C Robertson, L Glidewell, V Entwistle, MP Eccles, and JM Grimshaw, 2010, What is an adequate sample size? Operationalising data saturation for theory- based interview studies, Psychol Health, 25(10):1229-1245. Fricker, R, 2008, Sampling Methods for Web and E-mail Surveys, The SAGE Handbook of Online Research Methods, Fielding, N., R. M. Lee and G. Blank, Thousand Oaks, CA: SAGE Publications, Inc., 195-216. Grajales, FJ, S Sheps, K Ho, H Novak-Lauscher, and G Eysenbach, 2014, Social media: A review and tutorial of applications in medicine and health care, J Med Internet Res, 16(2):e13. Groves, RM, FJ Fowler Jr, MP Couper, JM Lepkowski, E Singer, and R Tourangeau, 2009, Survey Methodology, Hoboken, NJ: John Wiley & Sons. Heckathorn, DD, 1997, Respondent-driven sampling: A new approach to the study of hidden populations, Social Problems, 44(2):174-199. International Conference on Harmonisation, 2016, Revision OF M4E Guideline On Enhancing the Format and Structure of Benefit-Risk Information in ICH: Efficacy - M4E(R2), accessed August 21, 2019, http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/CTD/M4E_R2_Efficacy/ M4E_R2__Step_4.pdf. Johnson, RB and L Christensen, 2014, Educational Research, Thousand Oaks, CA: SAGE Publications, Inc. Johnson, RB and L Christensen, 2017, Educational Research: Quantitative, Qualitative, and Mixed Approaches, Thousand Oaks, CA: SAGE Publications, Inc. Johnson, TP, 2015, Handbook of Health Survey Methods Hoboken, NJ: John Wiley & Sons. Korn, EL and BI Graubard, 1999, Analysis of Health Surveys, Hoboken, NJ: John Wiley & Sons. 30 Contains Nonbinding Recommendations Levy, PS and S Lemeshow, 2013, Sampling of Populations: Methods and Applications, Hoboken, NJ: John Wiley & Sons. McCarthy, S, P O’Raghallaigh, S Woodworth, YL Lim, LC Kenny, and F Adam, 2016, An integrated patient journey mapping tool for embedding quality in healthcare service reform, Journal of Decision Systems, 25(Suppl 1):354-368. Medical Device Innovation Consortium, 2015, Medical Device Innovation Consortium (MDIC) Patient Centered Benefit‐Risk Project Report: A Framework for Incorporating Information on Patient Preferences Regarding Benefit and Risk into Regulatory Assessments of New Medical Technology. National Research Council, 2010, The Prevention and Treatment of Missing Data in Clinical Trials, Washington, DC: The National Academies Press. https://doi.org/10.17226/12955. O'Neill, RT and R Temple, 2012, The prevention and treatment of missing data in clinical trials: An FDA perspective on the importance of dealing with it, Clinical Pharmacology & Therapeutics, 91:550-554. Patrick, DL, 2013, What Does "Patient-Centered Outcomes" Mean?, International Society for Pharmacoeconomics and Outcomes Research (ISPOR) 18th Annual Meeting, 18. Prieto, VM, S Matos, M Alvarez, F Cacheda, and JL Oliveira, 2014, Twitter: A good place to detect health conditions, PLoS One, 9(1):e86191. Rothenberg, RT, 1995, Sampling in social networks, Connections, 18:104-110. Sandelowski, M, 1995, Sample size in qualitative research, Res Nurs Health, 18(2):179-183. Schouten, B, M Calinescu, and A Luiten, 2013, Optimizing quality of response through adaptive survey designs, Survey Methodology, 39:29-58. Stanford University Libraries, n.d., Data best practices, accessed March 14, 2018, http://library.stanford.edu/research/data-management-services/data-best-practices. Thompson, SK, 1987, Sample-Size for Estimating Multinomial Proportions, American Statistician, 41:42-46. University of California San Francisco, 2017, Clinical Trial Protocol Development, accessed August 07, 2019, https://hub.ucsf.edu/protocol-development. US Congress, 1991, Public Law 102-73: The National Literacy Act of 1991. US Department of Health and Human Services, 2000, Healthy People 2010. Valliant, R, JA Dever, and F Kreuter, 2013, Practical Tools for Designing and Weighting Survey Samples, New York, NY: Springer. Wilson, H, E Dashiell-Aje, M Anatchkova, K Coyne, A Hareendran, NK Leidy, CA McHorney, and K Wyrwich, 2018, Beyond study participants: A framework for engaging patients in 31 Contains Nonbinding Recommendations the selection or development of clinical outcome assessments for evaluating the benefits of treatment in medical product development, Quality of Life Research, 27:5-16. Wolf, JA, V Niederhauser, D Marshburn, and SL LaVela, 2014, Defining patient experience. Patient Experience Journal, 1(1):7-19., Patient Experience Journal, 1:7-19. 32 Contains Nonbinding Recommendations APPENDICES Appendix 1. Standards and Requirements Pertaining to Submission of Data Many existing FDA regulations, guidances, requirements, and other standards for data submissions apply to patient experience data. These pertain to the capture/collection, transmission, processing, storage, archiving, retention, and submission of data from clinical studies conducted to support a regulatory medical product application (e.g., IND, NDA, BLA) or medical product labeling language. The following is a partial list of such regulations, guidance(s), standards, and requirements: • FDA forms and submission requirements https://www.fda.gov/drugs/developmentapprovalprocess/formssubmissionrequirements/ • The International Council on Harmonisation (ICH) Guidelines12, such as ICH Harmonised Guideline for Good Clinical Practice: E6(R2) and the Electronic Common Technical Document (eCTD) • 21 eCFR, Volumes 1 – 813 • Guidance for Industry on Providing Regulatory Submissions In Electronic Format — Standardized Study Data (FDA 2014b) • Guidance for Industry on Providing Regulatory Submissions in Electronic Format — Submissions Under Section 745A(a) of the Federal Food, Drug, and Cosmetic Act (FDA 2014a) • Guidance for Industry on Providing Regulatory Submissions in Electronic Format — Certain Human Pharmaceutical Product Applications and Related Submissions Using the eCTD Specifications (FDA 2017b) • Guidance for Industry on Electronic Source Data in Clinical Investigations (FDA 2013) • The FDA Data Standards Catalog. For current and more detailed information on study data standards resources, please see: https://www.fda.gov/industry/fda-resources-data-standards/study-data-standards-resources. 12 https://www.fda.gov/RegulatoryInformation/Guidances/default.htm 13 https://www.ecfr.gov/cgi-bin/text-idx?tpl=/ecfrbrowse/Title21/21tab_02.tpl 33 Contains Nonbinding Recommendations Appendix 2. Glossary This glossary defines terms that will be used in the series of methodological Patient-Focused Drug Development FDA guidance documents that are required by the Cures Act, and that are part of the commitments made by FDA under PDUFA VI. The goal of this glossary is to provide standardized nomenclature and terminologies related to patient-focused medical product development. As appropriate, definitions from existing federal resources (e.g., “BEST (Biomarkers, Endpoints, and Other Tools) Resource”)14 have been incorporated. External resources were also used to define terms and have been cited. Attribute: A feature or characteristic of a medical product — such as efficacy or effectiveness, safety, means of administration, duration of effect, or duration of use — that may affect benefit- risk considerations. Benefit: The favorable effects of a medical product. Types of benefits include clinical benefit (see clinical benefit). Benefits may also include important characteristics of the medical product, such as convenience (e.g., a more convenient dosing regimen or route of administration) that may lead to improved patient compliance, or benefits that affect those other than the patient. (Sources: International Council on Harmonisation (ICH) Guideline, Revision of M4E Guideline on Enhancing the Format and Structure of Benefit-Risk Information in ICH (Efficacy – M4E(R2) (International Conference on Harmonisation 2016); ANSI/AAMI/ ISO 14971: 2007/(R)2016 Medical Devices—Application of Risk Management to Medical Devices (American National Standards Institute 2016).) Benefit-Risk Assessment: Evaluation of the benefits and risks of a medical product and making a judgment as to whether the benefits outweigh the risks associated with specified conditions of use. Biomarker: A defined characteristic that is measured as an indicator of normal biological processes, pathogenic processes, or responses to an exposure or intervention, including therapeutic interventions. Molecular, histologic, radiographic, or physiologic characteristics are types of biomarkers. A biomarker is not an assessment of how an individual feels, functions, or survives. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Caregiver: A person who helps a patient with daily activities, health care, or any other activities that the patient is unable to perform because of illness or disability, and who understands the patient’s health-related needs. This person may or may not have decision- making authority for the patient and is not the patient’s healthcare provider. Caregiver Preference: A statement of the relative desirability or acceptability to caregivers of attributes by which alternative health interventions may differ. Clinical Benefit: A positive, clinically meaningful effect of an intervention, e.g., a positive effect on how an individual feels, functions, or survives. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) 14 “Available at https://www.ncbi.nlm.nih.gov/books/NBK338448/. 34 Contains Nonbinding Recommendations Clinical Outcome: An outcome that describes or reflects how an individual feels, functions, or survives. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Clinical Outcome Assessment: Assessment of a clinical outcome can be made through report by a clinician, a patient, a non-clinician observer, or through a performance-based assessment. Types of COAs include patient-reported outcome, clinician-reported outcome measures, observer-reported outcome, and performance outcome. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Clinical Relevance: The extent to which an endpoint can capture and measure an aspect of a potential clinical benefit (improvement in how the patient feels, functions, or survives) that is important from a clinical perspective and from the patient’s perspective. Clinician-Reported Outcome (ClinRO): A measurement based on a report that comes from a trained health-care professional after observation of a patient’s health condition. Most ClinRO measures involve a clinical judgment or interpretation of the observable signs, behaviors, or other manifestations related to a disease or condition. ClinRO measures cannot directly assess symptoms that are known only to the patient (e.g., pain intensity). (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Concept (also referred to as Concept of Interest): In a regulatory context, the concept is the aspect of an individual’s clinical, biological, physical, or functional state, or experience that the assessment is intended to capture (or reflect). (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Data Analysis Plan: A roadmap for how the data will be organized and analyzed, and how results will be presented. A data analysis plan should be established when planning a research study (i.e., before data collection begins). Among other things, the data analysis plan should describe: (a) the data to be collected; (b) the analyses to be conducted to address the research objectives, including assumptions required by said analyses; (c) data cleaning and management procedures; (d) data transformations, if applicable; and (e) how the study results will be presented (e.g., graphs, tables). Data Management Plan (DMP): A written document that describes the data you expect to acquire or generate during the course of your research study; how you intend to manage, describe, analyze, and store said data; and what mechanisms you will use at the end of your study to preserve and share your data. (Source: Stanford University Libraries n.d.(b), “Data Management Plans (DMPS)” available at https://library.stanford.edu/research/data- management-services/data-management-plans) Disease Burden: The impacts, direct and indirect, of the patient’s health condition that have a negative effect on his or her health, functioning, and overall well-being. Disease burden includes: the physical and physiologic impacts of the disease and its symptoms; co-morbidities; emotional and psychological effects of the disease, its management, or its prognosis; social impacts; effects on relationships; impacts on the patient’s ability to care for self and others; time 35 Contains Nonbinding Recommendations and financial impacts of the disease and its management; and considerations on the impacts on the patient’s family. Endpoint: A precisely defined variable intended to reflect an outcome of interest that is statistically analyzed to address a particular research question. A precise definition of an endpoint typically specifies the type of assessments made, the timing of those assessments, the assessment tools used, and possibly other details, as applicable, such as how multiple assessments within an individual are to be combined. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Fit-for-Purpose: A conclusion that the level of validation associated with a medical product development tool is sufficient to support its context of use. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Generalizability: The extent to which study findings can be reliably extended to the target population of interest. Health Literacy: The degree to which individuals have the capacity to obtain, process, and understand basic health information and services needed to make appropriate health decisions. (Source: US Department of Health and Human Services (2000)) Health literacy also includes numeracy skills—such as calculating cholesterol and blood sugar levels, measuring medication doses, and understanding nutrition labels — and knowledge of health topics. Literacy: A person's ability to read, write, speak, and compute and solve problems at levels necessary to: (a) function on the job and in society; (b) achieve one's goals; and (c) develop one's knowledge and potential. (Source: Public Law 102-73. The National Literacy Act of 1991 (US Congress 1991)) Methodologically Sound: Assurance that the methods and processes used to obtain and analyze patient experience data are rigorous, robust and adhere to scientifically established principles and best practices for method development or implementation. Evidence generated by methodologically sound methods and processes increases confidence that the results can be trusted, interpreted and support the intended regulatory uses. Mixed Methods Research: Research that uses both qualitative and quantitative research methods. See definitions for qualitative and quantitative research methods. Observer-Reported Outcome (ObsRO): A measurement based on a report of observable signs, events, or behaviors related to a patient’s health condition by someone other than that patient or a health professional. Generally, ObsROs are reported by a parent, caregiver, or someone who observes the patient in daily life. ObsROs are particularly useful for patients who cannot report for themselves (e.g., infants or individuals who are cognitively impaired). An ObsRO measure does not include medical judgment or interpretation. (Source: “BEST (Biomarkers, Endpoints and Other Tools) Resource”). Examples of ObsROs include a parent’s report of a child’s vomiting episodes or a report of wincing thought to be the result of pain in patients who are unable to report for themselves. 36 Contains Nonbinding Recommendations Patient: Any individual with or at risk of a specific health condition, whether or not the individual currently receives any therapy to prevent or treat that condition. Patient Advocate: An individual or group of individuals, who may or may not be part of the target population and who has a role in promoting an interest or cause to influence policy with respect to patients’ health or healthcare. Patient-Centered: See patient-focused Patient-Centered Outcome: An outcome that is important to patients’ survival, functioning, or feelings as identified or affirmed by patients themselves, or judged to be in patients’ best interest by providers and/or caregivers when patients cannot report for themselves. (Source: ISPOR Plenary, Patrick (2013)) Patient Engagement: Activities that involve patient stakeholders sharing their experiences, perspectives, needs, and priorities that help inform FDA’s public health mission. Such activities may include testimony at Advisory Committee meetings, submission to a regulations.gov public docket, meetings attended by patients, FDA, and other stakeholders, other correspondence with FDA, interactions through social media, and interactions with or information from patient representatives or patient advocates. Patient Experience Data: Defined in Title III, section 3001 of the Cures Act, as amended by section 605 of the Food and Drug Administration Reauthorization Act of 2017,15 and includes data that are collected by any persons and are intended to provide information about patients’ experiences with a disease or condition. Patient experience data can be interpreted as information that captures patients’ experiences, perspectives, needs, and priorities related to: 1) the symptoms of their condition and its natural history; 2) the impact of the conditions on their functioning and quality of life; 3) their experience with treatments; 4) input on which outcomes are important to them; 5) their preferences for outcomes and treatments; and 6) the relative importance of any issue as defined by patients. Patient-Focused (also referred to as patient-centered): Ensuring that patients’ experiences, perspectives, needs, and priorities are meaningfully incorporated into decisions and activities related to their health and well-being. 15For purposes of this section, the term ”patient experience data” includes data that (1) are collected by any persons (including patients, family members and caregivers of patients, patient advocacy organizations, disease research foundations, researchers and drug manufacturers); and (2) are intended to provide information about patients’ experiences with a disease or condition, including (A) the impact (including physical and psychosocial impacts) of such disease or condition, or a related therapy, on patients’ lives; and (B) patient preferences with respect to treatment of such disease or condition.” The definition is codified at section 569C(c)(4) of the Federal Food, Drug, and Cosmetic Act, and applies to section 3002 of the Cures Act, which directed FDA to issue certain guidance documents regarding the collection of patient experience data, see section 3002(b). 37 Contains Nonbinding Recommendations Patient-Focused Drug Development (PFDD) (also referred to as patient-focused medical product development): A systematic approach to help ensure that patients’ experiences, perspectives, needs, and priorities are captured and meaningfully incorporated into the development and evaluation of medical products throughout the medical product life cycle. Patient Input: Information that captures patients’ experiences, perspectives, needs, and priorities. See also patient experience data. Patient Partner: An individual patient, caregiver, or patient advocacy group that engages other stakeholders to ensure the patients’ wants, needs, and preferences are represented in activities related to medical product development and evaluation. (Wilson et al. 2018) Patient Perspective: A type of patient experience data that specifically relates to patients’ attitudes or points of view about their condition or management of their condition. Patient perspectives may include perceptions, goals, priorities, concerns, opinions, and preferences. Patient Preference: A statement of the relative desirability or acceptability to patients of specified alternatives or choices among outcomes or other attributes that differ among alternative health interventions. (Source: FDA Guidance for Industry: Patient Preference Information –Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling (FDA 2016)) Patient Preference Information (PPI): Assessments of the relative desirability or acceptability to patients of specified alternatives or choices among outcomes or other attributes that differ among alternative health interventions. The methods for generating PPI may be qualitative, quantitative, or mixed methods. (Source: FDA Guidance for Industry: Patient Preference Information –Voluntary Submission, Review in Premarket Approval Applications, Humanitarian Device Exemption Applications, and De Novo Requests, and Inclusion in Decision Summaries and Device Labeling (FDA 2016)) Patient-Provided Input: Patient experience data or other information that comes directly from patients. Patient-Reported Outcome (PRO): A measurement based on a report that comes directly from the patient (i.e., study subject) about the status of a patient's health condition without interpretation of the patient's response by a clinician or anyone else. A PRO can be measured by self-report or by interview, provided that the interviewer records only the patient's response. Symptoms or other unobservable concepts known only to the patient (e.g., pain severity or nausea) can only be measured by PRO measures. PROs can also assess the patient perspective on functioning or activities that may also be observable by others. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Patient Representative: An individual, who may or may not be part of the target population, who has direct experience with a disease or condition (e.g., a patient or caregiver) and can provide information about a patient’s experience with the disease or condition. 38 Contains Nonbinding Recommendations Performance Outcome (PerfO): A measurement based on a standardized task(s) performed by a patient that is administered and evaluated by an appropriately trained individual or is independently completed. PerfOs require patient cooperation and motivation. These include measures of gait speed (e.g., timed 25-foot walk test), memory recall (e.g., word recall test) or other cognitive testing (e.g., digit symbol substitution test). (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Qualitative Research Methods: Methods associated with the gathering, analysis, interpretation, and presentation of narrative information (e.g., spoken or written accounts of experiences, observations, and events). Qualitative research methods may also include direct observations (e.g., non-verbal communication and behaviors). Quantitative Research Methods: Methods associated with the gathering, analysis, interpretation, and presentation of numerical information. Real-World Data: Data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources. (Source: FDA Guidance on Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices (FDA 2017c)) Real-World Evidence: The clinical evidence regarding the usage and potential benefits or risks of a medical product derived from analysis of real-world data. (Source: FDA Guidance on Use of Real-World Evidence to Support Regulatory Decision-Making for Medical Devices (FDA 2017c)) Reporter: In research studies designed to collect patient experience data, the reporter is the individual, group of individuals, or entity providing patient experience data. Reporters may be patients, parents, sexual/romantic partners, caregivers, physicians, or other healthcare professionals. Selection of an appropriate reporter in a given research study will depend on the definition of the target population of interest. If a patient in the target population can be reasonably expected to reliably self-report, one would expect the patient herself/himself to be the reporter in that research study. Representativeness: Confidence that a sample from which evidence is generated is sufficiently similar to the intended population. In the context of patient experience data, representativeness includes the extent to which the elicited experiences, perspectives, needs, and priorities of the sample are sufficiently similar to those of the intended patient population. Research Protocol: A document that describes the background, rationale, objectives, design, methodology, statistical considerations, and organization of a clinical research project. (Source: University of California San Francisco (2017)) A research protocol guides the study and associated data collection and analysis in a productive and standardized manner. Risk: Risks are adverse events and other unfavorable effects associated with a medical product. Risks include drug interactions, risks identified in the non-clinical data, risks to those other than the patient (e.g., fetus, those preparing and administering the medical product), and risks based 39 Contains Nonbinding Recommendations on pharmacologic class or current knowledge of the product. Factors such as potential misuse, abuse, or diversion of the product may also be considered. (Source: International Conference on Harmonisation (ICH) Guidelines – Efficacy M4E(R2)) Risk Tolerance: The degree to which a patient would accept increased probability or severity of a harm in exchange for a specific expected benefit. (Source: Medical Device Innovation Consortium (2015)) Science of Patient Input: Methods and approaches of systematically obtaining, analyzing, and using information that captures patients’ experiences, perspectives, needs, and priorities in support of the development and evaluation of medical products. Social Media: Web-based tools that are used for communication. Social media may include blogs, microblogs, social networking sites, professional networking sites, thematic networking sites, wikis, mashups, collaborative filtering sites, media sharing sites, and others. (Source: Grajales et al. (2014)) Subgroup: A subset of the study population or study sample defined by specific baseline characteristics. For example, demographic subgroups are commonly defined by subject sex, race, and age. Surrogate Endpoint: A type of endpoint used in clinical trials as a substitute for a direct measure of how a patient feels, functions, or survives. A surrogate endpoint does not measure the clinical benefit of primary interest in and of itself but rather is expected to predict that clinical benefit or harm based on epidemiologic, therapeutic, pathophysiologic, or other scientific evidence. From a U.S. regulatory standpoint, surrogate endpoints and potential surrogate endpoints can be characterized by the level of clinical validation: (a) validated surrogate endpoints; (b) reasonably likely surrogate endpoints; and (c) candidate surrogate endpoints. (Source: “BEST (Biomarkers, EndpointS, and other Tools) Resource”) Target Population (also referred to as the target patient population, the underlying population or intended population): The group of individuals (patients) about whom one wishes to make an inference. Trade-off: The extent to which a change in the level of one or more attributes of a medical product is offset by a change in one or more other attributes of that product. (Source: Medical Device Innovation Consortium (2015)) Treatment Burden: The impacts of a specific treatment or treatment regimen that have a negative effect on the patient’s health, functioning, or overall well-being. Treatment burden includes: side effects, discomfort, uncertainty about treatment outcomes, dosing and route of administration, requirements, and financial impacts. Treatment Effect: The amount of change in a disease/condition, symptom, or function that results from a medical intervention (as compared with not receiving the intervention or receiving a different intervention). 40 Contains Nonbinding Recommendations Treatment Outcome: The benefits or harms to a patient who receives an intervention; the impact on a patient’s health, function, or well-being — or on a clinical indicator thereof — that is assumed to result from an intervention. Unmet Medical Need: A condition whose treatment or diagnosis is not addressed adequately by available therapy. An unmet medical need includes an immediate need for a defined population (e.g., to treat a serious condition with no or limited treatment) or a longer-term need for society (e.g., to address the development of resistance to antibacterial drugs). (Source: FDA Guidance for Industry. Expedited Programs for Serious Conditions––Drugs and Biologics (FDA 2017a)). 41