Therapeutic Area Standards: What are they really?

The fifth reauthorization of the Prescription Drug User Fee Act (PDUFA V) gave rise to the goals that FDA agrees to meet in exchange for the user fees it collects (PDUFA V "goals letter"). Under section XII, part E of the goals letter, FDA agrees to the following:

Clinical Terminology Standards:Using a public process that allows for stakeholder input, FDA shall develop standardized clinical data terminology through open standards development organizations (i.e., the Clinical Data Interchange Standards Consortium (CDISC)) with the goal of completing clinical data terminology and detailed implementation guides by FY 2017.  

1. FDA shall develop a project plan for distinct therapeutic indications, prioritizing clinical terminology standards development within and across review divisions. FDA shall publish a proposed project plan for stakeholder review and comment by June 30, 2013. FDA shall update and publish its project plan annually. 

This section of the goals letter has given risen to the development of so-called "Therapeutic Area (TA) Standards." I was involved in numerous activities associated with TA standards development. I found there exists a substantial amount of confusion or lack of clarity regarding what these are and how they should be managed. I decided to write this post to explore TA standards from a data standards and regulatory review policy perspective that hopefully provides useful insight in what these standards are and how best to manage them in the future. 

Standard vs. Data Standard

When discussing TA standards, it’s useful to draw a distinction between a standard and a data standard. A standard is defined in dictionary.com as “something considered by an authority or by general consent as a basis of comparison; an approved model.”  There are many different kinds of standards: manufacturing standards, measurement standards, data standards, etc.  To understand the distinction, consider a ruler. The ruler can be marked in inches or centimeters. Which ruler is used to measure length depends on the measurement standard that has been selected for the task. Once a measurement standard is selected, then the data standard provides a consistent approach to document and/or share the measurement. If the measurement standard is inches, and the measurement is 10 inches, then the data standard describes whether it’s 10”, 10 in, or 10 inches. This distinction is important when considering TA standards. How to represent a measurement (i.e. an observation) requires two decisions: a business decision (what to measure, which measurement standard to use), followed by a data standards decision (how to standardize the representation of the measurement, which data standard(s) to use).

What is a Therapeutic Area Standard?

I find there is no widely established “standard definition” for a TA standard. One working (perhaps prevailing?) definition is that a TA standard is a data standard for a therapeutic area or indication. I argue that a TA standard is not a data standard. Let’s examine this definition more closely.

Consider a clinical observation, specifically a clinical laboratory test:  glycosylated hemoglobin  (HbA1C). The standardization of HbA1C data is straightforward. CDISC provides controlled terminology for the HbA1C lab test (represented by the NCI Enterprise Vocabulary Services code C64849). The CDISC SDTM Implementation Guide describes how to represent lab test data (which includes HbA1C data) using the LB domain. The result is a numeric value, expressed as a percentage of the total hemoglobin in blood. Anyone conducting a clinical trial that includes the collection of HbA1C need only look at the SDTM IG and CDISC controlled terminology to understand how to standardize this information. No additional data standards are needed.

Consider now a single therapeutic area: Diabetes Mellitus. Let’s assume that, for the purpose of determining efficacy of a new diabetes drug, only one outcome measure is necessary:  the HbA1C.

So what does a Diabetes Mellitus TA Standard then look like? What are we “standardizing” that isn’t already standardized?

One can envision a separate Diabetes TA Standards document that says, “if you’re studying a new drug to treat diabetes, you should collect HbA1C and here is how you should represent HbA1C data using these existing standards: SDTM + CDISC controlled terminology.” For this document to be truly useful, an independent scientific and/or regulatory body should first decide what design features and clinical observations are relevant for diabetes studies. This can be described as a “good clinical research practice guideline” for diabetes. One could consider this a standard but it is not a data standard. Such a guideline is analogous to a manufacturing or building standard. Just as a builder might say: “A hurricane-resistant building must/should contain these materials: ….,” a clinical researcher would say: “A good diabetes study must/should contain HbA1C testing.” 

FDA publishes such guidelines. These are called indication-specific guidances, which help sponsors design their development programs, including clinical trials to support U.S. approval of new drugs for a given indication. I was involved in the development of a standard template for these guidances so that there is consistency in the content and presentation across therapeutic areas. Other organizations may publish similar guidelines: professional societies, other government agencies (e.g. NIH), consortia, etc.

In this simple example, a researcher would only need to read the clinical research guideline for diabetes, and understand how to represent HbA1C data using existing exchange and terminology standards. No additional documentation is necessary. A Diabetes TA user guide is not needed. The “standard” for diabetes trial is the clinical research guideline itself and the existing data standards.

Of course therapeutic areas are much more complicated than this. Each TA has multiple relevant clinical observations, and any given observations may have additional metadata and qualifiers needed to interpret the observation. In this setting a TA “user guide” is useful to demonstrate how to represent all TA-relevant data and meta-data using existing data standards. But the user guide itself is not a data standard. The data standards are the exchange and terminology standards that the user guide references.

So an alternative definition for a TA Standard is a best practice guideline for conducting clinical trials for a specific therapeutic area, with an accompanying illustration (e.g. “user guide”) on how to use existing data standards (exchange and terminology standards) for that TA. If FDA generates the guideline, then it would be an indication-specific guidance, and any available user guide would ideally be incorporated by reference to the guidance.

An analogy would be a best practice specification for designing a kitchen. The kitchen “standard” would say: it must have cabinets, a refrigerator, a sink and faucet, and a stove and oven. It may have a garbage disposal, dishwasher, and trash compactor. The kitchen TA user guide might say, “this is what your kitchen would look like if you use standard Ikea cabinets and General Electric appliances.”

We should all therefore agree that a TA standard is not a data standard. A TA Standard is a use case for existing exchange and terminology standards. The data standard is the exchange and terminology standards that are used to standardize TA-specific data. A TA Standard is a use case for existing exchange and terminology standards.The FDA publishes a Data Standards Catalog that lists the data standards the Agency supports for various use cases. Because TA standards are not data standards, I do not think they belong in the FDA Data Standards Catalog as new data standards that FDA supports.  FDA does need a new approach to convey that the TA use cases are adequately supported by the data standards listed in the catalog. 

So what about the user guides? We should recognize that a TA standard has two components:

1. A clinical research ‘best research practice guideline’ or standard (i.e. the data requirements)
2. A description of how available data exchange and terminology standards can represent the TA data requirements. A user guide may be useful (but not necessary) to illustrate how this is done.

In the trivial example described here, there is really no need for a separate Diabetes TA user guide because it is clear how to represent HbA1C information using existing data standards. The problem arises when the data requirements are complex and the existing data standards do not provide a clear and unambiguous representation of the clinical data. Then, a user guide is helpful and necessary.

For a TA standard to be effective for FDA, the Agency needs to ensure that its regulatory data needs are met. FDA needs a process to ensure that:

1. the TA best practice research guideline is accurate (the TA-specific data requirements; in CDER this is largely a function of the Office of New Drugs(OND)). Ideally this is captured in an indication-specific guidance, and
2. data standards exist (or have been adequately modified) to represent the data requirements in a standard format (in CDER this is likely a collaboration between OND, the Office of Strategic Programs (OSP), the Office of Translational Sciences (OTS) and data standards development organization(s)).

Mapping Terminologies using RDF

I received an email this morning from a colleague describing how RDF could be used to map local terms in a company's information system to standard terms for regulatory submission. It reminded me of a recent conversation on terminology mapping.

Just before leaving FDA, I was involved in a lengthy conversation about the challenges in the post-marketing world in mapping adverse events to MedDRA. The FDA expends a tremendous amount of resources coding post-marketing adverse event reports using MedDRA. MedDRA is the terminology adopted for this use case by the International Conference on Harmonisation (ICH), of which FDA is a member.

The problem is magnified because Electronic Health Record systems in the U.S. don't use MedDRA. The Office of the National Coordinator for Health Information Technology has adopted SNOMED CT and ICD-9 for medical problems/conditions (which include adverse events; see my recent post on this topic).

Needless to say, the FDA could use a mapping from SNOMED CT and ICD-9 to MedDRA. This is not an easy task, but assuming such a mapping existed, how could one implement it easily? Here the RDF provides a solution.

First, the terminologies must exist in RDF format. I recently came across this web site: the NCBO Bioportal, which makes available common medical terminology as ontologies. I have not evaluated it thoroughly, but it certainly looks promising.

Then comes the hard part...identifying concepts across terminologies that are the same.

Then comes the easy part... making links across ontologies using the owl:sameAs property. Here's an example. Let's assume the terminologies already assert the following in RDF:

MedDRA

meddra:10027599 rdf:type meddra:MedDRAConcept.

meddra:10027599 rdfs:label “Migraine”.

SNOMED CT

snomed:37796009 rdf:type snomed:SNOMEDConcept.

snomed:37796009 rdfs:label “Migraine (Disorder)”.

One asserts the following triple in the database:

           meddra:10027599 owl:sameAs snomed:37796009.

Then as Medwatch reports roll in from EHRs with an Adverse Event coded in SNOMED CT, one simply loads and stores the report with the SNOMED code in the knowledgebase. Any reviewer or analyst that queries the knowledgebase using the MedDRA term will automatically retrieve all the reports with the SNOMED code because the system treats them the same.

A benefit of this approach is that one doesn't have to map the entire terminology. It can be done incrementally. A large percentage of reports refer to a relatively small number of concepts. Those can be mapped first, leaving a small manual mapping process for the relatively less common terms as they come in. This would be huge improvement over today's process.

Another benefit is that one could leverage other organizations' mappings. If those are posted on the web in RDF, they can easily be imported and used. One would need additional metadata, such as provenance information, to help determine whether the mapping is reliable. We all do this now manually pretty routinely when evaluating information on the web. I'm more likely to trust a news report from www.cnn.com than from nationalenquirer.com, for example. An organization could develop a list of "trusted sources" for mapping information, or could conduct multiple searches, using different mappings from different sources, to see how they affect the search results.

The possibilities boggle the mind.

Observations, Assessments, and BRIDG

In a previous post, I discussed my view of how clinical data are generated and used (see Modeling Clinical Data). I discussed the differences between an observation and an assessment. This is an important distinction in clinical medicine.

An observation is a measure of the physical, physiological, or psychological state of an individual. It can be subjective (i.e. patient reported) or objective (reported by a provider or a device). They are reported without bias (as much as possible) and without interpretation by the observer. A serum potassium level of 5.5 mg/dL is an example of an observation. A pain score of 1, using a scale 0-3, is an example of a subjective observation. 

Observations are used as inputs to assessments. The assessment represents an assessor's interpretation or analysis of the observations. The result of an assessment is generally a medical condition and its properties (e.g. severity, change from last assessment). I would like to say "...is always a medical condition..." but I try never to say never or always, because invariably an exception emerges. Because an assessment includes an assessor's interpretation, bias can be a problem. The same set of observations can, and not infrequently, be interpreted differently by different assessors. Formal adjudication processes are sometimes put in place in clinical trials to minimize this type of bias. In health care, second opinions are quite commonly solicited to seek a better assessment; one that is closer to the truth. 

As a simple example, let's take the serum potassium of 5.5 mg/dL. To do a proper assessment, more information is needed. What is considered the normal range for the laboratory that conducted the test? (e.g. 3.0-4.5 mg/dL). Are there other clinical observations suggesting clinical hyperkalemia (e.g. EKG findings)? Is the patient on medications or does the patient have a known medical condition that can cause hyperkalemia (does the finding make sense)? Could this be due to hemolysis of the sample (this is a common cause of falsely elevated potassium measurement; it may require calling the lab and getting missing information about the biospecimen)? Could it be laboratory error (is a repeat measurement necessary)? Depending on the assessment, the assessor may determine that a new medical condition: hyperkalemia is indeed present, and may need to measure additional observations to determine its cause, and may need to order an intervention to bring the level down. In this example, the patient was recently placed on an angiotensin converting enzyme (ACE) inhibitor for the treatment of hypertension. ACE inhibitors are associated with hyperkalemia. The hyperkalemia was an adverse event related to ACE inhibitor use. 

An important clinical distinction between observation results and assessment results is that only assessment results get treated and tracked on a patient's problem list. As a medical student, it was ingrained into me "never treat the lab test or the x-ray; always treat the patient." 

Another important conclusion is that an Adverse Event is a Medical Condition; a special type of medical condition: one that is temporally associated with some medical intervention. In this example, the intervention was the administration of an ACE inhibitor. 

So when I look at BRIDG 4.0, I don't see the distinction between observations and assessments. In fact, the results of assessments are modeled as other observations. Specifically, a PerformedObservationResult is a generalization of AdverseEvent in the model. I believe this is incorrect. Furthermore,  the BRIDG definition of an Adverse Event is:

Any unfavorable and unintended sign, symptom, disease, or other medical occurrence with a temporal association with the use of a medical product, procedure or other therapy, or in conjunction with a research study, regardless of causal relationship. 

I disagree with this definition. A sign or symptom is an observation and, for the reasons I state here, is not an adverse event. I would modify the definition to read:

Any unfavorable and unintended disease or other medical condition with a temporal association with the use of a medical product, procedure or other therapy, or in conjunction with a research study, regardless of causal relationship.

There are other BRIDG classes that have this same issue (e.g. PerformedMedicalConditionResult). I don't attempt to provide a comprehensive list here. 

In discussions with the BRIDG modeling team, my understanding is that observation results and assessment results are handled the same way from a data management perspective, so the current modeling paradigm works from that respect. They propose developing a higher, conceptual presentation layer that draws the distinction between observations and assessments without necessarily changing the underlying model. I am not a modeler so I don't know if this is the right approach. I'm certainly willing to explore what a more subject-matter-expert-friendly presentation layer for BRIDG might look like and how that would address my concern. But I do have an underlying unease that these two very different concepts in clinical medicine: observations and assessments, can be collapsed in this way in an information model without some adverse consequences downstream from a computational perspective. 

I welcome other thoughts on this issue. 

Using the RDF to Generate (and Validate) an SDTM Demographics Domain

In a previous post, I discussed how the Resource Description Framework could be used to improve the management of biomedical knowledge. The discussion was theoretical. It was not possible to appreciate how RDF could be used in the short term to provide immediate value. RDF, or any other technology for that matter, will not be adopted or implemented unless it can solve real problems especially in the short term.

Here I discuss a simple, practical  application of the RDF to demonstrate how it can solve a real problem now. In another prior post, I described that the ideal role of the SDTM is a standard report from a database that is used for analysis. This example automates the creation of an SDTM demographics domain from an RDF database (called a triple store). First, I create a simple ontology of a study. Then, I use it to generate sample study data in RDF. Then I then store the ontology and the data in a simple RDF triple store database (a "knowledgebase"), and then I use SPARQL (the RDF query language) to query the database and generate an SDTM demographics domain. I discuss how this strategy can also be used to validate the data. I used a commercial, off-the-shelf (COTS) product: TopBraid Composer. The RDF file used for this exercise is available for download in Turtle format.

First I created a mini study ontology, containing only the classes and properties needed for this small exercise. You'll recognize many of the classes from BRIDG and the SDTM. I added a new class called SDTM_Domain which will contain a resource for each instance of an SDTM domain/dataset.

Mini Study Ontology - Classes

I then created the properties. First are the object properties (in blue)...those that relate two classes with each other, then the datatype properties (in green)...those that describe the data:

Mini Study Ontology - Properties

For example, the :conductedAt property relates the Study_Site class with the Country class. It enables asserting that Site 0001 was conducted in Germany, for example. These relationships are captured as Domain and Range information for each property using the standard rdfs:domain and rdfs:range properties. Another example is that the :age property has the :Subject class as its domain and the datatype xsd:integer as its value (i.e. range).

Using this ontology, I populated the triple store using dummy instance data for 10 subjects (you'll see the number 10 next to :Study_Subject in the diagram above to indicate the database has 10 study subject instances). Similarly I entered 4 instances of study sites, 2 instances of Sex (male, female), etc.

Finally, I created a single instance of an SDTM demographics domain, calling it :demographics:

Demographics domain Resource

As a property of this resource, I selected a standard property "spin:query" using the SPIN ontology (SPARQL Inference Notation) to embed the SPARQL query in a triple in the database. My understanding is that the instructions on how to generate the DM dataset (written in SPARQL) now becomes part of the knowledgebase. I need experts to confirm this is correct. Here is what the SPIN query looks like.

SPARQL Query to generate DM Domain

So when I run the query, I get the DM domain of the study as a report out of the knowledgebase.

DM Domain generated from RDF data

The fact that I was able to do this having very limited technical experience speaks, I think, to the power and simplicity of this technology. Another benefit is that the knowledgebase is perfectly functional to generate one domain now, but can easily be expanded iteratively over time to generate all domains from RDF study data. Another big advantage is that validation rules in RDF can be added to the knowledgebase to enable reasoning to identify validation errors. In fact, the existing triple in this knowledgebase,

             :age rdfs:range xsd:integer.

which basically says permissible values for ages are integers, represents an executable validation rule in the knowledgebase. If one enters a non-integer value for age, a reasoner can identify and surface the contradiction....which is basically a validation report. More complex validation rules can be constructed using the RDF, for example, for positive integers, cardinality constraints, value set constraints, etc.

As a next step, it would be useful to redo this example using the publicly available CDISC Foundational Standards in RDF specification. I was going to do this but haven't gotten around to it.

Longer term, these datasets should be "generate-able" from a BRIDG ontology. I believe an OWL representation of BRIDG will pave the way to generate even more useful reports for analysis (see another post: BRIDG as a Computable Ontology). Then one could populate the knowledgebase with, or incorporate by reference, more and more validation rules and even FDA policy statements expressed in the RDF and automate the ability not only to detect invalid data, but also data that doesn't conform to FDA study data submission policies as described in guidances and regulations.

In summary, the RDF provides the capability to implement practical solutions now that provide an alternate mechanism to automatically generate SDTM datasets using simple COTS tools, while at the same time provides the flexibility to increase the capabilities of the knowledgebase to support more and more solutions, such as:

1. Generate all SDTM datasets 
2. Validate the data 
3. Determine conformance with FDA submission policies
4. Generate more useful views/reports for analysis

Once this capability in the knowledgebase is fully developed, then one would not need to exchange the tabular reports, but can exchange the RDF data themselves. Or better yet, given the distributed or "linked data" capabilities of the semantic web, the recipient can simply be granted access to the RDF data on the web.

Analyses Across Multiple Studies

One of the main reasons for FDA building the Janus Clinical Trials Repository (CTR) is to facilitate analyses across multiple studies. This is particularly useful for safety analyses of rare events where pooling across studies is desirable to increase power.

I was at the FDA during the time when Vioxx went off the market. Although I was not directly involved in FDA's analysis of the available data for adverse myocardial ischemic events, I do remember the review team's struggles associated with both finding/accessing the data and getting them into an analyzable format. If the CTR had been available, the analyses could have been performed much more quickly.

As the CTR was being built to support this use case, unexpected challenges emerged. I would best characterize them as an inadequate degree of semantic interoperability with the SDTM; i.e. ambiguities in the way certain SDTM concepts are defined. The result is one cannot automate an "apples to apples" comparison of data across studies using existing SDTM variables.

One example is the Reference Start Date (RFSTDTC). The definition is (in SDTM v. 1.4):

Reference Start Date/time for the subject in ISO 8601 character format. Usually equivalent to date/time when subject was first exposed to study treatment. Required for all randomized subjects; will be null for all subjects who did not meet the milestone the date requires, such as screen failures or unassigned subjects.

The problem here is the word "usually." The net result is that RFSTDTC is sponsor-defined and can mean something different from study to study. Because study day (--DY) is derived from the RFSTDTC, it is also problematic. One cannot be sure that analyses that rely on the RFSTDTC mean the same thing across multiple studies without first doing a manual review of the study and verifying that RFSTDTC has a constant definition across the studies. Given that these studies sometimes span different applications and sponsors, it is not a trivial task.

What we really need is a standard definition for RFSTDTC that is constant across all studies. One approach is to take the possible different ways that sponsors are defining RFSTDTC and identify each as a separate concept, with a different name and concept identifier (e.g. date of randomization; start date of the first treatment epoch; date of first exposure to investigational drug). For some studies, these may all be the same, but not for others.

Semantica

As many of you know, I was with FDA for many years and I recently decided to leave U.S. federal government service. It was a difficult decision. I enjoyed my job at the Agency and especially enjoyed working with such a talented and dedicated group of professionals. I was fortunate to be at the right place at the right time. My latter years were marked by a rapidly increasing realization that the Agency was drowning in data and ill-equipped to handle the ever increasing deluge of data and information needed to make timely and effective regulatory decisions. From this realization, the Office of Computational Science (OCS) was born and I was lucky to be one of the founding members to help shepherd it in its infancy. In the short time it's been in existence, under the strong leadership of Dr. Lilliam Rosario, it has had a tremendous impact in modernizing the regulatory review process. I'm convinced the best is yet to come.

Some of you may not know, but I left the Agency briefly in 2009-2010 for about 14 months. During that time, I did some independent consulting. The experience was very rewarding, but there were so many unfinished projects back at FDA that interested me. I gladly returned to work on those projects (such as the Janus Clinical Trials Repository) when the OCS was launched. Now that these projects have become part of the fabric of the organization, I felt it was a good time to leave knowing that Computational Science at the Agency is alive and well.

So I'm returning to the consulting world. I have named my new company Semantica LLC in part because I believe better representation of and communication around the meaning, or semantics, of clinical trials data is the key to improving clinical research processes, from discovery to patient care. I look forward to remaining engaged and working with many of you on modernizing the clinical research enterprise from a medical informatics perspective. It's an exciting journey, one that I think will make a huge a difference in patient care and public health.

Quality Data in Clinical Trials

Everyone involved in planning, conducting, analyzing clinical trials wants quality data, but what do we mean by “quality” data in clinical trials? Here is a definition that I ran across that I particularly like. I don’t know who wrote this. If you know, please leave a comment:

In clinical trials, quality data support the objectives and analyses described in the protocol and accurately reflect a subject’s experience related to those objectives.

In the time I have worked helping to standardize clinical trial data, I continue to run across the misconception that standardizing the data somehow make it higher quality data. It is important to understand that

Standardized Data does NOT equal quality data. However,

Standardized data makes it easier to assess data quality.

Let’s explore these two statements further.

Quality Data is all about what is collected, available for analysis, reported, and from FDA’s perspective, submitted for regulatory review. What gets collected, analyzed, reported depends on good science and regulatory policy.

High Quality data are the result of many factors:

• Good protocol design
• Good study execution
• Good data collection and management processes
• Qualified research staff
• Others ….

Standardized data is all about how to structure the data to make it more useful. In the SDTM, it means standard domain names, file names, standard tabular structures, standard terminology, and standard data types.

Well Standardized data are the result of good:

• Data standards
• Understanding of the data standards
• Implementation of the data standards
• Conformance to the data standards 

Here are some examples of poorly standardized, high quality data:

AGE AGEU 25 YEARS 36 YEARS 54 YEARS 87 YEARS

AGE_IN_YEARS 42 19 24 66

AGE 24 years 18 months 47 years 38 years

Determining quality is a slow, manual process. It requires visual inspection of the data. One cannot write a reusable software program to automatically check that the age of the subject is reasonable.

Here are some examples of well standardized, but poor quality data:

AGE AGEU ? YEARS 259 YEARS 12 YEARS 999 YEARS

AGE AGEU ? YEARS 48 YEARS ? YEARS ? YEARS

AGE AGEU 54 YEARS 1,089 YEARS ? YEARS ? YEARS

Because the data are standardized, we can now automate quality checks. We can write a computer program to check data quality or us:

• If AGE is <missing> then generate quality alert report
• If [AGE > 120 and AGEU = YEARS] then generate quality alert report

Also, these quality checks can be built into the electronic data capture (EDC) instrument to improve the data collection processes that do affect quality.

We are surrounded by data quality checks at the point of data collection. We’ve become used to them. How often do you fill out an online form and try to submit it, only to be alerted that a particular data field is not filled out properly?

Data quality check on usps.com

We need more quality checks in clinical trials data collection processes.

So what about validation rules? There are two types: conformance rules (how well the data conform to the data standards) and quality checks (sometimes also called business rules).

Conformance rules depend on the standard. If the standard changes, the rules may change.

Quality checks are data standards independent. They make sense whether the data are standardized or not.  An age less than 0 is a data quality problem, whether we are dealing with legacy data or standardized data. Standardized data do however enable data quality checks. This is a huge benefit.

Conformance rules are best managed by the standards development organization that creates and maintains the standard. They understand what it means to be standards-conformant. The users of the data best maintain quality checks. They understand the implications if data are missing or of poor quality.

In the case of SDTM submissions to FDA, the current set ofvalidation rules contains a mixture of conformance rules and quality checks. It makes sense to move to a governance model where CDISC manages the conformance rules and FDA manages the quality checks.