Rules in Study Protocols

When you read a study protocol, your are bombarded by rules. Some are explicitly stated. Many are implicit and must be teased out. Rules are extremely important in ensuring that protocols are conducted correctly. Rules are critical for a good study outcome. Unfortunately, we don't have a good way to standardize protocol rules. This makes it challenging to automate study conduct activities and quickly analyze if a study "followed the rules."

Let's dissect the components of a rule. A rule basically looks like this:

IF {condition A} THEN {Do B} ELSE {Do C}

the ELSE clause is optional and it is assumed to default to "do nothing" if condition A is not met. 
Rules can be complex :

IF {condition A} THEN {
(IF {condition E} THEN {Do F} ELSE {Do G} }

Evaluating a Rule is an Activity whose outcome is binary:  either the condition(s) is/are met ("true") or not met ("false"). One could argue for a 3rd category, not applicable, for cases where the reason to have a rule in the first place doesn't apply (e.g. when to conduct a pregnancy test in a male) 

In clinical studies, rules often depend on other activities. I call these prerequisite activities (PA). For example:
IF {migraine occurs} THEN {take study drug}
or a more precise way of expressing it:
IF {headache assessment = MIGRAINE} THEN {take study drug} 

In this case the prerequisite activity is a headache assessment and the condition is met when the headache assessment outcome indicates that a migraine is present. 

Regarding how prerequisite activities are evaluated, sometimes it is sufficient that the PA simply has begun (PA status = started) or completed (PA status = complete) or, more commonly, completed with a specific expected outcome (PA expected outcome = migraine). 

When looking at rules more closely, they can be expressed as start rules for other activities. Let's call these target activities (TA). 

Target Activity: Study Drug Administration_01
Start Rule: MigrainePresent_01 -- Prerequisite Activity:  HeadacheAssessment_01
                                                       PA Expected ActivityOutcome:  MIGRAINE

StudyDrugAdministration_01 is a planned activity that just sits there, waiting to be performed. As soon as the headache assessment is performed and whose outcome is a documented MIGRAINE, the rule outcome is set to TRUE and the target activity can begin. 

One can add qualifiers in the rule to describe exactly when the target activity is performed. for example a delay = 30 minutes means wait exactly thirty minutes after the condition is met before starting the activity. maxdelay = 60 minutes means wait no more than 60 minutes before starting the activity. mindelay = 30 minutes means wait a minimum of 30 minutes before starting the activity. 

I have tried this paradigm in multiple scenarios and so far it seems to work (randomization, eligibility determination, delayed start, assessments). 

In a future post, I'd like to explore how these rules can be expressed computationally using the RDF. 


Is Death an Adverse Event?

Throughout the course of a drug's marketing life cycle, it is critical that sponsors and regulatory agencies understand if a drug administration is associated with the death of the patient, but is death an Adverse Event? I argue that death is not an adverse event but instead may be the outcome of an AE. I think that it's important to draw this distinction to improve AE reports and automate death/causality analyses.

Here is my argument. First, let's review some definitions.

The U.S. Code of Federal Regulations (21 CFR 312.32) defines an adverse event as
  • any untoward medical occurrence associated with the use of a drug in humans, whether or not considered drug related. 
The bolded text is mine, as I want to consider how to interpret these terms.

Untoward is an adjective meaning unexpected, inappropriate, inconvenient. (Oxford Dictionary). Synonyms include inconvenient, unlucky unexpected, surprising, unusual.

A medical occurrence is more difficult to define. I take it to mean a medical condition: i.e. disease, disorder, injury, or transient physiological state that impairs or may impair health.

Taking all of this into consideration, my working definition of an Adverse Event is any unexpected medical condition (disease, disorder, injury, transient physiological state) that impairs or may impair health and emerges or worsens after a medical intervention (e.g. drug use).

Notice that I broaden the definition to medical interventions such as medical device use, or medical procedure, because AEs in those scenarios are equally relevant from a public health perspective.

With these definitions in mind,  I think one of the pitfalls in causality assessments is wrong use of the term adverse event. I believe Adverse Events are medical conditions. Medical conditions fluctuate over time. An AE can remain stable, improve, or resolve over time. It can also worsen to the point where the patient dies. All of these are potential outcomes of an AE.

Here is a hypothetical case of crazy conclusions that can emerge if our definitions are not precise. Imagine that one is developing software to automate AE causality assessments. 

Consider an 13 y/o male on chronic treatment with Drug X for seizures. Over the course of the year, he grew 4 inches and gained 25 lbs, yet his dose of anti-convulsant medications did not change to keep pace with his increased body mass. Towards the end of the year, he was having breakthrough seizures with a documented low therapeutic level of Drug X in his blood. A computer program may select the following concepts for further causality assessment:
Suspect Drug:  Drug X

Adverse Events:  Seizures, Weight Gain

A computer program would look at this and ask did Drug X cause weight gain and seizures in this case? On a superficial level, it is a fair question since the subject was on Drug X when those events happened.

But wait a minute. A human assessor immediately recognizes the frivolity of such a question in this case. Clearly the growth and weight gain are consistent with puberty-associated growth spurt. The problem is the dose of Drug X wasn't increased in response to the increased body mass. 

The fact that the causality question even arises is a misinterpretation of an Adverse Event. Applying our working definition of an AE, Seizure is not an Adverse Event in this case. It's in fact the Indication, a medical condition that is the target of a medical intervention (i.e. anticonvulsant therapy).  But, you say, it's also a medical condition that worsened with treatment. The key here is that it didn't unexpectedly worsen. A sub-therapeutic dose of an anticonvulsant can be expected to result in breakthrough seizures. 

How about weight gain? Again, this is not an AE. It's an observation, well technically it's an analysis of two (or more) observations, body weight, over time. Is it indicative of an AE? Not necessarily. In this case, it is the consequence of normal growth. Obesity, on the other hand, would be an AE depending on how it's defined. There is no evidence in this report of obesity.

So what is the causality assessment for this case?  The assessment cannot be done. There is no adverse event.

Here's another case:  A person takes Ambien, gets drowsy and drives a car and is involved in a motor vehicle accident.
Suspect Drug:  Ambien
Adverse Event: Motor Vehicle Accident (MVA)

Based on our definition, an MVA is not an Adverse Event since it is not a medical condtion. The AE here is  drowsiness. The MVA is a consequence/outcome of the AE.

So let's get back to death? It's commonly listed as an AE in safety reports. Is it a Medical Condition? No. Death marks the end of a complex physiologic process we call Life and it happens to all living organisms. Death (like an MVA, or a bad fall) is or may be a consequence/outcome of a Medical Condition. In the case of Death we call it the Cause of Death. The cause of death may or may not be an AE.

This reminds me of the clinical data lifecycle:  Observations lead to Assessments/Adjudications to diagnose/assess Medical Conditions (including AEs) which leads to Interventions to (treat, mitigate, cure, prevent) those conditions, which leads to more Observations.  In turn Medical Conditions have consequences/outcomes sometimes beyond our control, like Death (or Falls, or Motor Vehicle Accidents).

These information buckets are well established and distinct in clinical medicine. We save a lot of confusion and misinterpretation of clinical data when we classify them appropriately. We need to do a better job of distinguishing AEs from observations, and not confusing the consequence(s) of an AE from the AE itself. Recognizing this distinction is important to automate adverse event assessment, including those resulting in death.


Managing Study Workflow using the Resource Description Framework (RDF)

Imagine a study conduct tool where:

  1. One enters the result of an observation and the tool immediately lets you know what observation(s) to collect next
  2. The tool immediately performs a data quality check and creates an alert if there is a problem
  3. One enters the results of all the screening tests and the tool immediately tells you whether the subject is eligible to continue in the trial
  4. If the subject is not eligible, it tells you why he or she failed screening.
  5. In an adaptive trial design, it analyzes the observations in real time and informs you what protocol-specified modifications one can make in the subject's treatment plan
These are all possible today if we start representing study data including the protocol, using the RDF. 

In my last post, I spoke about a paper on this topic. I am now sharing that paper so those interested can take a look at it. I'm also posting the slides I presented at the annual PhUSE meeting in Edinburgh, Scotland (Oct 8-11, 2017), which was a very successful meeting by the way. A record number of attendees (695) participated in the conference.

I hope you will find it interesting. Please send me your comments. Thank you.

Paper: Managing Study Workflow using the Resource Description Framework

Slides: Managing Study Workflow


Eligibility Criteria, Screen Failures, and another RDF Success Story

It's T minus 6 weeks  (approximately) for the PhUSE 13th Annual Conference in Edinburgh, Scotland and I'm beaming with excitement. I'm involved in two study data projects using RDF and both are going very well. The first one, in collaboration with Tim Williams and an enthusiastic project team of PhUSE volunteers is called Clinical Trials Data in RDF, which among its various goals, will demonstrate how study data in RDF can be used to automatically generate highly standards conforming, submission quality SDTM datasets.

But it's the second paper that I want to discuss today. It's called "Managing Study Workflow Using the RDF." The paper is in pre-publication status so I can't share today, but I plan to post a copy here after the conference. I include the Abstract below.

In a nutshell, the paper describes how one can represent study activity start rules using SPIN (SPARQL Inference Notation), a type of RDF, to identify which study activity(-ies) are to be performed next based on what activities have already been performed. Well it turns out that the start rule for the Randomization activity in a typical double blind trial is in fact the Eligibility Criteria for the study. Here it is, in an executable form that, when combined with a standard COTS (commercial off the shelf) RDF inferencing engine can automatically determine eligibility. How cool is that?

A typical eligibility rule consists of multiple subrules all of which themselves must be TRUE for the overall rule to be true (e.g. AGE must be >=18 years AND RPR must be negative AND Pregnancy Test must be negative AND etc.); exclusion criteria can be negated and added as a subrule. The ontology also describes how to skip subrules that can logically be skipped (e.g. Pregnancy Test must be negative in a male subject). The end result is that identifying an Eligible Subject is automatic and performed simply by entering the screening test results in the knowledgebase. (Think of a knowledgebase as an RDF database).

Without going into the details (wait for the paper!), the rule points to all the screening activities that matter, checks each one for the expected outcome/result, and returns a TRUE or FALSE response if the conditions of the rule are or are not met. If the rule outcome is TRUE, the subject is eligible and the Randomization activity is enabled. If the rule is FALSE, then just the opposite. The paper describes the data from eight hypothetical subjects that were screened for a hypothetical study with just a few screening tests/criteria. The ontology correctly came up with the correct eligibility outcome for all eight.

But there is more....by adding a few more simple SPIN rules to the ontology, the inferencing engine can readily provide a list of all Screen Failures, and the tests that caused them to fail. It can also identify the tests that were logically skipped and therefore ignored for eligibility testing purposes. Do you want to determine which Screening Failure subjects received study medication? Another SPIN rule can do that too. The possibilities are quite exciting. It makes RDF, in my humble opinion, a strong candidate for representing clinical trial data during study conduct. No other standard that I know of supports this kind of automation "out of the box." in RDF, the model and the implementation of the model are the same!! And, once one is ready to submit, you press another button, and submission quality SDTM datasets are generated (which the first project I mentioned intends to demonstrate).

For more details, contact me, or wait until after the PhUSE meeting in October for the full paper.

A clinical study is fundamentally a collection of activities that are performed according to protocol-specified rules. It is not unusual for a single subject to undergo hundreds of study-related activities. Managing this workflow is a formidable challenge. The investigator must ensure that all activities are conducted at the right time and in the correct sequence, sometimes skipping activities that logically need not be done. It is not surprising that errors occur.

This paper explores the use of the Resource Description Framework (RDF) and related standards to automate the management of a study workflow. It describes how protocol information can be expressed in the RDF in a computable way, such that an information system can easily identify which activities have been performed, determine which activities should be performed next, and which can be logically skipped. The use of this approach has the potential to improve how studies are conducted, resulting in better compliance and better data.


Quality Data in Clinical Trials, Part 2

It's been two years since I wrote about quality data in clinical trials. As I re-read that post now, I agree with most of what I said, but it's time to update my thinking based on experience gained since then with study data validation processes.

I made the point that there are two types of validation rules: conformance rules (to data standards) and business rules, a.k.a. data quality checks. I had suggested that conformance rules are best managed by the standards development organization, the fact is that sponsors and FDA support multiple standards (SDTM, MedDRA, CDISC Terminology, WHO Drug Dictionary) so it's up to FDA to manage the collective set of conformance standards across the broad data standards landscape.

The division between conformance rules and business rules is still quite important. They serve different functions. Ensuring conformance to standards enable automation. Ensuring data quality enable meeting the study objectives. One can assess data quality on legacy data. It is a slow, manual process. Standardized data enable automated data quality checks than can more easily uncover serious data content issues that can impede analysis and review.

As a former FDA reviewer, and a big proponent of data standards, I can honestly say that FDA reviewers care very little about data standards issues. All they care about is that the data be sufficiently standardized so they can run standard automated analyses on the data. The analyses drive the level of standardization needed by the organization. These analyses include the automated data quality checks. One cannot determine if AGE < 0  (a data quality check), if AGE is called something else or is located in the wrong domain (conformance rule).

It's like driving a car. You want to get from point A to point B quickly (data quality issues), you don't really care what's under the hood (standards conformance issues). That is for mechanics (or data analysts) to worry about.

FDA now has a robust service to assess study "Data Fitness" (being described as data that are fit for use). Data Fitness combines both conformance and business rules. They are not split, and the reviewer is left to deal with data conformance issues, which they care little about as there is generally a manual work-around, along with the data quality issues, which is of most important to them and have the biggest impact on the review. Combining the two is a mistake. I believe Data Fitness as a concept should be retired and the service split into two: Standards Conformance, and Data Quality. The Data Quality assessment service should only be performed on data that have passed the minimum level of conformance needed by the organization. If a study fails conformance testing, it wasn't standardized properly and those errors need to be corrected. In the new era requiring the use of data standards, FDA reviewers should not be burdened with data that do not pass a minimum level of conformance validation.

Consider this hypothetical scenario as an example to drive home my point. FDA requires sponsors to submit a study report supporting the safety and effectiveness of a drug. The report should be submitted digitally using the PDF standard.  The report arrives and the file cannot be opened using the review tool (i.e. Acrobat) because of 10 errors in PDF implementation (not realistic in today's day and age, but possible nonetheless). Those 10 errors are provided in a validation report to the reviewer for action. The reviewer doesn't care about the technical details of implementing PDF. They want a file that opens and is readable within Acrobat. All can agree that the reviewer should not be burdened evaluating and managing standards non-conformance issues.

If you replace study report with study data, and PDF with SDTM, this scenario is exactly what is happening today. But somehow that practice remains acceptable. Why? Well because there are other "tools" (akin to a simple text editors in the document world) that allow reviewers to work with non-conformant data, albeit at much reduced efficiency. These "workarounds" for non-standard study data are all too prevalent and acceptable. With time this needs to change to take full advantage of standardized data for both Sponsor and FDA alike.

My future state for standardized study data submissions look like this: study data arrives, they undergo standards conformance validation using pass/fail criteria. Those that pass go to the reviewer and the regulatory review clock starts. Those that fail are returned for correction. (The conformance rules are publicly available so that conformance errors can be identified and corrected before submission.) During the filing period, automated data quality checks are performed and that report goes to the reviewer. Deficiencies result in possible information requests. Serious data quality deficiencies may form the basis of a refuse to file action.

Finally, let's retire use of the term "DataFit" in favor of what we really mean: Standards Conformance or Data Quality. Let's not muddle these two important issues any longer.


The Semantic Web Way of Thinking

Before I knew anything about the Resource Description Framework (RDF) and the Semantic Web, I would say that I had a traditional way of thinking about the world. Take a clinical trial for instance. First you have a research idea or hypothesis, then you design a trial to test that hypothesis, then you write the protocol, recruit subjects, conduct the trial, analyze the data, and reach conclusions. All of these steps are important but I failed to see any commonality in any fundamental way. For example, writing the protocol and analyzing the data are very different processes needing very different skill sets. How could they possibly be similar?

Enter the RDF and suddenly everything is related in some way with everything else. It may be obvious but no less profound to notice that everything in the world is a type of "Thing." The way we come to understand the world via the scientific process is to classify Things, group Things, separate Things into different buckets or classes based on their properties. Here's an obvious example, written in pseudo-RDF-turtle syntax.

A Car is a subClassOf Thing.
A Red Car is a subClassOf Car.

How does a computer know a car is a red car? Well, one can define a property of Car called Color and one of the options for Color is Red.

A Red Car is [any Car with Color value = Red].

So I can ask the computer to find every Red Car and it knows to look for those with Color property is Red. This is very straightforward, almost to the point of being insultingly simple. But wait...

To make scientific discoveries, we first identify what properties are important for the type of Thing we are studying, and we measure those properties. Let's say you have an investigational drug A and you want to know if it's effective for Multiple Sclerosis. You have the following assertions.

DrugA is a subClassOf Thing.
EffectiveDrug is a subClassOf Drug.

How can a computer discover that DrugA is an EffectiveDrug?  In the same way as the Red Car example, there are properties of DrugA that semantic web tools can analyze to determine that the drug is an EffectiveDrug. Sometimes those properties are difficult to define, or difficult to measure, but the principle is the same.

So, getting back to the different steps in the lifecycle of a study, they are also Things that we can call Activities. There are rules that determine when Activities begin and end, and rules that determine which Activity is performed next. One can define a property of Activities called State or Status (e.g. not yet started, ongoing, completed, aborted). So the lifecycle of a study is broken down to a series of activities, each with its own properties: hasState; hasStart Rule; hasEnd Rule. Suddenly processes that look very different now look very similar.

This is the semantic web way of thinking. The universe is made up of things: similar things and different things. All are grouped together and distinguished from one another by their properties. The challenge is identifying those properties that matter, documenting them, and using semantic web tools to do the grouping and sorting for us. This is how the Semantic Web can work for us and help us make new discoveries.