Selecting Comparators in Active Surveillance Analyses
(31 March, 2010)
OMOP aims to inform the appropriate use of observational healthcare data, such as administrative
claims and electronic health records, for identifying and evaluating drug safety issues within an active
surveillance network. To support this aim, the Partnership has assembled a research community of
disparate data sources within both a centralized system and a distributed network. OMOP has also
established a methods development community, where research collaborators have implemented
multiple different approaches for analyzing observational data. While the specific design of each
method may vary, the fundamental objective remains the same: to estimate the strength of the
association between drug exposure and outcome, as a means of identifying and prioritizing drug‐
condition relationships that may warrant further evaluation.
As part of its empiric approach to its methodological research, OMOP is evaluating the performance of
all analysis methods in their ability to identify true drug safety issues and minimize false positive
findings. Each method deemed computational feasible is being assessed for two analysis problems:
Health Outcomes of Interest (HOI), and Non‐Specified Associations (NSA). For both problems, 10 drugs
of interest (DOI) were identified as mature products with well‐characterized safety profiles. For the HOI
problem, 10 outcomes were selected and one or more operational definitions were implemented for
each outcome. Amongst the 10 HOIs, 9 DOI‐HOI pairs represent ‘true positive’ safety issue test cases, 2
pairs were selected as ‘true positive benefits’, and 44 pairs were classified as ‘negative controls. When
accounting for the multiple outcome definitions, the HOI experiment contains 35 ‘true positives’ and
178 ‘negative controls’, for 213 test cases in total. For the NSA problem, conditions were selected as
either ‘true positive’ safety issues or ‘negative controls’ based on the status of the condition in relation
to the structured product labels for the Drugs of Interest; this process resulted in a test suite that
contained 599 ‘true positives’ and 27,800 ‘negative controls’ for 28,399 test cases in total. The
rationale, process and description of the test cases is described elsewhere. Each method will be used to
produce a score (eg. estimate or test statistic) for each of the test cases, and a series of measures (such
as Mean Average Precision and ‘precision‐at‐k’) will be applied to those scores to assess their relative
performance in classifying the status of the test cases.
All methods have user‐defined parameters that allow the program to be configured to specific settings
pertinent to the drug‐condition relationship under study. For example, many programs allow the user
to specify whether the ‘time‐at‐risk’ is defined as a constant window after drug initiation, or based on a
variable period based on the length of observed exposure. Other methods allow the user to specify
whether the outcome of interest is measured based on incident or prevalent occurrence.
These parameters are used to define how data is extracted and manipulated for use in analysis, and can
impact the resulting estimate calculation. Largely, the degree to which these parameters influence the
performance behavior of the methods is undetermined. OMOP is exploring multiple configurations of
each method in an attempt to uncover patterns about which parameter settings perform best in
identifying particular drug‐condition associations within certain types of data.
Many methods attempt to estimate the strength of the drug‐condition relationship by comparing the
observed rate of drug‐condition co‐occurrence to an expected rate. Some methods estimate the
expected rate based on a selection of one or more comparator drugs. Four methods require a user
parameter for ‘comparator drugs’.
- Maximized sequential probability ratio test (maxSPRT), developed by Harvard Pilgrim and Group
- Conditional sequential sampling procedure (CSSP), developed by Harvard Pilgrim and Group
- High‐dimensional propensity score cohort design (hdPS), initially designed by Schneeweiss et al.
and implemented by the OMOP research team.
- Incident user design (IUD), developed by Alan Brookhart of the University of North Carolina at
It is expected that the performance of these methods could be highly sensitive to the selection of the
comparator drugs. This is consistent with the traditional challenge in pharmacoepidemiologic
observational database study design of selecting the appropriate comparator group. The desire is to
construct a comparator which is sufficiently similar to the target group that effects observed may be
plausibly attributable to the exposure of interest. Analysts take great pains to define study
inclusion/exclusion criteria when extracting the data and apply additional analysis techniques (such as
matching, stratification, and multivariate modeling) to minimize effects of confounding and other biases.
Such design decisions are pivotal to the final result, but can be resource intensive and require subjective
expert opinion. As such, we propose using these methods as an opportunity to develop and evaluate
alternative approaches to comparator selection.
One research opportunity is to evaluate the performance of objective heuristics that could be used to
supplement subjective decision‐making. In the case of comparator selection, OMOP intends to leverage
existing biomedical ontologies to construct alternative definitions of ‘comparator drugs’. NDF‐RT is a
drug classification system, maintained by the Department of Veteran’s Affairs. NDF‐RT provides a multi‐
axial hierarchy for chemical ingredients in medical products (ex. ‘lisinopril’). The relationships contained
within NDF‐RT include: ‘may treat’ and ‘may prevent’, which enumerate indications and other potential
uses of a product (ex. ‘Hypertension’); ‘mechanism of action’, often thought of a drug class (ex.
‘Angiotensin‐converting Enzyme Inhibitors’); ‘chemical structure’ (ex: lisinopril); ‘Physiologic Effect’ (ex.
‘Arterial Vasodilation’). For purposes of this research, OMOP will focus on the ‘may treat’ and
‘mechanism of action’ relationships.
We propose experimenting with five alternative comparator drug definitions:
- select all drugs with the same ‘may treat’ (indication) relationship as the DOI
- select all drugs with the same mechanism of action (class) as the DOI
- select all drugs within a different mechanism of action that share the same primary indication as
- select the most prevalent drug within a different mechanism of action that shares the same
primary indication as the DOI
- Select one or more drugs based on subjective expert opinion
The first four comparator definitions can be systematically extracted from the OMOP common data
model using NDF‐RT. The results of applying these objective heuristics are detailed below. Note, the
‘primary indication’ requires subjective assessment of the available options when there are multiple
‘may treat’ relationships for a give product. Prevalence was estimated as proportion of people with at
least one exposure, as extracted from OSCAR across all sources in the OMOP data community. The
selections of drug were consistently the most prevalent in all sources. The fifth definition reflects
current practice and requires subject matter expertise, including from the collaborators who developed
It should be recognized that the use of objective heuristics is prone to the same limitations as any
subjective assessment involved in comparator selection. The distinction is the relationships between
drugs have been pre‐defined through clinical review from those who curate and maintain NDF‐RT, as
opposed to decided at analysis‐time for a specific question of interest. As a result, relationships in NDF‐
RT, may not be consistent with expectations, either to differing clinical opinions or misclassification. In
some instances, the classification may not be as granular as desired, and therefore the proposed
heuristics may identify a broader set of comparator drugs that initially desired. The intent of conducting
the research to evaluate multiple alternative comparator definitions is to allow an assessment of the
acceptability of objective heuristics, as a potential supplemental approach to current practice in
Full text: » Download PDF (230 KB)