Early Prediction of Antibiotics in Intensive Care Unit Patients
BS Biomedical Informatics, Kent State University, 2004
MD, Northeast Ohio Medical University, 2008
SUBMITTED TO THE CENTER FOR BIOMEDICAL
INFORMATICS AT THE HARVARD MEDICAL SCHOOL IN
PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE
DEGREE OF MASTER OF MEDICAL SCIENCE
Thesis Supervisor: Peter Szolovits, PhD
Title: Professor of Computer Science and Engineering & Health Sciences and Technology,
Massachusetts Institute of Technology/Harvard University
Thesis Committee Members:
David Gagnon, MD, PhD, MPH, Associate Professor of Biostatistics, Boston University School of Public Health
Leo Celi, MD, MSc, MPH, Assistant Clinical Professor of Medicine, Harvard Medical School
Introduction. Predictive models derived from electronic health records in the intensive care unit (ICU) have traditionally used data from the first 24 hours of admission, or up to 48-72 hours. While these may have high accuracy and have fewer limitations due to missing data, they are not useful for decision-making during the early hours of an admission. Infections are common in the ICU and international guidelines recommend that antibiotics be administered as soon as possible. Established goals for early administration of antibiotics range from 1-6 hours. Using structured admission data, we attempt to develop a predictive model to provide early identification of patients who warrant antibiotic administration, from a cohort of patients who were not identified by clinicians as having an infection. Methods. The Multi-parameter Intelligent Monitoring for Intensive Care II (MIMIC II) is a database of patients admitted to the Beth Israel Deaconess Medical Center ICU between 2001 and 2008. Using the MIMIC- II database and a combination of natural language processing and inpatient orders, we identified patients who did not receive antibiotics within 6 hours of admission. Sociodemographic, clinical, and process variables were extracted for each patient. The dataset was divided into a training and test set with an 80:20 split. Logistic regression models were built. Results. 9478 patients met inclusion criteria. Of these, 1403 (14.8%) did not receive antibiotics during the first 6 hours but were subsequently started on antibiotics within two days of hospital admission. The most common antibiotics started were vancomycin, levofloxacin, and metronidazole. A forward-selection logistic regression on the training set, based on a candidate list of variables comprised from theory and bivariate testing, was significant with a c-statistic of 0.67. A logistic regression model on the test set had a c-statistic of 0.65. Most of the variables could not be tested due to data not missing completely at random. The only variables that were significant were physicians' ordering of lactic acid and liver function tests (LFTs). Conclusion. It is possible to build a significant logistic regression model based on admission data. The importance of ordering behavior, a proxy for clinician decision-making, indicates that all relevant data is not captured in structured fields.
Predictive models in the ICU such as the Acute Physiology and Chronic Health Evaluation (APACHE) score  and the Simplified Acute Physiology Score (SAPS)  typically use 24 hours or more of admission data. While these models have been validated in diverse patient populations and used in various settings, they cannot be applied to decisions that must be made within 24 hours without significant customization. For example, early administration of antibiotics have garnered significant international interest due to excess mortality and the Surviving Sepsis campaign . Decisions regarding antibiotics should be made shortly after admission and ideally within 4 hours . As there is great clinical uncertainty surrounding infections, it is often difficult to determine whether a patient has an infection at the time of admission.
An infection in the ICU may be the primary cause of admission or may be present in addition to, or because of, another diagnosis. It is a cause of morbidity, mortality, and high healthcare costs. Sepsis, one form of infection, has a treated mortality between 20 and 50 percent and is the 10th leading cause of death in the United States . The financial cost per hospital admission can be as much as $50,000 per patient which sums to $17 billion in annual costs in the United States. In addition, the incidence has been increasing for unclear reasons .
It has been well established that early administration of antibiotics reduces morbidity and mortality in patients with infection. A landmark study by Kumar found that after the onset of hypotension in patients with sepsis, each one-hour delay in initiation of antibiotics resulted in increased mortality, with 46% overall mortality if antibiotics were not started in the first six hours . This finding has been confirmed in other studies , . Delayed administration of antibiotics has been associated with acute lung injury in patients with pulmonary sepsis , increased medical complications , and increased rate of transfer to the ICU .
However, it is not always easy to determine if there is an infection at the time. The gold standard is growth of pathogenic bacteria from a culture, but such data is not available on admission and may take up to 48 hours to return. Procalcitonin is a relatively new marker but is expensive , can take 7-10 days to return, and has low positive predictive value . In addition, emergency room physicians may be dealing with high volume and other critically ill patients, which may both contribute to a delay in antibiotics. If a patient needs emergency resuscitation or an immediate procedure, efforts will first be made to stabilize the patient and antibiotics may not be considered until later. The decision to treat must also be balanced by possible side effects of antibiotics, contribution to antimicrobial resistance, and cost.
One recent avenue of research has been on sepsis bundle protocols , . The concept is that there is a predefined checklist of criteria available to the physician, which when met, is tied to a bundle of standardized but institution-specific orders which includes diagnostic studies as well as treatment, including antibiotics. When patients meet certain criteria, an alert may appear in the electronic medical record. Because an order set is tied to the criteria, it is less likely that a particular necessary order will be omitted. However, all of these protocols require clinical suspicion of an infection in order to activate, which may not be clear.
In general, previous predictive models of infection and/or sepsis have focused on a particular infection, type of infection, or context , , , used data from structured and unstructured sources, used up to 48 hours of data from admission , and have investigated novel markers . While customized models lead to higher accuracy, the tradeoff is a greater number of models and cognitive overload. A small number of validated models are clinically in use today. Another impediment to use is unstructured data, which is time-consuming for a physician or nurse to enter, assuming the data was collected. Models that use more than a few hours of data may be used for retrospective studies, analysis of treatment options, or for mortality studies, but has low clinical utility as a decision support tool given that the goal for antibiotic administration is less than 1 hour .
We propose development of a model trained on the general outcome of infection, using only structured data, and using commonly available variables that were present on or shortly after admission. By using admission data, we attempt to identify infections and/or sepsis at an earlier stage. We focus on commonly available and inexpensive blood tests that would be available in other ICUs . In addition, our model differs from prior models in a second important way. Previous studies have used an outcome of prediction of infection in an unselected cohort of patients. We hypothesize that patients with certain infections will be easy for clinicians to identify. For example, a patient with a fever, a markedly elevated white count, and a cough productive of sputum likely has pneumonia, and would be easy to distinguish from a patient who did not have these characteristics. The usefulness of a predictive model is in discriminating between patients that clinicians have a hard time separating . For this reason our initial cohort is patients who are not started on antibiotics within the first 6 hours, indicating that there was no suspicion of infection. We exclude patients who were started on antibiotics within 6 hours as these are patients that clinicians are already able to identify.
The Multi-parameter Intelligent Monitoring for Intensive Care (MIMIC II) database consists of high-resolution data of all ICU patients admitted to the Beth Israel Deaconess Medical Center (BIDMC) from 2001 to 2008. It was created through a collaboration between the BIDMC, Philips Healthcare, and the Massachusetts Institute of Technology (MIT). As it is a de-identified database , institutional review board (IRB) approval for this study was not required. IRB approval was obtained from both MIT and BIDMC for the development, maintenance and public use of MIMIC-II.
The database consists of data from more than 25,000 patients, including pediatric and adult, and from the medical, surgical, and neurological ICUs, and the cardiac surgery unit. While data from outside the ICU at the BIDMC is generally not available, complete hospital course information is available for patients who were transferred to or from the ICU. Clinical data consists of vital sign information, laboratory data, high resolution waveform information, nursing notes, discharge summaries, and medication orders. Documentation of medication administration is not available. As the emergency department was using a different information system, emergency department notes and orders are not available, but laboratory data from the ED course is present.
The outcome of the study is prediction of infection in a cohort of patients in whom infection was not suspected. To operationalize this, we extracted a cohort of patients who did not receive antibiotics during the first 6 hours of admission, but were subsequently started on antibiotics within the first 2 days. As the incubation period of bacteria is 48 hours, the selection of this time window necessitates that all patients who were started on antibiotics must also have had the infection present upon admission to the hospital. Contrariwise, if an infection at any point during the hospitalization were an outcome, a patient could have developed an infection on hospital day 3 that became clinically apparent on hospital day 5. Looking at initial laboratory data would degrade performance given that no signs or symptoms of infection would have been present until day 3.
While we have access to culture and microbiologic data, which have been used as outcomes in previous studies, we selected a clinician-centered outcome due to the fact that cultures are often not able to be obtained from patients. In addition, when cultures are able to be obtained, results may be negative in the presence of infection due to slow-growing or difficult-to-grow bacteria, and in intra-abdominal sepsis . The initial cohort further represents a group of patients that the admitting clinician or clinicians were not able to distinguish, shown by the fact that none received antibiotics during the first 12 hours. Further analysis of the cohort started on antibiotics late was performed to ascertain that this group had a high rate of infection and will be presented in the results section.
ICD-9 discharge diagnoses of infections were not used as the study outcome for two reasons. Several recent studies have found ICD-9 codes for sepsis to be inaccurate , , . For example, a study by Martin found a positive predicted value of 88.9% and a negative predictive value of 80.0%. Sensitivity and specificity were not reported. A study by Ollendorf concluded that using ICD-9 codes for sepsis in research may be “prone to substantial error.”. The second reason is that ICD-9 discharge diagnosis codes are time-insensitive. Because an infection could have occurred at any time during the hospitalization, parameters of infection might not have been present upon admission (because infection was not present upon admission), which was used as the input time period for predictive variables.
The inclusion criteria for the study are adult patients, defined as > 15 years, who are admitted directly to the ICU, go from the emergency department straight to the ICU, and were not transferred to the BIDMC from another hospital. We excluded patients who were transferred from the wards to the ICU as we could not rule out hospital-acquired infection. Transfers from other hospitals were excluded as we could not determine if antibiotics had been administered at the previous setting. The table icustay_detail was used as a master list of hospitalizations. As MIMIC-II was constructed from real EHR data, it has the missing data consistent with live systems including missing hospital admission IDs. Hospitalizations from the icustay_detail that were missing hospital admission IDs were linked by using subject IDs and dates of admissions from the admissions table. A consort diagram is shown in Figure 1 .
Figure 1 – Consort Diagram
Description of Natural Language Processing
As documentation of medication administration is not available in the database, inpatient orders were used to determine if and when patients received antibiotics. It is assumed that if a medication order was placed, the medication was received by the patient. The initial cohort was created based on absence of antibiotic orders during the first 6 hours. As we did not have access to ED orders, and some of these patients may have received antibiotics in the ED, we further used natural language processing (NLP) to identify these patients. Since no prior studies could be identified that used NLP for this particular task, we created a custom algorithm. A prior study used a commercial NLP tool, MedLEE, on oncology nursing notes.
A new cohort was extracted from the MIMIC-II database, which consisted of adult hospitalizations that were not transferred from another facility nor the wards. Out of 17,005 hospitalizations, 86.9% of patients had at least one nursing note during this time period. Nursing admission notes were selected by taking the first nursing note from each hospitalization. A random selection of notes were reviewed and found to be consistent with being admission notes. The caregiver id had to be assigned a label of “RN” and a time period of 24 hours was specified, such that all notes documented more than 24 hours after the time of ICU admission were excluded. Because nursing shifts are 8-12 hours in length, additional time was included so that a nurse who documented his/her findings after the shift would not be excluded. However, if multiple notes were present, only the first one was used, and other notes (by medical students, nursing students, respiratory therapy, and so on) were excluded. Each note is computerized but consists of free-text. While there are no requirements on the content of a nursing note, it is expected that all administrations of antibiotics will be documented. Through manual review of a random selection of notes, it was determined that in addition to excellent documentation of antibiotic administration in the emergency department, the notes also frequently contained information on past medical history, reason for admission, medications, allergies, and descriptions of the plan.
A list of relevant antibiotics is displayed in Table 1. The list was compiled from published drug references. Cefazolin, an antibiotic which is frequently used perioperatively, was excluded, as we are interested in antibiotics used for the treatment of infection and not prophylactic antibiotics. Because medication orders are stored as free-text fields in MIMIC-II, we could not identify antibiotics based on a unique identifier, and also needed to use variations of commonly misspelled words. For this reason the antibiotic list in Table 1 was designed to be as inclusive as possible. Topical antibiotics were excluded.
|Amikacin |Amoxicillin |Ampicillin |Azithromycin |
|Aztreonam |Vancomycin |Cefaclor |Cefadroxil |
|Cephalexin |Cefamandole |Cefepime |Cefixime |
|Cefotaxime |Cefoxitin |Cefpodoxime |Cefprozil |
|Ceftazidime |Ceftriaxone |Cefuroxime |Chloramphenicol |
|Ciprofloxacin |Clarithromycin |Clindamycin |Colistin |
|Dapsone |Daptomycin |Dicloxacillin |Doripenem |
|Doxycycline |Ertapenem |Erythromycin |Ethambutol |
|Flucloxacillin |Fosfomycin |Gatifloxacin |Geldanamycin |
|Gentamicin |Imipenem |Isoniazid |Kanamycin |
|Levofloxacin |Linezolid |Meropenem |Methicillin |
|Metronidazole |Minocycline |Moxifloxacin |Trovafloxacin |
|Nafcillin |Nalidixic acid |Tobramycin |Netilmicin |
|Nitrofurantoin |Norfloxacin |Ofloxacin |Paromomycin |
|Penicillin |Piperacillin |Polymyxin |Pyrazinamide |
|Quinupristin |Rifabutin |Rifampicin |Rifampin |
|Spectinomycin |Streptomycin |Sulfadiazine |Sulfamethoxazole |
|Telithromycin |Tetracycline |Ticarcillin |Tigecycline |
|Trimethoprim | | | |
A list of approximately 50 words/expressions that were likely to indicate antibiotic administration was developed based on expert opinion. These mainly consisted of actual antibiotic names, both generic and trade names, and terms such as “abx,” “antibiotic,” and “antibiotics.” From the initial nursing notes, a training and test set were created with an 82:18 split.
In order to identify additional keywords that may indicate administration of antibiotics in the ED, all notes that corresponded with hospitalizations in which patients were ordered for antibiotics within the first 6 hours were extracted from the training set. As there has been a proliferation of antibiotic ordering in the emergency department and a trend toward giving early antibiotics, it is assumed that if patients were ordered for antibiotics on the inpatient side, they would have received the first doses of antibiotics in the ED, and documentation would be present in the initial nursing note. While we do not have ED nursing notes, the ICU nurse is required to receive signout from the emergency department nurse during which administration of antibiotics and other important considerations will be relayed. All notes in which patients received early inpatient antibiotics (defined as within 12 hours) were combined into a single document and tokenized using Unix text tools. Case and punctuation were removed and words were sorted by frequency. Words that were likely to indicate antibiotic use based on expert opinion were added to the previously constructed list of 50 words.
Once the list of words was complete, notes in the training set were searched for notes containing specific keywords to determine whether each word was in fact associated with the outcome of receiving antibiotics. During this process, some words were excluded from the list and other ones were added based on manual reading of notes in the training set. Given that there were more than 12,000 notes in the training set, only a small fraction of the total notes were examined. Once this process was complete, documents from the training set were then randomly selected based on an overall composite of search terms/phrases. Further changes were made to the list of terms. A list of the final selection of words/expressions may be found in the Appendix.
Once the above process was complete, the test set was used to determine accuracy. 80 nursing notes were randomly selected from the test set, which had not been seen by the NLP classifier nor a reader. Based on manual reading by a physician, each note was classified as patient receiving or not receiving antibiotics in the ED. After an assignment was made for each note, the natural language processing classifier was run on these notes. Sensitivity, specificity, positive predictive value, and negative predictive value were calculated, with manual review being considered the gold standard.
The results of the consort diagram above include both inpatient orders and natural language processing. In other words, patients were categorized as not having received antibiotics within the first 6 hours if both inpatient orders during that time window were negative for any of the specified antibiotics, as well as if there was no indication of the patient receiving antibiotics based on the initial MICU nursing note. If the initial MICU nursing note was not available, inpatient orders were used.
Variables were selected based on a combination of prior theory and bivariate testing. While the specific question of this research project has not been tested, the literature was evaluated for predictive models on infections and sepsis, both in the ICU, non-ICU inpatient, and outpatient settings. Variables that were found to be significant (p-value = 2 |Tudela 2010 |
|Procalcitonin > 0.4 |Tudela 2010 |
|albumin |Wildi 2011 |
|presence of SIRS |Wildi 2011 |
|liver disease (cirrhosis or chronic hepatitis) |Bates 1997 |
|hickman catheter or indwelling vascular catheter |Bates 1997, Shapiro 2008 |
|altered mental status |Bates 1997 |
|focal abdominal signs |Bates 1997 |
|clinical suspicion of endocarditis |Shapiro 2008 |
|Age > 65 |Shapiro 2008 |
|chills |Shapiro 2008 |
|vomiting |Shapiro 2008 |
|Neutrophil % > 80 |Shapiro 2008 |
|WBC > 18,000 |Shapiro 2008 |
|Bands > 5% |Shapiro 2008 |
|Creatinine > 2 |Shapiro 2008 |
|gender |Martin 2003 |
|ethnicity |Martin 2003 |
| | |
To the above list were added the most common laboratory variables present in MIMIC to create a candidate list. This final criterion was used because none of the previous studies took an exploratory approach to variable selection, leading to the possibility that important predictive variables were missed. As an example, Gil and others first identified eosinopenia as a sensitive and specific predictor of bacterial infections in 2003, despite the fact that this commonly-obtained marker has been available for over 100 years . A plethora of predictive models created before 2003 did not include eosinophils as a candidate variable. In addition, since systemic bacterial infections cause changes to numerous laboratory parameters, it is plausible that other common laboratory tests would have abnormal values. A list of the most common laboratory tests ordered is at the end of the Appendix.
The dataset was divided into a training and test data set with an 80/20 split.
All variables were inspected for missing data and erroneous values. In particular, a number of temperatures, respiratory rates, and oxygen saturations had physiologically impossible values and were removed. For laboratory values, the first value available within 4 hours of admission was used. For vital signs, the high, low, mean, and initial was calculated based on 4 hours of data. For the statistical analysis, the training set was divided into patients who did not receive antibiotics and patients who received antibiotics after 6 hours but within two days. For continuous variables, t-tests were done with equal or unequal variance as appropriate, with the outcome variable. Fisher’s exact test was used for categorical variables. All continuous variables were kept continuous. Variables with a p-value of < 0.15 were added to the candidate list that was previously formed. As there was a large amount of missing data, ordering of individual laboratory tests were coded as variables. From this list of variables, a forward selection logistic regression was performed using SAS 9.3 with a significance to enter of 0.05. A 20% test set was withheld.
To adjust for the effects of co-morbidities in predictive models, Charlson and Elixhauser scores have been used. Both of these rely on prior co-morbidity data that is present at the time of admission. Since prior data is not available in MIMIC-II (excluding patients that have had multiple hospitalizations), Elixhauser scores were not used, which would falsely improve model performance by incorporating data into the prediction that had not yet occurred in real life . For the same reason, SAPS and SOFA (Sequential Organ Failure Assessment) scores were not used. While SOFA scores can be calculated based on initial data, the sofa_first computed in MIMIC-II was based on data from the first 24 hours of admission.
9,478 patients met inclusion criteria. Of these, 1403 (14.8%) were not started on antibiotics within the first 6 hours but received antibiotics during the first two days. The most common antibiotics patients were started on are vancomycin, levofloxacin, and metronidazole. Additional antibiotics started and the number of times each antibiotic was administered are displayed in Table 3. If multiple orders were present for the same patient, these were added together.
Most Common Antibiotics Administered
|Vancomycin |1838 |
|Levofloxacin |1143 |
|Metronidazole |631 |
|Piperacillin-Tazobactam Na |401 |
|Ciprofloxacin HCl |363 |
|Azithromycin |167 |
|Clindamycin |152 |
|Gentamicin |152 |
|Ceftriaxone |146 |
|CeftriaXONE |96 |
|Sulfameth/Trimethoprim DS |94 |
|Ampicillin |90 |
|Ceftazidime |86 |
|Ampicillin-Sulbactam |71 |
|Meropenem |69 |
|CefePIME |60 |
|Aztreonam |44 |
|Oxacillin |42 |
|Sulfameth/Trimethoprim SS |41 |
|Linezolid |41 |
|Erythromycin |35 |
|Amoxicillin |29 |
|Amoxicillin-Clavulanic Acid |27 |
|Cefepime |25 |
|Nafcillin |22 |
|Vancomycin Oral Liquid |22 |
|Clarithromycin |21 |
|Doxycycline Hyclate |20 |
|Cefpodoxime Proxetil |18 |
|Imipenem-Cilastatin |14 |
|Dicloxacillin |13 |
|Sulfameth/Trimethoprim |12 |
|Dapsone |10 |
|Nitrofurantoin (Macrodantin) |9 |
|Daptomycin |8 |
|Penicillin G Potassium |7 |
|Clindamycin HCl |7 |
|Amikacin |6 |
|Cefotetan |6 |
|Tobramycin |5 |
|Penicillin V Potassium |5 |
|Sulfameth/Trimethoprim Suspension |4 |
|Isoniazid |3 |
|Amoxicillin Oral Susp. |3 |
|Minocycline HCl |3 |
|DiCLOXacillin |2 |
|Ethambutol HCl |2 |
|Pyrazinamide |2 |
|Cefuroxime Sodium |2 |
|Nitrofurantoin Monohyd (MacroBID) |2 |
|SulfADIAzine |1 |
|Tetracycline HCl |1 |
|Rifampin |1 |
|Minocycline |1 |
Natural Language Processing
80 notes were randomly selected from the test set. The results of the validation are displayed in Figure 2. 95% confidence intervals are displayed in parentheses.
| |Antibiotics received |No antibiotics received |
|NLP positive result |16 |6 |
|NLP negative result |1 |57 |
Sensitivity: 94.1% (82.9-100)
Specificity: 90.5% (83.2-97.7)
PPV: 72.7% (54.1-91.3)
NPV: 98.3% (94.9-100)
Results of t-tests and Fisher exact tests are shown in Table 4. P-values of < .05 are shown in bold. In a cohort of patients who were not suspected of having an infection, patients who received antibiotics were more likely to be older, on Medicare, in the medical ICU, have greater derangement in laboratory values, and were more likely to have additional tests ordered on admission. All results are for the training data.
|Characteristic |No antibiotics (N = 6,477) |Antibiotics (N = 1105) |P Value |
|Age – yr |60.9 |62.9 |0.001 |
|Male sex – No (%) |3798 (58.7%) |617 (55.9%) |0.08 |
|Insurance – No. Medicare (%) |2481 (38.3% |503 (45.5%) |
When to check for vancomycin trough level? Vancomycin levels are taken 1-2 hours pre - dose (trough). Do not wait for the level to come back before giving the dose that is due unless CrCl <20mL/min or specifically instructed to do so by microbiology or pharmacy. The trough level should be between 10-20mg/L. Adult Vancomycin intermittent infusion: Prescribing,