Basic Statistics Should Be a Core Competency of Every Citizen of the World

A medical educator recently argued in her blog that medical school admissions requirements should minimize requirements in math and science topics, especially areas like calculus and physics. There is no question that medicine, and even informatics for that matter, require knowledge and competency in many areas beyond math and science.

However, the problem with the math we teach to potential healthcare professionals and informaticians, indeed to everyone in society, is that we teach the wrong math. I took three semesters of calculus in college and can say that I have almost never used any of it. On the other hand, I had almost no education in statistics, a type of math I use not only in my work, but also in my function as an informed citizen. Indeed, most healthcare professionals, whether clinicians or researchers, use statistics daily. Likewise, as thoughtful citizens in society, we also encounter statistics daily in the news and other aspects of our lives.

For these reasons, I believe that statistics should be a core competency of every citizen in the modern world.

It is not even the mathematics in statistics that are most important, but rather the concepts and the thinking they engender. Every citizen in the world should understand the basic concepts of inferential statistics and be able to answer such questions as:

• What does statistical significance mean? How is it different from a clinical (not necessarily in the medical context) significance?
• What is the difference between absolute and relative risk? What is the meaning of large relative risk differences in the setting of small absolute risk?
• In health-related topics, how do we discern and compare different types of health risks?
• Also in health, what do sensitivity and specificity of diagnostic tests mean, and how does prevalence impact the risk of disease in the face of positive or negative diagnostic tests?

One of the most articulate advocates of this view is John Allen Paulos, whose books Innumeracy and A Mathematician Reads the Newspaper inform us why basic numeracy and statistical competency are so important. These kinds of engaging writings, and basic education about statistics, should be a part of every high school education, not to mention in the education of clinicians and informaticians.

Universal EHR? No. Universal Data Access? Yes.

A recent blog posting calls for a "universal EMR" for the entire healthcare system. The author provides an example and correctly laments how lack of access to the complete data about a patient impedes optimal clinical care. I would add that quality improvement, clinical research, and public health are impeded by this situation as well.

However, I do not agree that a "universal EMR" is the best way to solve this problem. Instead, I would advocate that we need universal access to underlying clinical data, from which many different types of electronic health records (EHRs), personal health records (PHRs), and other applications can emerge.

What we really need for optimal use of health information is not an application but a platform. This notion has been advanced by many, perhaps most eloquently by Drs. Kenneth Mandl and Isaac Kohane of Boston Children's Hospital [1,2]. Their work is being manifested in the SMART platform that is being funded by an ONC SHARP Award.

Mandl and Kohane point to the iPhone as an example of building a platform on top of a common data store. I see this in action every day on my iPhone, when different applications make use of various data stores built into the phone, such as its GPS data. (Android and other phones offer similar functionality.) Not only Google Maps uses this data, but also my LA Fitness app that tells me where the nearest club is located when I am in a different city and hoping to find a gym.

A common data store, on top of which a thousand flowers (or apps) can bloom, is the ideal situation to the health information system "ecosystem." This will allow new ideas and innovations to flourish, while insuring that interoperable data will be accessible by all apps that have appropriate and authorized access. It will insure competition and a healthy marketplace to bring out the best in health information technology.

References

1. Mandl, KD and Kohane, IS (2009). No small change for the health information economy. New England Journal of Medicine. 360: 1278-1281.
2. Mandl, KD and Kohane, IS (2012). Escaping the EHR trap--the future of health IT. New England Journal of Medicine. 366: 2240-2242.

The Workforce Group of the ONC Health IT Policy Committee Makes Its Recommendations

For the last nine months, I have had the opportunity to be part of a workgroup of the ONC Health IT Policy Committee focusing on the health IT workforce issues. This week, Larry Wolf, co-chair of the workgroup made a presentation of the group's recommendations to a meeting of the full Health IT Policy Committee.

The recommendations of the workgroup can be summarized as follows:

1. ONC should summarize and publicize the results of the several workforce development programs it has funded.
2. ONC should summarize and widely disseminate the core competencies for members of the workforce that it has identified.
3. ONC should publicize the resources and best practices that they and other organizations have made available.
4. There is an emerging need for soft and hard skills related to team-based care, population health and patient engagement. ONC should recommend new program development and funding to address these needs.
5. ONC should learn from what is happening with the current workforce. It should do this by recommending funding of studies on the impact of health IT on the workforce, such as turnover, enrollment in healthcare vocations (schools), and new jobs, such as nurse informaticists.
6. The current Standard Occupational Classification (SOC) does not address health IT. ONC should host an SOC input process from the health IT community.

While I agree with all of the recommendations that our group made, I would have added two additional recommendations, which actually build on the fourth and fifth recommendations in the list. The first of these emanates in part from the slide presentation, which might be read by some to imply that informatics and IT jobs in healthcare are "technician" jobs. While healthcare organizations certainly need well-trained health IT technicians, this ignores the larger role that informatics will play as advanced health IT is adopted and used as an integral part of efforts such as quality measurement and improvement, accountable and coordinated care, and biomedical advances such as personalized medicine. Health IT is not merely a support function, but a critical component of our armamentarium to achieve the triple aim of improved health, better care, and lower cost.

My second additional recommendation builds on the recommendation for learning about the current workforce. In light of the larger role for health IT described in the previous paragraph, we need a much more comprehensive understanding than just impact on the current workforce and new jobs. We need to better understand not only of current workforce practices but also how to develop and educate the best workforce going forward into the new era of accountable and coordinated care and new advances such as personalized medicine.

I look forward to the continued efforts of the workgroup and our academic program at Oregon Health & Science University is certainly incorporating this forward-looking view as we revise and augment the curricula of our programs.

PCORI Clinical Data Research Network Funding Opportunity Casts Broad Vision for Clinical Data Use for Research

A couple weeks ago, the Patient-Centered Outcomes Research Institute (PCORI), an organization funded under the Affordable Care Act (ACA), released a funding opportunity to establish a series of Clinical Data Research Networks (CDRNs). The goal for these CDRNs is to develop a data infrastructure that will allow them to provide the data infrastructure for comparative effectiveness research (CER) to be done both within and outside their networks. (A separate announcement for complementary Patient-Powered Research Networks was also made and is included in the PDF announcing funding for the CDRNs.)

Naturally my institution is evaluating whether we have the resources and commitment to apply ourselves, and this gave me a reason to review the funding opportunity announcement in detail. I generated the following summary for my local colleagues, but also can state that no matter what we do, the funding opportunity presents a comprehensive vision for what CDRNs in general should look like, and indeed advances our moving toward the Institute of Medicine (IOM) vision of the learning health system [1].

A total of $56M will be awarded for eight or so $7M projects. Applicants can ask for a higher budget than $7M, but that must be approved by PCORI before a proposal at a higher funding level is submitted. The proposal process will entail two steps, the first of which is the submission of a letter of intent by June 19, 2013. PCORI will then invite some but not all of those submitting a letter of intent to submit full proposals, which will be due by September 27, 2013.

Each CDRN will be required to engage two or more different health systems and have one million or more patients enrolled among them. All systems will be required to have an electronic health record (EHR) system in place and the ability to standardize data among them. In essence, the system must adhere to the data standards specified by Stage 2 of the federal meaningful use program. This includes adoption of standards such as Consolidated Clinical Document Architecture (CCDA) for patient summaries, SNOMED CT for problem lists, RxNorm for electronic prescriptions, ICD-10-CM for diagnoses, ICD-10-PCS for procedures, CMS PQRI 2009 Registry XML Specification for quality reporting, and HL7 Version 2.5.1 for reporting of public health laboratory data and immunization administration. (A succinct overview of these standards in provided in a summary of all of the Stage 2 meaningful use requirements by Metzger and Rhoads [2].)

Another requirement is the ability of the CDRN to identify and recruit cohorts of patients with defined conditions. In particular, three patient cohorts must be able to be identified:

• A disorder of applicant’s choosing that includes 10,000 identified patients
• One or more rare diseases with prevalence less than 1 per 1500 persons in the US
• Overweight or obese patients, identified for presence of diabetes or pre-diabetes

Among the other requirements for the CDRN are:

• Ability to capture complete information on these patients over time, which could be a challenge due to patients getting care in multiple locations that has been shown to be widespread [3, 4]
• Process for patient as well as clinician engagement in governance as well as setting of research priorities.
• Commitment and active involvement of leadership of all participating organizations to play an active role in the governance of the CDRN and its meeting all key objectives
• Willingness to serve as a national data infrastructure resource for the conduct of CER by researchers both within and outside of the network - the award itself does not fund any research but the CDRN must engage CER researchers, including those outside network
• Demonstration of capacity to connect with patients for collecting data and recruiting them into clinical trials
• Capacity to support large CER randomized trials, with the embedding of research activity in functioning healthcare systems while not disrupting their usual healthcare business functions - this will require support of these activities from the respective administrative and executive leadership of participating organizations
• Integrating human subjects oversight, institutional review board (IRB) activities, and informed consent procedures across the network
• Policies to maintain data security, patient privacy, and confidentiality, along with organizational privacy
• Providing access to biological specimens for research purposes
• Robust governance, with the ability to identify and act upon unanticipated problems or issues
• Description of the efficient use of human and other resources to accomplish the project

One additional capacity I would have liked to see required would be participation in informatics and related research looking at the organization and function of the CDRNs and their data collection and usage issues. The development of CDRNs is necessary to advance the learning health system, but it is something for which there are still many unknowns. A research agenda to explore the issues of how to collect and use for research the operational clinical data on a network of multiple healthcare delivery systems and one million patients is critical to make sure our knowledge of how to do this effectively is learned and documented.

References

1. Smith, M, Saunders, R, et al. (2012). Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC, National Academies Press.

2. Metzger, J and Rhoads, J (2012). Summary of Key Provisions in Final Rule for Stage 2 HITECH Meaningful Use. Falls Church, VA, Computer Sciences Corp.

3. Bourgeois, FC, Olson, KL, et al. (2010). Patients treated at multiple acute health care facilities: quantifying information fragmentation. Archives of Internal Medicine. 170: 1989-1995.

4. Finnell, JT, Overhage, JM, et al. (2011). All health care is not local: an evaluation of the distribution of emergency department care delivered in Indiana. AMIA Annual Symposium Proceedings, Washington, DC. 409-416.

The Road to Big Data Passes Through Informatics

I have written a number of postings over the last year about various aspects of electronic health record (EHR) data, from the transition of the work of informatics from implementation to analytics to the problems that still prevent us from making optimal use of data, such as the difficulties of data entry. One of my themes has been that knowledge will not just fall out of the data; we will need to improve the quality and completeness of data to learn from it. The requirements for getting better data include widespread adherence to data standards, engaging and motivating those who enter data to improving it, making it easier for those individuals to enter quality data, and evolving our healthcare system to valuing this data. If we are not able to meet these challenges with our current data, it is unlikely we will be able to do so when we have "big data," i.e., that which is orders of magnitude larger and more complex beyond what we have now. No field has devoted more thought, research, or evaluation to the challenges of clinical and health data than informatics. Thus, whether it is tackling issues of how to implement systems in complex clinical settings; meeting the needs of clinicians, patients, and others; or how to maximize the quality of data, the road to making the best use of (big or non-big) data must pass through informatics.

An example of the fact that knowledge will not just fall out of the data comes from some research activity I have been involved in over the last couple years, which is the Text Retrieval Conference (TREC) Medical Records Track [1]. As those familiar with the field of information retrieval (IR) know, TREC is an annual "challenge evaluation," sponsored by the National Institute for Standards and Technology (NIST) [2]. Challenge evaluations bring research groups with common interests and use cases together to apply their systems to a common task or set of tasks, using a common data set, and comparing results using agreed-upon metrics (ideally in a scholarly and not an overly competitive forum). TREC operates on a yearly cycle, consisting of 5-7 "tracks" that each represent a specific focus of IR research. TREC began with the straightforward tasks of "ad hoc" retrieval (user entering queries into a search engine seeking relevant documents) and "routing" (user seeking relevant documents from a new stream of documents based knowledge of previous relevant documents). In subsequent years, TREC evolved to its current state of diverse tracks representing newer problems in IR, such as Web search, video searching, question-answering, cross-language retrieval, and user studies. (Some of these tracks have spawned their own challenge evaluations, especially in the area of cross-language evaluation, an important issue in Europe and Asia.) Virtually all tracks have focused on generic content, typically newswire or Web content, with very few being "domain specific," although I have been involved in two domain-specific tracks in the areas of genomics literature [3] and medical records [1].

In TREC and IR jargon, test collections consist of an adequately large and realistic collection of content, such as documents, medical records, Web pages, etc. [2]. Test collections also include a set of topics, usually at least 25-50 for statistical reliability [4], that are instances of the task being studied. A final component is human relevance judgments or assessments over the content items, indicating which are relevant and should be retrieved for each topic. Success is usually measured by some sort of aggregate statistic that combines the base measures of recall (proportion of relevant content items in the test collection retrieved) and precision (proportion of relevant content items in the search retrieved). (For those familiar with medical diagnostic test characteristics, these correspond to sensitivity and positive predictive value. The reciprocal of precision is also sometimes called number needed to retrieve, since it measures how many overall documents must be read or viewed for each relevant one retrieved.)

The use case for the track TREC Medical Records Track was identifying patients from a collection of medical records who might be candidates for clinical studies. This is a real-world task for which automated retrieval systems could greatly aid in ability to carry out clinical research, quality measurement and improvement, or other "secondary uses" of clinical data [4]. The metric used to measure systems employed was inferred normalized distributed cumulative gain (infNDCG), which takes into account some other factors, such as incomplete judgment of all documents retrieval by all research groups.

The data for the track was a corpus of de-identified medical records developed by the University of Pittsburgh Medical Center. Records containing data, text, and ICD-9 codes are grouped by "visits" or patient encounters with the health system. (Due to the de-identification process, it is impossible to know whether one or more visits might emanate from the same patient.) There were 93,551 documents mapped into 17,264 visits.

I was involved in a number of aspects of organizing this track. I contributed in both guiding the task (or use case) as well as leading some of track infrastructure activities, namely development of search topics and relevance assessments. This work has been aided greatly by students with medical and other expertise in the OHSU Biomedical Informatics Graduate Program.

The results of the TREC Medical Records Track provide a good example of why the road to big data passes through informatics, or in other words, why there is still considerable work to be done from an informatics standpoint before knowledge simply falls out of data. While the performance of systems in the track has been good from an IR standpoint, they also show these systems and approaches have a considerable ways to go before we can just turn the data analytics crank and have medical knowledge emanate. The magnitude of how far we need to go comes from the precision at various levels of retrieval (e.g., precision at 10 retrieved, 50 retrieved, 100 retrieved, etc.), demonstrating how many nonrelevant visits are retrieved. In the case of typical ad hoc IR, we can probably quickly dispense with documents are relatively easy to identify as not relevant. But this may be a more difficult task for complex patients and complex records.

A failure analysis over the data from the 2011 track carried out at OHSU demonstrated why there are still many challenges that need to be overcome [5]. This analysis found a number of reasons why visits frequently retrieved were not relevant:

• Notes contain very similar term confused with topic
• Topic symptom/condition/procedure done in the past
• Most, but not all, criteria present
• All criteria present but not in the time/sequence specified by the topic description
• Topic terms mentioned as future possibility
• Topic terms not present--can't determine why record was captured
• Irrelevant reference in record to topic terms
• Topic terms denied or ruled out

The analysis also found reasons why visits rarely retrieval were actually relevant:

• Topic terms present in record but overlooked in search
• Visit notes used a synonym for topic terms
• Topic terms not named and must be derived
• Topic terms present in diagnosis list but not visit notes

A number of research groups used a variety of techniques, such as synonym and query expansion, machine learning algorithms, and matching against ICD-9 codes, but still had results that were not better than manually constructed queries (which also require a form of informatics expertise in knowing how to query the clinical domain). The results data also show this is a challenging task, as the performance of different systems varied widely on different topics.

From my perspective, these results show that successful use of big data will not come just from smart algorithms and fast computer hardware. It will also require the informatics expertise to design and implement EHRs, high-quality and complete clinical data, and a proper understanding of the clinical/health domain to make most effective use of the data. As such, achieving the value of big data passes through informatics.

References

1. Voorhees, E and Hersh, W (2012). Overview of the TREC 2012 Medical Records Track. The Twenty-First Text REtrieval Conference Proceedings (TREC 2012), Gaithersburg, MD. National Institute for Standards and Technology.

2. Voorhees, EM and Harman, DK, Eds. (2005). TREC: Experiment and Evaluation in Information Retrieval. Cambridge, MA, MIT Press.

3. Hersh, W and Voorhees, E (2009). TREC genomics special issue overview. Information Retrieval. 12: 1-15.

4. Buckley, C and Voorhees, E (2000). Evaluating evaluation measure stability. Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Athens, Greece. ACM Press. 33-40.

5. Edinger, T, Cohen, AM, et al. (2012). Barriers to retrieving patient information from electronic health record data: failure analysis from the TREC Medical Records Track. AMIA 2012 Annual Symposium, Chicago, IL, 180-188.

Biomedical and Health Informatics vs. Data Science, mHealth, etc. - New Disciplines or New Terminology?

When I entered the field of informatics in the 1980s, a great deal of the research was driven by "artificial intelligence" (AI). Many people were trying to build "rule-based expert systems," while those interested in knowledge representation were constructing "semantic networks." We rarely hear these terms in quotes these days, perhaps with the exception of AI that one hears occasionally. It is not, however, that no one is trying to build systems that guide decision-making and represent knowledge in complex ways, but we just different terminology now, such as clinical decision support and ontologies.

Fast forward to the present, and we see the introduction of new terms, most prominently right now data science [1] and mHealth [2]. Many who are doing work in these areas talk of them as the primary focus of their work. I question, however, whether these are truly new disciplines, or just concentrations (at least for those working in health-related areas) within biomedical and health informatics [3]?

I am most concerned about mHealth, when I see new people coming forward with brilliant ideas and truly innovative technologies, yet not incorporating the experiences from decades of work in informatics. I do not deny that some aspects of using mobile connected devices for health are truly novel, yet what I consider to be the basic principles of informatics still apply, namely things like scalability, interoperability, usability, and so forth. I just see nothing novel enough about mHealth to not call it part of informatics.

The same holds, in my opinion, for data science. There are certainly "computationalist" techniques of which many who work in informatics are not skilled. "Big data" applications will require specialized knowledge. But informatics is a broad field, and no one can master everything. There are other aspects of informatics, such as (I am repeating myself from the previous paragraph here) scalability, interoperability, usability, and so forth that must be married from the results of data science to make the latter's output truly usable. One case in point is the growing number of analyses that predict undesired outcomes, such as hospital readmissions [4]. I am as intellectually interested in these applications as much as anyone, but until it is shown these analyses can be actionable, they will mostly remain interesting theoretical exercises.

I am excited for mobile health applications and advanced uses of data techniques to improve health, healthcare, and research. I hope that those pursuing them do not lose sight of the larger picture of providing end-to-end value for the use of data, information, and knowledge in health-related endeavors, i.e., the goal of biomedical and health informatics [3].

References

1. Davenport, TH and Patil, DJ (2012). Data Scientist: The Sexiest Job of the 21st Century. Harvard Business Review, October, 2012. http://hbr.org/2012/10/data-scientist-the-sexiest-job-of-the-21st-century/.
2. Krohn, R and Metcalf, D (2012). mHealth: From Smartphones to Smart Systems. Chicago, IL, Healthcare Information Management Systems Society.
3. Hersh, W (2009). A stimulus to define informatics and health information technology. BMC Medical Informatics & Decision Making. 9: 24. http://www.biomedcentral.com/1472-6947/9/24/.
4. Gildersleeve, R and Cooper, P (2013). Development of an automated, real time surveillance tool for predicting readmissions at a community hospital. Applied Clinical Informatics. 4: 153-169.

ONC Health IT Curriculum: Funding Ends, Materials Remain Available

The end of March, 2013 will mark a major change to a project to which I have devoted a great deal of my professional life over the last few years, which is the Office of the National Coordinator for Health IT (ONC Health IT Curriculum project. This project began in April, 2010, when five universities were awarded funding to each produce four out of a total of 20 "components" (comparable to courses) for the curricular materials whose primary audience would be the ONC Community College Consortia, although we would later expand this to all of higher education providing educational programs in informatics and health IT. Oregon Health & Science University (OHSU) was awarded additional funding to serve as the National Training and Dissemination (NTDC), tasked with developing mechanisms for disseminating the curricular materials, providing technical support, and training community college faculty in their use.

First things first for those who may have worries: The curricular materials will continue to be available on the NTDC Web site, which I plan to maintain with funding from my own department through the end of 2013. In the meantime, ONC is also developing a transition plan to house the materials on their Web site. The materials have also shown up in various other places, as is allowed under the Creative Commons license that we adopted, and of course anyone is free to download and use them under the terms of that license. The best description of this third and final version of the curriculum is in a previous blog posting of mine, with some additional notes (mainly on the new version of the VistA for Education system) provided in another posting.

A more serious challenge for the ONC Health IT Curriculum is how to maintain and update the materials going forward. After the grant funding ends this month, our support for the materials will end. This not only means the end of the online support we have provided, but also that no errors or other problems will be fixed when drawn to our attention. Of course, this does mean that others cannot update and improve their own copies of the materials (since we provide all source materials for download); it is just that the NTDC will no longer be able to correct problems and upload fixes to the source content.

Another problem is that as of now, there is no definitive plan for updating of the materials, which means they will gradually become out of date over time. (We did manage to get them updated for Stage 2 of meaningful use but have not been able to update the privacy and security materials for the revisions to HIPAA released in January, 2013.)

Nonetheless, the materials will still be valuable in that they provide a foundation for educators and others who can then update them as they adapt them for their own purposes. In essence, this is the main contribution of the project, which is to provide a higher foundation from which many who teach informatics and health IT can draw. The primary audience for this project always has been educators, and the materials, even if they have some "bugs" or become somewhat out of date, will still provide a base upon which others can build. The materials may also be able to help educational institutions stand up new programs (or enhance existing ones) more easily.

It may well be that there will be resources in the future to allow updating and even expansion of the materials. But for now, educators and everyone else will have plenty to work with, and I am confident that motivated teachers and others will be able to make effective use of the materials.

Other (Non-Physician) Certifications in Informatics, Health Information Technology, and Related Areas

Although the physician subspecialty certification has received the lion's share of the attention when it comes to certification of professionals in informatics, there are actually a great deal of other certification options for other professionals who work in informatics, health information technology (HIT), and related areas. While I am playing a big role in several aspects of the physician certification process, I believe that appropriate professional recognition is important for all who work in informatics. This is demonstrated by the demographics of the enrollment in our informatics educational program at Oregon Health & Science University (OHSU), where only about 30% of the students are physicians.

A number of people, including students in our educational program, have written me lately to ask about these certifications. That has led me to do some research to try to come up with a list and analysis to make sense of it all. I will use this post to provide a list of all certifications in informatics and related areas of which I am aware, and provide some commentary on gaps and limitations.

This process also raises the issue of just how important certification is or should be in our field. In all honesty, I am not sure. It is likely that the real determination of its value will come from employers and others who "vote with their dollars" by making certification part of their hiring and/or promotion criteria. I am not aware that this has happened yet on any widespread basis. In the case of the physician subspecialty, I am certain it will be years before being "board-certified" in clinical informatics will really matter in getting hired or advancing in one's career. But I am equally certain that it eventually will matter a great deal.

From this perspective, let us review what I have learned about certifications in informatics and related disciplines. A first finding is that the certifications fall into two broad categories, which are those that require formal education or training to be eligible to take the certification exam and those that do not.

The certifications requiring formal specific education are in nursing informatics and in health information management (HIM). (And, in a few years, in the new medical subspecialty.) The certification in nursing informatics is provided by the American Nursing Association (ANA). Eligibility for the certification requires having a bachelor's degree in nursing or a "relevant field," along with specified practice and educational experience in both nursing and informatics.

The HIM field has two certifications that require formal education in programs certified by the Commission on Accreditation for Health Informatics and Information Management Education (CAHIIM). These certifications, and the required minimum education, include:

• Registered Health Information Technician (RHIT) - completion of an associate degree in HIM in a CAHIIM-accredited program
• Registered Health Information Administrator (RHIA) - completion of a baccalaureate degree in HIM in a CAHIIM-accredited program

The HIM field also has other certifications that do not require specific education and are focused on specific job functions:

• Certified Coding Associate, Specialist, and Specialist Physician-Based (CCS, CCA, CCS-P) - the historical credential for those who code medical records in ICD-9, CPT-4, and other standard code sets
• Certified Health Data Analyst (CHDA) - those who "acquire, manage, analyze, interpret, and transform data into accurate, consistent, and timely information"
• Certified in Healthcare Privacy and Security (CHPS) - those who design, implement, and administer privacy and security protection programs in healthcare organizations
• Clinical Documentation Improvement Professional (CDIP) - those who "improve and maintain quality information for those involved in healthcare as well as support the integrity of the patient’s health record"

There are a number of other certifications that do not require specific formal education. These certifications are mostly in technical areas and use the "health information technology" moniker.

The Healthcare Information & Management Systems Society (HIMSS) has had a certification called the Certified Professional in Healthcare Information & Management Systems (CPHIMS) that has been in existence for about a decade. I actually took the CPHIMS exam (and passed!) when it first came out. It is billed as a professional certification program for healthcare information and management systems professionals.

HIMSS also has created a new Certified Associate in Healthcare Information & Management Systems (CAHIMS), which is designed for emerging professionals within the industry (i.e., with five or less years of experience). This certification is said to demonstrate knowledge of HIT and management systems, and aims to facilitate entry-level HIT careers. It is also designed to be a career pathway to the CPHIMS credential.

Another certification is the HITPro Program, which was developed as part of the Office of the National Coordinator for HIT (ONC) Workforce Development Program. There are six certifications based on the six HIT workforce roles designated by ONC, which has also funded short-term training programs in community colleges, although the HITPro exams can be taken by anyone. The six workforce roles covered by the exams include:

• Practice workflow and information management redesign specialists
• Clinician/practitioner consultants
• Implementation support specialists
• Implementation managers
• Technical/software support
• Trainers

The curriculum for the HITPro exams is based on another ONC-funded project, which is the ONC Curriculum Development Centers Project in which I have had substantial involvement. The curricular materials are freely available for download from the ONC National Training and Dissemination Center (NTDC). In addition, an exam guide for these exams (as well as the CompTIA exam described below) was recently published.

There are two companies that offer a combination of both training programs or materials, along with certification exams. The first of these is Health IT Certification, which offers certification in four areas, along with (optional) training for each:

• CPHIT - Certified Professional in Health Information Technology
• CPEHR - Certified Professional in Electronic Health Records
• CPHIE - Certified Professional in Health Information Exchange
• CPORA - Certified Professional in Operating Rules Administration

The second company offering both training materials and certification is CompTIA, which provides many IT-related certifications and has recently added the CompTIA Healthcare IT Technician among its certification offerings. This certification covers the knowledge and skills required to implement and support HIT systems in a variety of clinical settings. The CompTIA Web site states that the Healthcare IT Technician certification exam covers "regulatory requirements, organizational behavior, IT operations, medical business operations, and security." The "recommended experience" for the exam is the baseline CompTIA A+ certification or 500 hours of hands-on technical experience in HIT.

Of course, all of the certifications that have been developed, including the clinical informatics physician subspecialty, still leave out some major elements of the informatics workforce. One of these in particular is the group of advanced informatics professionals who have master's and doctoral degrees or other advanced training in informatics. This group includes physicians who are not eligible for the subspecialty, other healthcare professionals who work in informatics, and others who hold advanced degrees or have had advanced training in the field. I have noted in previous postings that AMIA has established a plan for advanced interprofessional informatics certification of other individuals with doctoral degrees, including healthcare doctorates as well as PhDs and other doctoral degrees. I am not aware of official plans of anyone to certify master's-level informaticians, but it would not surprise me to see CAHIIM or even AMIA take the lead on that at some point soon.

This posting has shown that there are clearly many certifications for informatics and related areas. While it is not clear how much employers value or require these certifications, we do know that employers do have a desperate need for skilled talent, perhaps best exemplified by the report last year from the College of Health Information Management Executives (CHIME). This report concluded that the shortage of people with appropriate skills and experience was one of the critical challenges hindering the successful implementation of HIT projects. However, a common complaint heard among new graduates of HIT educational programs is that most positions they desire require experience, which many do not have, since they are just entering the field. Whether formal certifications will hedge against lack of experience is not clear.

Another perspective is to share my own approach to making hiring decisions in my academic department, where I am involved in hiring both those who work in informatics and HIT as well as those who do not (e.g., administrative staff). For each job search, every candidate brings a set of attributes. One of those attributes is their formal education and training. Another is experience in the work and/or setting that the job will entail. There are other factors, such as interviews and recommendations. All hirers apply some sort of calculus, not necessarily a purely quantitative one, to choose the "best" candidate for a given position.

Why is this relevant to informatics certification? It is relevant because in the early days of certification, whether for physicians or others, certification will be one of many attributes considered by a hirer. Other attributes will include experience, and many HIT and informatics personnel bring substantial experience, even if they have little or no formal education or training or any type of certification.

A final perspective I can share is a common question I am asked, which is whether someone should seek formal training in informatics. As the head of an educational program in the field, I obviously have a bias in favor of education. But I do tell those who ask this question that their formal education will be one of those attributes in the calculus of people who may hire them. Everything else being equal (which in reality almost never happens), a job candidate with a credential in informatics (e.g., master's degree, certificate, even the 10x10 course) will have a leg up over someone who does not.

There are clearly many certifications for different people and skills levels in informatics and HIT. It is likely that certification will play a growing role in an individual maintaining competitiveness in the job market. Of course, like all "knowledge fields," informatics will continue to evolve, as we already see the focus shift from implementation to use of data. Therefore the real key for anyone working in informatics is to keep up with the changes and new directions for the field.

Invitation to Join the OHSU Informatics Discovery Lab

As the value of informatics is demonstrated throughout healthcare, personal health, research, and public health settings, there is a growing need for collaboration among academia, industry, healthcare delivery organizations, public health agencies, and others. Academic programs will lose their relevance if its efforts are not aligned with the greater advancement of the disciplines in which they work. There are very high expectations that wide deployment of informatics will significantly improve healthcare and biomedical research from where we are now. This will not be possible without academia closely collaborating with companies, health-related organizations, healthcare delivery systems, public health agencies, and other health stakeholders.

Our department already looks outward as a essential characteristic of our large educational program.  Through our students and graduates, we disseminate knowledge, best practices, and research that advances and improves our understanding of our field. However, as I have written over the years in this blog, another local blog, and our local newspaper, our academic program at Oregon Health & Science University (OHSU) must move beyond its historical work of research funded primarily by federal research grants and education funding by tuition and training grants. We also need to make this move from a business sense, increasing revenue diversification in order to insure the longevity and financial viability of the program.

To this end, we are rolling out a new initiative that we are calling the OHSU Informatics Discovery Lab (IDL). The IDL will be a collaborative environment where students, teachers, and researchers, as well as representatives from healthcare delivery organizations, industry, and philanthropy, can partner to:

• Foster highly relevant informatics research focused on real-world problems
• Uncover commercially viable informatics opportunities
• Accelerate informatics innovation and deployment
• Provide companies and employers greater access to a faculty with both broad and deep informatics expertise and a well-trained informatics workforce pool
• Expand the number and variety of informatics practicum and internship opportunities available for students
• Make available informatics educational opportunities beyond traditional certificate and degree programs for current professionals, members of industry, and others requiring deeper informatics education and experience

The IDL will include informatics scientists, implementers, educators, and students with a unique combination of training, experience, and skills. Areas of expertise already include:

• Healthcare analytics
• Workflow redesign
• Qualitative and quantitative evaluation methodologies
• Natural language processing, machine learning, and information retrieval
• Biomedical terminologies, ontologies, and coding
• Biomedical data structure, representation, and normalization

The talent and skills of the IDL will allow us to bring a wide-ranging, multidisciplinary methodology to problem solving to apply the right approach to the problem. The figure below depicts our methodology.

There are a large variety of problems that need to be solved in health and biomedicine; however, focus is essential to successfully concentrate effort to solve specific problems. We have selected the following challenge areas not only for their importance, but also for their strong fit with our department's unique competencies. Specific projects within these areas will be identified and implemented based on our partners’ needs and interests:

• Predictive Analytics for Healthcare Process Redesign
• Usability for EHR Data and Users
• Population management and Care Coordination Information Systems
• Mobile Telemedicine
• Precision Medicine
• Enhancing Search in Biomedicine

A challenge for launching and sustaining the IDL is of course its business model. Possible partnership models we are exploring include:

• Supporters. Philanthropic donors who help fund lab research activities gain wider access to the lab’s research, facilities, faculty, and staff. Supporters are eligible for a lab-wide NDA, which allows discussion of lab innovations as a whole, instead of on a per-project basis. Supporters are also eligible to send their employees to the lab for sabbaticals and other learning development experiences.
• Sponsored Research. This includes specific projects of interest to both the commercial partner and the mission of the lab. These projects are funded by the commercial partner and managed and executed by the lab in collaboration with the partner, based on a project-specific statement of work and deliverables.
• Fee for Service. Sponsoring organization require DMICE to provide certain services that is strictly generation of data or analysis for a set fee with no ownership of IP by OHSU.
• Collaborative Development. This model comprises exploratory problem-solving intended to create shared intellectual property.
• Consulting. Partners may access the expertise and experience of the lab for small, focused advisement, direction, evaluation, and feedback on their projects or areas of interest.
• Company Start-up. Faculty can contemplate to start companies that can either provide unique resource, services and/ or develop products that have tangible IP to address an unmet need.
• Product/Software Development. In partnership with companies jointly develop software and also be a beta testing site.
• Custom Education. Tailoring OHSU's informatics educational assets for new and innovative purposes.
• Clinical Trial design. Outcomes research and patient information analysis.

There will be many benefits to IDL partners, including:

• Oregon and OHSU are national leaders in healthcare reform. OHSU is Oregon’s largest academic health center, with Oregon and OHSU having the right population size and scope to test and implement new health care ideas and approaches. 
• Access to resources and collaborative faculty expertise from bench to bedside to policy in medicine, nursing, dentistry, and public health informatics. Leverage significant existing investments and dedicated resources focused on biomedical and health informatics research and education at OHSU, including other public and private funding supporting our work. 
• Health simulation environments, analytical models, health management platforms, and domain expertise on deploying marketable technologies for use in the multi-billion dollar health sector. 
• Invaluable community benefit gained by demonstrating the translation of information into meaningful knowledge to cure disease and save lives. 
• Access to health informatics students and post-doctoral fellows. 
• Corporate partner company employees are eligible to use the IDL for sabbaticals and other learning development experiences. 
• Corporate partners are eligible for OHSU Corporate Catalyst Program benefits (donations of $25,000+). 

The IDL development effort is being led by Aaron M. Cohen, MD, MS, Associate Professor and Director of Commercial Partnerships and Collaboration in the Department of Medical Informatics & Clinical Epidemiology (DMICE) at OHSU. Dr. Cohen has been the recipient of several grants from the National Institutes of Health and others, applying text mining and machine learning to problems in biomedicine. Before joining OHSU, Aaron was employed at Intel Corporation as a Senior Staff Software Engineer and Software Architect where he led the development of video teleconferencing systems, 3D browser media, and multimedia and telephony standards. The IDL will be guided by an external steering committee representing the Lab’s many stakeholders: DMICE students and alumni; biomedical researchers; healthcare delivery organizations; healthcare information technology vendors; entrepreneurs and technology transfer experts; and philanthropic organizations.

We look forward to getting feedback and potential partners to work with us as we roll out the IDL. We invite you to join us on this journey.

Different Strokes for Different Social Media Folks

This week has been a milestone for this blog, with the number of page views surpassing the 100,000 mark since its inception in March, 2009. This achievement gives me a chance to reflect on my use of social media, which seems to be different than for others. Maybe social media is just like anything else in life, with different people preferring differing aspects and uses of it.

Clearly I have enjoyed being a blogger. This blog has provided me a nice platform from which to share my thoughts and views with a worldwide audience. As I have noted before, I am not a stream of consciousness blogger, feeling compelled to post things continuously, such as every day. Rather, I prefer that my postings carefully reflect thoughts and ideas on specific topics.

Another social media activity I enjoy is Facebook. I have three main networks on Facebook, and I enjoy seeing them interact with my personal and professional life. These networks include my professional colleagues, my family and friends, and my high school classmates. Facebook is also a great medium for sharing and annotating photos and other digital artifacts.

Two social media activities I personally find less valuable to myself are Twitter and LinkedIn. I know this is at odds with some dear friends and colleagues. However, tweets just seem too short (I often have more to say than can be expressed in 140 characters!) and fleeting (it seems you either catch something in the Twitter stream or never see it again) to sustain my interest. Sometimes I try to get involved in the Twitter dialogue at conferences, but soon find it distracting to try to otherwise participate in the meeting (whose primary value is usually the direct personal interaction). I have the most fun with it when I use it to editorialize about presentations at those meetings, but I find it difficult to sustain any sort of dialogue when doing that.

As for LinkedIn, while I am sure it is highly valuable for some people, I find that my major interaction with it is to receive requests for connections and endorsements. I am happy to connect with anyone on LinkedIn, but I have yet to find value in the hundreds of connections I have made. I also do not like to make generic LinkedIn endorsements, instead preferring to serve as real references for colleagues and current or former students when they need it for specific opportunities.

I know that other people have different preferences for social media, and perhaps my own preferences will change over time. And of course, it is likely that the social media tools and sites will change over time, or that new ones will emerge. For now, however, I will keep blogging and Facebooking while still trying to determine the value of other social media.

Data Mining Systems Improve Cost and Quality of Healthcare - Or Do They?

Several email lists I am on were abuzz last week about the publication of a paper that was described in a press release from Indiana University to demonstrate that "machine learning -- the same computer science discipline that helped create voice recognition systems, self-driving cars and credit card fraud detection systems -- can drastically improve both the cost and quality of health care in the United States." The press release referred to a study published by an Indiana faculty member in the journal, Artificial Intelligence in Medicine [1].

While I am a proponent of computer applications that aim to improve the quality and cost of healthcare, I also believe we must be careful about the claims being made for them, especially those derived from results from scientific research.

After reading and analyzing the paper, I am skeptical of the claims made not only by the press release but also by the authors themselves. My concern is less about their research methods, although I have some serious qualms about them I will describe below, but more so with the press release that was issued by their university public relations office. Furthermore, as always seems to happen when technology is hyped, the press release was picked up and echoed across the Internet, followed by the inevitable conflation of its findings. Sure enough, one high-profile blogger wrote, "physicians who used an AI framework to make patient care decisions had patient outcomes that were 50 percent better than physicians who did not use AI." It is clear from the paper that physicians did not actually use such a framework, which was only applied retrospectively to clinical data.

What exactly did the study show? Basically, the researchers obtained a small data set for one clinical condition in one institution's electronic health record and applied some complex data mining techniques to show that lower cost and better outcomes could be achieved by following the options suggested by the machine learning algorithm instead of what the clinicians actually did. The claim, therefore, is that if the data mining were followed by the clinicians instead of their own decision-making, then better and cheaper care would ensue.

As done in many scientific papers about technology, the paper goes into exquisite detail about the data mining algorithms and the experiments comparing them. But the paper unfortunately provides very little description about the clinical data itself. There is a reference to another paper from a conference that appears to describe the data set [2], but it is still not clear how the data was applied to evaluate the algorithms.

I have a number of methodological problems with the paper. First is the paucity of clinical details about the data. The authors refer to a metric called the "outcomes rating scale" of the "client-directed outcome informed (CDOI) assessment." No details are provided as to exactly what this scale measures or how differences in measurement correlate with improved clinical outcome. Furthermore, the variables of the details of care for the patient that the data mining algorithm supposedly outperforms are not described either. Therefore anyone hoping to understand the clinical value that this approach is claimed to have improved is not able to do so.

A second problem is that there is no discussion about the cost data or what cost perspective (e.g., system, clinician, societal, etc.) is taken. This is a common problem that plagues many studies in healthcare that attempt to measure costs [3]. Given the relatively modest amounts of money spent on the care that is reported in their results, amounting only to a few hundred dollars per patient, it is unlikely that the data includes the full amount of the costs of treatment for each patient, or over an appropriate time period. If my interpretation of the low value of the cost data is correct (which is difficult to discern from reading the paper due, again due to lack of details), the data do not include the cost of clinician time, facilities, or longer-term costs beyond the time frame of the data set. If that is indeed the case, then it would be particularly problematic for a machine learning system, since such systems make inferences limited only to the data that is provided to the model. Therefore if poor data is provided to the model, its "conclusions" are suspect. (This raises a side issue as to whether there is truly "artificial intelligence" here, since the only intelligence applied by the system is the models developed by their human creators.)

A third concern is that this is a modeling study. As every evaluation methodologist knows, modeling studies are limited in their ability to assign cause and effect. There is certainly a role in informatics science for modeling studies, although we saw recently that such studies have their limits, especially when revisited over the long run. In this study, there may have been reasons for the clinicians following the more expensive path or confounding reasons why such patients had worse outcomes, but they cannot be captured by the approach used in this study.

This is related to the final and most serious problem of the work, which is that the modeling evaluation is a very weak form of evidence to demonstrate the value of an intervention. If the authors truly wanted to show the benefits of the system and approach they developed, they should have performed a randomized controlled trial that compared their intervention with an appropriate control group. This would have led to the type of study that the blogger mentioned above erroneously described this to be. Such a study design would assess some of the more vexing problems we face in informatics, such as whether the advice coming from a computer will change clinician behavior. Or, when such systems are introduced into the "real world," whether the "advice" provided will prospectively lead to better outcomes.

I do believe that the kind of work addressed by this paper is important, especially as we move into the area of personalized medicine. As eloquently described by Stead and colleagues, healthcare will soon be reaching the point where the number of data points required for clinical decisions will exceed the bounds of human cognition [4]. (It probably already has.) Therefore clinicians will require aids to their cognition provided by information systems, perhaps one like that described in the study.

But such aids require, like everything else in medicine, robust evaluative research to demonstrate their value. The methods used in this paper may indeed be the methods to provide this value, but the implementation and evaluation described miss the mark. That miss is further exacerbated by the hype and conflation the ensued after the paper was published.

What can we learn from this paper and its ensuing hype? First, bold claims require bold evidence to back them up. In the case of showing value for an approach in healthcare - be it test, treatment, or informatics application - we must use evaluation methods that provide best evidence for the claim. That is not always a randomized controlled trial, but in this situation, it would be, and the modeling techniques used are really just preliminary data that (might) justify an actual clinical trial. Second, when we perform technology evaluation, we need to describe, and ideally release, all of the clinical data so that others can analyze and even replicate the results. Finally, while we all want to disseminate the results of our research to the widest possible audience, we need to be realistic in explaining what we accomplished and what are its larger implications.

References

[1] Bennett, C. and K. Hauser (2013). Artificial intelligence framework for simulating clinical decision-making: a Markov decision process approach. Artificial Intelligence in Medicine. Epub ahead of print.
[2] Bennett, C., T. Doub, A. Bragg, J. Luellen, C. VanRegenmorter, J. Lockman and R. Reiserer (2011). Data mining session-based patient reported outcomes (PROs) in a mental health setting: toward data-driven clinical decision support and personalized treatment. 2011 First IEEE International Conference on Healthcare Informatics, Imaging and Systems Biology (HISB 2011), San Jose, CA. 229-236.
[3] Drummond, M. and M. Sculpher (2005). Common methodological flaws in economic evaluations. Medical Care. 43(7 Suppl): 5-14.
[4] Stead, W., J. Searle, H. Fessler, J. Smith and E. Shortliffe (2011). Biomedical informatics: changing what physicians need to know and how they learn. Academic Medicine. 86: 429-434.

AMIA Clinical Informatics Board Review Course Announced

This week, the American Medical Informatics Association (AMIA) released the details of the Clinical Informatics Board Review Course (CIBRC), for which I will serving as Course Director. I am excited to see this come to fruition, and look forward not only to these course offerings but also the expansion of the program as other certifications in informatics for non-physicians become reality in the years ahead.

The course will be offered four times in person before the first offering of the certification exam in October. There will also be an online version of the course that can be accessed in tandem with the live courses or taken alone. A question bank of practice questions will also be made available. The four offerings of the course will be:

• April 12-14 - Bethesda, MD (registration has opened for this course and will soon be available for the others)
• June 7-9 - Philadelphia, PA
• August 9-11 - Portland, OR (I am thrilled to have one offering of the course in my own city!)
• September 7-9 - Rosemont (Chicago), IL

The details of the course are provided on the AMIA Web site:

• Overview of the course
• Course Design
• Faculty
• Frequently asked questions
• History of the course and subspecialty

One question those contemplating pursuing the certification exam might ask is whether this review course is the best preparation for them. I believe the course will be best for those already familiar with the content of the field. The review course material will necessary be at a high level, aiming to provide a broad overview of what is on the exam so that the potential exam candidate can get the big picture and determine where they need to shore up their knowledge.

For those with less knowledge of the field, a better approach might be to start with a course that more fundamentally builds their knowledge base. The 10x10 ("ten by ten") course is option, although even it is only a single course and may not be enough for those with little or no formal training in the field. The next Oregon Health & Science University (OHSU) offering of its 10x10 course runs from April through July; registration is available on the AMIA Web site. For those individuals desiring more training, a more comprehensive course of study leading to a Graduate Certificate or master's degree may be a better option. AMIA has a database of educational programs, which includes our program at OHSU.

The rollout of the course is an exciting event. For convenience, here is an index to some of my important previous postings on the clinical informatics subspecialty:

• Eligibility for the subspecialty (January, 2013)
• Some challenges for building capacity of the subspecialty (September, 2012)
• Mapping the OHSU informatics program content to the core content (July, 2012)

For those who are not physicians and desire certification in informatics, do not despair! AMIA and others are at work developing plans for certification of non-physicians (as well as physicians who not eligible for the board certification, which requires an active primary specialty certification). In the meantime, I look forward to seeing those who are eligible in one of the course offerings!

From HITECH to Accountable Care: A Student and Workforce Development Program Success Story

This year's State of the Union address by President Barack Obama noted that controlling the costs of healthcare, particularly of Medicare and Medicaid, is a critical element of addressing the government's debt problem, especially in the long run. A key element of this approach, as the President noted, will be new models of care delivery. As he stated, "medical bills shouldn’t be based on the number of tests ordered or days spent in the hospital; they should be based on the quality of care that our seniors receive." One of the key elements for operationalizing this approach is the development of accountable care organizations (ACOs) [1], whose goals are to achieve Dr. Donald Berwick's "triple aim" of improved health, improved healthcare delivery, and reduced cost [2].

To this end, President Obama invited Oregon's Governor, Dr. John Kitzhaber, to be present at his address. Dr. Kitzhaber's presence acknowledged his leadership and innovation in healthcare reform that is taking place in Oregon. With help from the Obama Administration, Oregon is revamping its entire Medicaid program under a new brand of ACOs known as Coordinated Care Organizations (CCOs). The Oregon CCOs will provide value-driven coordinated care for Oregon Medicaid patients in pursuit of the "triple aim." As with all ACOs, managing information, including that beyond the electronic health record (EHR), will be critical for the success of Oregon's CCOs. Indeed, a recent post by Dr. John Halamka posited that "ACO = HIE + analytics," a shorthand way of stating that ACOs (and CCOs) will require robust health information exchange (HIE) and data analytics.

The importance of health information technology (HIT) to accountable care was recognized in the Health Information Technology for Clinical and Economic Health (HITECH) Act when it was passed in 2009 [3]. Indeed, one of the original roles for HITECH was to serve as a "down payment" for healthcare reform [4].

I am pleased to report of one recent instance, small but significant, where HITECH funding has indeed resulted in an outcome that is helping healthcare reform and accountable care. It is the story of two individuals who pursued our University-Based Training (UBT) program and have both been hired by Health Share of Oregon, the CCO for the Portland, Oregon tri-county region.

Isolde Knaap worked in the State of Oregon's Department of Human Services for over 20 years in various positions as a data analyst/system developer/research analyst. During that time, she built a wealth of experience managing data and systems in public health and child welfare. Aiming to advance her skills and career more into the HIT realm, she applied and was accepted into the Oregon Health & Science University (OHSU) Graduate Certificate Program, funded by the UBT training grant. Her previous educational background include a Bachelor of Arts in Modern Languages with a secondary teaching certificate.

After graduation from the program, Isolde was hired by Health Share of Oregon as a Senior IT Project Manager. Like all CCOs, Health Share Oregon is tasked to improve health outcomes and reduce health cost through collaboration with previously disparate health care providers. Isolde's position will help align IT processes so that the care delivery transformation activities of Health Share Oregon achieve the triple-aim goals while also achieving administrative simplification.

One of the courses in her program of study Isolde found most useful was Introduction to Standards and Interoperability. This gave her an appreciation for the challenges of health information exchange, a critical function for CCOs who share financial accountability for providing care for their patients. She stated that she was asked in various interviews what she knew about EHRs, and replied that she could confidently state that she had taken the courses, Clinical Information Systems and Clinical Information Systems Laboratory, and later served as a teaching assistant in the latter. She also purposefully chose a practicum with a local Veteran's Administration hospital to get exposure to their VistA EHR (which was used in the Clinical Information Systems Laboratory course).

Charles Sorgie is another UBT graduate who has been hired by Health Share of Oregon, serving as a Senior IT Business Analyst. Charles has a long history of work in the IT field as a software developer, designer, and architect. He had no experience in the healthcare domain but had developed a documentation management system to support electronic design automation that was subsequently adopted by the aerospace industry to manage their maintenance documentation. In addition, he was involved more recently in the process modeling and enterprise modeling fields, which led him to become interested in ways to organize and model the processes, resources, and work products involved in complex business interactions. His educational experience was a Master of Science in Computer Science.

In his position with Health Share of Oregon, Charles is involved in the gathering of stakeholder requirements and acting as a coordinator and liaison between those stakeholders and the IT teams tasked with the deployment of solutions to satisfy those requirements. He notes that the OHSU UBT program proved him a basic understanding of the challenges in the support of clinical processes, their privacy and security requirements, their workflow, and the standards used to support that workflow (e.g., HL7). It also gave him a background to the basic terminology used (e.g., personal health information, or PHI) as well as the challenges surrounding the business processes that support those clinical processes. All of this helps him be more productive in his current position.

Both Isolde and Charles report to Daniel Dean, the Chief Information Officer (CIO) of Health Share of Oregon. Their connection to Mr. Dean was made possible by another resource provided by our UBT grant, which is Virginia Lankes, who is the Career Development Specialist that the grant enabled us to hire. The connections Ms. Lankes had nurtured with Mr. Dean made him aware of our two students as they were completing their studies.

Isolde and Charles are thus examples of how the UBT program has developed the HIT workforce and how the HITECH program is contributing to healthcare reform. The two of them also provide a refreshing counterexample to the common adage that one must have a clinical background to succeed in biomedical and health informatics. As I have written, the work of informatics is shifting from implementation to data, and their experience and expertise in using data seems to have played an important role in their new positions.

The story of Isolde and Charles provide more examples of how the UBT funding of our educational program has helped individuals advance their careers and added jobs to the economy. Their experiences build on the successful outcomes that other students in the program have had, as I documented in 2011 and 2012.

Postscript: Sure enough, the day after this posting, an article appeared in the New England Journal of Medicine, describing the Oregon CCO program [5]. It is a well-written overview and freely available, so I will add this postscript to provide this additional information.

References

1. Berwick, D. (2011). Making good on ACOs' promise--the final rule for the Medicare shared savings program. New England Journal of Medicine 365: 1753-1756.
2. Berwick, D., T. Nolan and J. Whittington (2008). The triple aim: care, health, and cost. Health Affairs 27: 759-769.
3. Blumenthal, D. (2010). Launching HITECH. New England Journal of Medicine 362: 382-385.
4. Blumenthal, D. (2009). Stimulating the adoption of health information technology. New England Journal of Medicine 360: 1477-1479.
5. Stecker, E. (2013). The Oregon ACO experiment — bold design, challenging execution. New England Journal of Medicine Epub ahead of print.

The Informatics Professor on Video

Some video interviews of me have appeared on the Web. They are part of some interesting and worthy sites.

The first video was an appearance on the Web-based show, Health IT Live!, which features a host of informatics experts from around the world. I took part in Healthcare IT Live! Episode #6. The interview discussed a variety of topics, mostly related to my research and educational work in biomedical and health informatics.

I was also recently interviewed for a site called AskimoTV, which purports to provide access to experts, initially via video interviews and later to those wanting one-on-one consultations. My interview was done in three topical segments:
1. Biomedical and Health informatics: Improving healthcare through information
2. Barriers to Retrieving Patient Information from Electronic Health Record
3. Building A Health Informatics Workforce In Developing Countries

These interviews demonstrate the simple power of the Web and its ability to disseminate information in ways we could not have imagined a couple decades ago.

Marketing the OHSU Informatics Program

The Oregon Health & Science University (OHSU) Biomedical Informatics Graduate Program is undertaking a new marketing campaign. This campaign is, of course, a business decision, but I also find it a great opportunity to spread the word about our program as well as careers in the profession of informatics.

We are especially interested in touting the benefits of education and careers in informatics for younger people. The mid-career individuals who enter our program mostly know well the problems in healthcare and how informatics can address them. But we find that younger people, probably because of their less experience with the healthcare system (and its dysfunction), are not aware of how informatics can not only lead to a rewarding career but also benefit health, improve healthcare delivery, and advance basic and clinical research. There are also opportunities in bioinformatics and computational biology of which they may actually be more aware. The graphic below shows our message.

As such, the focus of this marketing campaign aims at younger people. Not that we do not want to continue to attract the more common mid-career healthcare or other professional who enters the clinical informatics track of our program, but our marketing focus is aimed at those who are younger and probably have little knowledge of healthcare, biomedical research, or genomics. The campaign aims at both of the major tracks in our program, clinical informatics and bioinformatics.

Another reason for the challenge in marketing our program is the complexity of its tracks, degrees and certificates, and other aspects.

An additional message we are promoting concerns the value of our program. We note that while our tuition is comparable with most other programs, we feature a large and world-renowned faculty, great longevity of over 20 years, and nearly 500 alumni who have successfully obtained employment in a wide variety of settings.

The components of the marketing campaign include:

• A banner ad on Radiolab.org - This Web site, run by New York Public Radio, features podcasts on a variety of topics around science and arts. In addition to having the banner add to click-through to our Web site, our program is also acknowledged at the beginning and in the middle of each podcast (here is an example).
• Print ads in west coast college newspapers - Print ads are a challenge these days, as readership of college newspapers has declined as with all print media. But such ads do still reach our target audience, and will be supplemented by our usual attendance of graduate school fairs and other events at these colleges. The ad we are running is shown above.
• Google Adwords campaign
• Ad placement on Web pages via Quantcast
• Ongoing Webinars - I gave our first Webinar on January 31, 2013, which is available for viewing, along with the slide deck.

Many of these components send interested people to a landing page for the program.

We will see how this translates to new student enrollment, especially among the younger people who will someday be users, implementers, and leaders of the new data-rich, information-driven healthcare system that guides our vision now.

Implementing the Learning Healthcare System Can Be Facilitated Using the Principles of Evidence-Based Medicine

The enthusiasm for big data and for the use of analytics and business intelligence with that data is reaching a fevered pitch. I share that enthusiasm, but also know from both my clinical and my informatics experience that knowledge will not emanate just by turning on the data spigot from the growing number of electronic health record (EHR) systems now in operational use. However, if we approach the problem properly, I believe we can achieve the goals of the learning healthcare system as eloquently laid out in various reports from the Institute of Medicine (IOM) [1, 2].

One sensible approach was published recently in Annals of Internal Medicine [3]. The authors were from Group Health Cooperative in Seattle, a leader in the use of data and information systems to improve the quality and outcomes of care. The paper is summarized well by a figure that shows a continuous cycle of design-implementation-evaluation-adjustment of improved care, with interaction with the external environment through scanning for identification of problems and solutions and dissemination to share what has been learned in their setting.

A complementary approach to learning from EHR and other clinical data can be to apply the basic approach of evidence-based medicine (EBM) [4]. In some ways, EBM is antagonistic to EHR data analytics, with the former giving the most value to evidence from controlled experiments, especially randomized controlled trials (RCTs), while the latter makes use of real-world observational data that may be incomplete, incorrect, and inconsistent.

But I maintain that we can look to the process of EBM to guide us in how to best approach the "evidence" of EHR data analytics and the learning health system. EBM is not just about finding RCTs. Rather, it uses a principled approach to find and apply the best evidence to make clinical decisions. In particular, EBM done most effectively uses four steps:

1. Ask an answerable question
2. Find the best evidence
3. Critically appraise the evidence
4. Apply it to the patient situation

When I teach EBM, I emphasize that the first step of asking an answerable question may be the most important. It is not enough, for example, to ask if a test or treatment works. Rather, we need to know at a minimum whether it works relative to some alternative approach in a particular patient population or setting. This same approach is obviously necessary in the learning health system. Just as RCTs do not inform us passively, neither will EHR data analytics approaches.

In the second step, the principle from EBM is very much the same, even if the techniques of obtaining evidence are very different. The "evidence" in the case of the learning health system is the data in EHR and other systems that, as noted above, may be incomplete, incorrect, and inconsistent. We therefore need to determine if we have the proper data and, if so, whether it can applied to answer our question.

For the third step, just as with EBM, we must critically appraise our evidence. Can we trust the inferences and conclusions from the data? Are there confounding variables of which we may not be aware? This may be critical with EHR data where assignment of cause and effect could be difficult, if not impossible. The solution likely comes back to asking the right question, i.e., one we can have confidence in the correct answer.

Finally, we have to ask, can the data be applied in our setting? Just as some RCTs answer questions in patient populations very different from those of the clinician making decisions, it must be ascertained if the results obtained from this approach can be applied to a specific patient or setting.

The growing quantity of clinical data in operational clinical systems provides a foundation for the learning healthcare system. However, we must approach the questions we ask and how we answer them with caution and a sound methodology. The approach of EBM offers a framework for carrying out this very different but complementary work.

References

1. Eden J, Wheatley B, McNeil B, and Sox H, eds. Knowing What Works in Health Care: A Roadmap for the Nation. 2008, National Academies Press: Washington, DC.
2. Smith M, Saunders R, Stuckhardt L, and McGinnis JM, Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. 2012, Washington, DC: National Academies Press.
3. Greene SM, Reid RJ, and Larson EB, Implementing the learning health system: from concept to action. Annals of Internal Medicine, 2012. 157: 207-210.
4. Straus SE, Glasziou P, Richardson WS, and Haynes RB, Evidence-Based Medicine: How to Practice and Teach It, 4e. 2010, New York, NY: Churchill Livingstone.

The Health IT "Grand Experiment": Mid-Study Check-Up

It seems that whenever there is something negative about health information technology (HIT) in the popular press, I get emails from people inside and outside the field, asking "what's wrong?" A case in point is a recent article in the New York Times [1], reporting on a "negative" point of view from two researchers from the RAND Corp. that was published in the journal Health Affairs [2]. One of the interesting twists of the Health Affairs piece was that it was written by researchers from RAND, the same organization that published a modeling study in 2005 that concluded that investment in HIT could provide potential annual savings to the healthcare system of $142-$371 billion [3]. About the same time, another model-based study from the Center for Information Technology Leadership (CITL) found similar potential savings [4]. This data in part led to the inclusion of HIT in the Health Information Technology for Economic and Clinical Health (HITECH) Act, the program from the American Recovery and Reinvestment Act (ARRA) of 2009, also known as the economic stimulus bill, that invested up to $29 billion in the adoption and "meaningful use" of the electronic health record (EHR) and other HIT [5].

What can we conclude from this recent publication and reporting about it in the popular press? As always, it is best to look at exactly what has been claimed, what evidence supports it, and where it fits in the larger picture of this topic.

The 2005 RAND study modeled savings that could occur from HIT adoption [3]:

• Reduced adverse drug events that extend hospital length of stay in the inpatient setting and avoid hospitalization in the outpatient setting
• Increased used of cost-effective immunizations and screening interventions
• Improved efficiency of chronic disease management

Their model, however, noted that HIT adoption alone would not be enough; also required would be "interconnected and interoperable systems" that were "adopted widely" and "used effectively." This had an implicit assumption of change in the healthcare delivery system away from payment for volume toward payment for value. The paper describing this work was published in Health Affairs, along with several dissenting views [6-8]. An analysis by the Congressional Budget Office also took issue with the conclusions [9].

The CITL study used a somewhat similar modeling approach and drew similar conclusions. The CITL model focused on different types of health information exchange (HIE), from simple transmission of documents to full semantic interoperability of EHR systems. The latter approach was shown to achieve the most benefit, up to $77 billion per year.

Can we assess the correctness of these modeling studies, now that we have substantially increased EHR adoption through HITECH? The recent paper from RAND noted that the question is not simple to answer, but that HIT probably has fallen short of its promises, especially in terms of reducing costs [2]. Of course, one of the challenges in answering the question of cost-reduction is that it is difficult to attribute avoidable cost in the healthcare system. We do know that healthcare costs have reduced their rate of growth in the last few years, probably mainly due to the economic recession [10]. But we cannot know for sure how much of that reduction might be due to HIT adoption.

But an even bigger reason why we cannot know if the modeling studies are true is that we have achieved the kind of HIT environment that these studies assumed in the development of their models. The original RAND study assumed, as noted above, interconnected and interoperable systems that were adopted widely and used effectively. The authors of the new RAND paper note that HIT failure has come in large part because of failure to reach those assumption. In particular,

1. We do not have interconnected and interoperable systems. In part, this is because many EHR systems are still closed and proprietary. In addition, HIE efforts are still early and nascent.
2. We also do not have wide adoption yet of systems, especially advanced systems. While HITECH has led to increased adoption, there is still a long ways to go.
3. And probably the biggest shortcoming has been lack of EHRs being used effectively. The adoption incentives in Stage 1 of meaningful use focus (by design) on building the data foundation. More effective use will come based on that foundation in Stage 2 and beyond.

The RAND authors conclude that the potential of HIT in reducing costs is still very real, but critical focus on interoperability, patient-centeredness, and usability must be prioritized.

Therefore my view echoes that of the RAND researchers in the new Health Affairs piece, which is that yes, HIT has not yet delivered on its promise to improve efficiency and reduce cost in the healthcare system. But the proposition that it inherently is not able to do so is also not known. As such, if we hope for that improvement, the grand experiment should go on. There is no question that the required time will be longer, the resources required will be larger, and the cultural change will be more difficult. There is also quite valid concern that there are some untended consequences of the staged approach in HITECH, which may be locking clinicians and hospitals into monolithic systems that are difficult to use and expand. I sympathize with the notion of current market-leader systems locking us into an "EHR trap," where the EHR should not be a monolithic application but instead a platform on top of which we can build apps that provide innovative functions and/or make new use of the data [11].

Over the last few years, I have ended many a talk on informatics noting that a "grand experiment" in our field was taking place, with the complete results unlikely to be years away. This study can be viewed as a mid-study assessment, and we can conclude that the benefits have not yet accrued, but that it may be too early to conclude that they will not occur. Although I agree that we probably need some mid-course correction in our approach, I also argue that we cannot go back nor should we end the experiment prematurely. We also must remember the motivations for implementing HIT and reforming healthcare in the first place, which is the error-prone and financially dysfunctional existing system, which both undermines competitiveness of US companies globally due to high employee healthcare costs as well as threatening to bankrupt the US government through unsustainable Medicare cost increases.

References

1. Abelson R and Creswell J, In Second Look, Few Savings From Digital Health Records, New York Times. January 10, 2013. http://www.nytimes.com/2013/01/11/business/electronic-records-systems-have-not-reduced-health-costs-report-says.html.
2. Kellermann AL and Jones SS, What will it take to achieve the as-yet-unfulfilled promises of health information technology? Health Affairs, 2013. 32: 63-68.
3. Hillestad R, Bigelow J, Bower A, Girosi F, Meili R, Scoville R, et al., Can electronic medical record systems transform health care? Health Affairs, 2005. 24: 1103-1117.
4. Pan E, Johnston D, Walker J, Adler-Milstein J, Bates DW, and Middleton B, The Value of Healthcare Information Exchange and Interoperability. 2004, Center for Information Technology Leadership: Boston, MA.
5. Blumenthal D, Launching HITECH. New England Journal of Medicine, 2010. 362: 382-385.
6. Himmelstein DU and Woolhandler S, Hope and hype: predicting the impact of electronic medical records. Health Affairs, 2005. 24: 1121-1123.
7. Goodman C, Savings in electronic medical record systems? Do it for the quality. Health Affairs, 2005. 24: 1124-1126.
8. Walker JM, Electronic medical records and health care transformation. Health Affairs, 2005. 24: 1118-1120.
9. Orszag P, Evidence on the Costs and Benefits of Health Information Technology. 2008, Congressional Budget Office: Washington, DC, http://www.cbo.gov/ftpdocs/91xx/doc9168/05-20-HealthIT.pdf.
10. Hartman M, Martin AB, Benson J, and Catlin A, National health spending in 2011: overall growth remains low, but some payers and services show signs of acceleration. Health Affairs, 2013. 32: 87-99.
11. Mandl KD and Kohane IS, Escaping the EHR trap--the future of health IT. New England Journal of Medicine, 2012. 366: 2240-2242.