Generative Artificial Intelligence – Know and Manage the Risks

Generative artificial intelligence (genAI) is the latest buzz in the healthcare arena. Since the November 2022 release of ChatGPT, the number of genAI applications on the market has increased exponentially. Forecasters predict that by 2032, the genAI market could grow to $1.3 trillion. 

Artificial intelligence (AI) has been in use for decades, and most of us interact with it on a daily basis. Traditional AI focuses on using learned intelligence and pattern recognition to perform specific tasks, like making recommendations for movies to watch on a streaming platform or using a chatbot to answer online questions. Traditional AI systems do not create anything new—they are trained to follow specific rules and recognize patterns to perform a function.

GenAI is the newest subset of AI. It differs from traditional AI in that it generates entirely new content (text, images, music, audio, video) based on large language models that are trained on enormous amounts of specific information. Furthermore, genAI has the capacity to improve efficiency and personalize experiences across many different industries.
 
In the healthcare arena, genAI has the potential to vastly improve the quality of care, lower costs, and enhance the practitioner and patient experience. The healthcare functions that may benefit from genAI include:

• Documentation support: GenAI assisted documentation uses ambient recording of patient-practitioner conversation to create patient histories, perform record/medication reconciliation, and draft clinical documentation.
• Clinical decision support (CDS): GenAI CDS tools can help practitioners enhance the diagnostic process by alerting them to subtle changes in the patient’s condition and developing specific treatment plans.
• Patient support: Chatbots can interact with patients to collect relevant clinical information, respond to their questions and requests, help them navigate care, and direct them to accurate clinical information. 

Despite the excitement surrounding the use of genAI, it is prudent to advance cautiously while considering all the potential risks associated with this technology. These risks include bias, accuracy issues, privacy concerns, adverse impact on patient autonomy, and liability. Other challenges include integration of genAI into organizations’ IT systems and infrastructure and its associated costs. Consider the following when evaluating the role that genAI might play in your organization:

• Evaluate the application. Conduct a thorough risk assessment of the system and vendor, including an evaluation and test of the underlying model in your own environment. Ensure the genAI application will integrate into your current IT system and infrastructure and that you can safely incorporate it into existing workflows.
• Draft comprehensive contracts. Work with an attorney to draft a contract that outlines the requirements for the appropriate use of the application, the collected data, and associated costs, and who will assume liability in given scenarios. Include provisions addressing compliance with federal and state laws concerning protection of patient privacy and patient information.
• Establish a governance committee. Assemble an interdisciplinary team to evaluate all new technology, oversee existing technology, and establish policies, procedures, and workflows. Include representatives from leadership, clinical services, engineering, patient safety, and risk management.
• Train users. Educate all practitioners and staff on the AI application and stress the importance of reporting unusual events. Emphasize that AI augments, but cannot replace, medical decision-making and communication with patients.
• Obtain consent. When using genAI applications for documentation support, obtain written consent from the patient. Explain to the patient the purpose, benefits, and risks of using AI scribes. Provide information on how data will be collected, stored, and protected. Ensure patients are aware of the technology’s limitations. Develop patient education materials to supplement the conversation with the patient. Provide an opt-out method for patients who refuse genAI documentation support. Respect their decision and provide alternative documentation methods.
• Know the law. Because some genAI applications collect clinical data by recording patient-practitioner conversations, wiretapping laws, which vary by state, may apply. Some states are “two-party” states, which means that before a private, in-person conversation can be recorded, all parties must be aware of and consent to the recording. Consult an attorney to ensure applicable laws are followed.
• Ensure human oversight. Develop clear policies and procedures that require review of generated content before final sign off. Monitor and audit for adherence to applicable policies and procedures.
• Monitor. Continually review the system’s performance in partnership with the vendor. Understand that AI performance may change or degrade over time. Be aware that minor changes in clinical protocols can greatly impact AI model performance.

AI may help solve some of the many challenges we face in healthcare today. Nevertheless, organizations must acknowledge and manage the risks associated with this technology so that patients are safe, care is equitable, and patient rights are respected and honored. 



Copyrighted. No legal or medical advice intended. This post includes general risk management guidelines. Such materials are for informational purposes only and may not reflect the most current legal or medical developments. These informational materials are not intended, and must not be taken, as legal or medical advice on any particular set of facts or circumstances.