Using GPT-4o reasoning to transform cancer care

Color Health uses the reasoning capabilities of GPT-4o to help doctors transform cancer care | OpenAI

Color Health

Loading…

Share

Color Health⁠ is working with OpenAI to pioneer a new way of accelerating cancer patients’ access to treatment. Their new copilot application uses GPT‑4o to identify missing diagnostics and create tailored workup plans, enabling healthcare providers to make evidence-based decisions about cancer screening and treatment.

Color has been working to improve access to healthcare for a decade, serving more than 7 million patients since it was founded. In 2023, they partnered with the American Cancer Society to help employers and health plans take control of cancer—the second most common cause of death in the United States and the leading driver of American healthcare costs.

Color’s copilot is helping clinicians create customized, comprehensive plans to start cancer treatment

Color Health uses OpenAI’s APIs to integrate patient medical data with clinical knowledge. The outcome is a copilot application that creates customized, comprehensive treatment plans for providers to review and use in their patient care.

“Color’s vision is to make cancer expertise accessible at the point and time when it can have the greatest impact on a patient’s healthcare decisions,” says Othman Laraki, CEO of Color Health.

"As a healthcare company, technology that improves access and equity has to go hand-in-hand with technology that supports patient safety and privacy. OpenAI's HIPAA-compliant data protection standards are key."

The copilot application’s output is analyzed by a clinician at every step and, if need be, modified before being presented to the patient. It works as follows:

1. It extracts, processes, and normalizes patient information, such as family history and individual risk factors, along with clinical guidelines and data from trusted sources. The Color team was particularly impressed with GPT‑4o’s ability to extract and normalize information that was buried within pages of inconsistently structured and phrased information, often in different formats, such as with PDFs or clinical notes. 2. Using this data, it answers key questions like, “What screenings should the patient be doing?” to identify missing diagnostics and generate a personalized screening plan. It also generates documentation required to complete any diagnostic workups, such as medical necessity documents and insurance pre-authorizations. 3. The clinician-in-the-loop evaluates the output, which includes source information. The clinician can edit the copilot’s output, which also helps refine future iterations. 4. Once the clinician-in-the-loop is satisfied with the result, they can add the information to the patient’s existing treatment plan.

Missed screenings and delayed cancer treatment impact patient outcomes

Screening, diagnosis, and treatment for cancer is notoriously complex and time-consuming. And every delay makes a difference: patients whose treatments are delayed by just four weeks face a 6–13% higher risk of mortality⁠.

Screening needs are also often highly individualized. More than a third of Color’s patients, for example, require earlier, different screening approaches based on individual risk factors not addressed by standard guidelines. “I've witnessed the complexities of developing personalized cancer screening plans for my high-risk patients,” says Dr. Keegan Duchicela, a primary care physician at Color. “The guidelines are constantly evolving, and individual risk factors aren't always immediately clear.”

Beyond screening, diagnostic workups create more challenges. Documenting and performing a single patient’s diagnostic workup can take weeks, with the majority of patients arriving at their first oncology appointment without a complete workup. “Today, there are real gaps in oncology care based on where a patient receives initial diagnosis,” says Dr. Allison Kurian, a professor at Stanford University School of Medicine and clinically active oncologist. “Many of my patients require weeks to complete all of the tests and evaluations necessary to provide appropriate treatment, during which precious time is lost and additional administrative burden is placed on clinicians.”

Building a fast, safe, and secure proof of concept with OpenAI

Color began working with OpenAI in 2023, with the goal of using AI to improve cancer patient care and health equity. With the challenges of cancer screening, diagnosis, and treatment in mind, Color was looking for a solution that could:

• Interpret inconsistently-formatted patient data
• Analyze dense healthcare guidelines
• Protect patient data privacy
• Support clinician-in-the-loop workflow design to ensure patient safety
• Integrate with electronic health records (EHRs) and core hospital systems

During initial exploration, Color set up their approach for rapid experimentation, including testing the performance of GPT‑4 and GPT‑4o in complex tasks such as extracting information from PDFs of clinical guidelines for cancer diagnosis. These PDFs are often hundreds of pages of complicated diagrams that outline care paths based on diagnostic workup. Together, OpenAI and Color developed a method of asking GPT‑4 Vision to describe screenshots of these diagrams that was most effective in maintaining output accuracy.

OpenAI also helped guide the Color team to prototype clinical workflows using the standard ChatGPT interface and generate sample cases using a custom GPT–gaining effective proofs of concept before committing extensive engineering resources.

With OpenAI’s expert guidance, powerful models, and HIPAA-compliant data protection standards, Color was able to focus on deconstructing complex medical decision-making, refining prompts, and designing clinician-in-the-loop workflows to create the initial version of the copilot. For example, OpenAI engineers guided Color to use retrieval-augmented generation (RAG) instead of model fine-tuning to increase output quality and rewrite clinical documentation for easier processing by ChatGPT. Ultimately, after experimenting, Color selected OpenAI as its AI solutions provider, with GPT‑4o at the core of its cutting-edge copilot application.

Reducing time to treatment for cancer patients

To measure the impact of this tool, Color is partnering with the University of California, San Francisco Helen Diller Family Comprehensive Cancer Center (UCSF HDFCCC). For the initial implementation,…