HEALTHCARE’S LANGUAGE MODEL RACE TO ACCURATE AND EFFICIENT NLP
Herein, we introduce HER-BERT (Health Entity Resolver – BERT), a Small Language Model (SLM) that is deeply tuned to healthcare domain expertise for the focused downstream task of NER and Entity Resolution. HER-BERT successfully navigates the tradeoffs between model efficiency and precision and exceeds state-of-the-art accuracy with less computational overhead.
Given that 80% of currently available healthcare data exists in an unstructured format, accurate and reliable NLP solutions are critical to meaningful population health and value-based solutions. While recently published generative LLMs have demonstrated a remarkable ability to solve generative tasks (often with zero-shot, or few shot learning), LLMs may not achieve similar success for the needed NLP tasks of Named Entity Resolution (NER) and Entity Resolution (ER). In many cases, a smaller model that is fine-tuned deep domain expertise would be more economical and more precise.
Large Language model limitations
It is easy to get caught up in the hype surrounding recently published LLMs. While their capabilities are clearly impressive, their massive size introduces significant cost and latency to an application.
Foremost, LLM’s (like GPT) were never intended to solve all downstream tasks. While its Decoder architecture is ideal for generative tasks (chat bots, question/answer, and narrative generation), it is NOT particularly adept at text classification tasks.
Make it stand out
Whatever it is, the way you tell your story online can make all the difference.
LLMs are trained to be helpful generalist assistants – a ‘Jack-of-all trades.’ Despite being exposed to a massive corpus of data, LLM pre-training is naïve (unsupervised). While they display strong versatility across an extensive array of domains, their capabilities represent a compromise solution that dilutes performance for very niche down-stream tasks.
Finally, exposing potentially sensitive health information to external LLMs poses a privacy and compliance risk.
Small Language model Advantages
When considering a language model, we believe that a ‘one-size fits all’ data strategy to be overly simple. The healthcare domain has a unique vernacular of formal definitions, acronyms, and catch phrases.
HPI: 61 YO M w RLE pain. VDUS shows CFV DVT. The patient started on LMWH.
Any medical provider would have a clear understanding of the above narrative. Your NLP should share that same understanding.
It should not be surprising that SLMs outperform LLMs for specialized use cases. Instead of tuning a model broadly, SMLs are fine-tuned deeply on a niche domain using labeled data. The supervised training imparts more capability per parameter relative to the naïve training offered by the general corpora used to train LLMs.
The smaller number of parameters make SLMs more computationally efficient, requiring less memory, and can suffice with smaller training datasets, and train with quicker iteration cycles. SLMs can be deployed locally and do not require specialized hardware.
While it is true that SLM solutions require more effort up front (to tune a base model), in the end it will likely be significantly more performant and less expensive. This is particularly important for repetitive tasks such as search, or computer-assisted coding.
Sub specialization is our competitive advantage
In healthcare, the era of the general practitioner has long since passed. While it is not always the most convenient solution, patients generally prefer the expertise of sub-specialists.
Like the healthcare industry itself, we at Archimedes Medical believe that applications need deep domain expertise. Our goal is to optimize a model that uses smaller, open models for the specific downstream tasks of NER and entity resolution.
As anyone will tell you, the most important factor for fine-tuning a model is data. Having as much quality data as possible is critical. Healthcare uses a variety of vernacular: structured medical terminologies, medical slang, acronyms, and aphorisms make constructing training data for semantic textual matching a difficult task.
While many organizations lack sufficient medical expertise to meaningfully fine-tune a semantic model, as a physician-led organization, we possess a deep domain expertise as our competitive advantage. Our models are trained on a > 100M token, physician-curated corpus that was optimized for Sematic Textual Similarity.
Health Entity Resolution-BERT
Fine-tuning is an approach where weights of a pre-trained model are refined on a specialized corpus of labeled data (supervised learning) for the task of semantic textual similarity.
Our model, HER-BERT, uses an Encoder architecture that provides a much more economical fit while optimizing model output for the intended task.
· Encoders generate condensed outputs from larger inputs (LLMs use Decoders that are designed to expand content) and are better suited for downstream classification tasks.
· Encoder models are many magnitudes smaller and are therefore, more performant.
· Most base Encoder models are ‘open’, meaning that the pre-trained model weights are published and may be modified through fine tuning using domain-specific data.
Encoder models are deterministic (Clinical Decision Support applications need to be grounded).
Give us a try!
As healthcare enterprises search for lean and reliable data processing solutions, many cost-conscious CTO’s are looking beyond the weighty LLM models.
If you have interest in exploring our solution, please drop us an email and we will provide you with a free, limited account to access our online sandbox to explore the model. For more information about unlimited access, we offer flexible, rate-based API access that will scale with your organization.
Archimedes Medical, LLC
