How good is the evidence underlying NCCN treatment guidelines? – Healthcare Economist

Centers for Medicare & Medicaid Services (CMS) and most
private payers in the U.S. have accepted the NCCN guidelines as a mandated
reference to determine payment for “off‐label” use of
anticancer drugs that are not approved for the specific indication by the U.S.
Food and Drug Administration.  Thus, a key
question is what types of evidence are driving these guidelines. 

A paper by Liu et al. (2018) examines the quality of evidence that underlies these guidelines.   They find that:

In total, 1,782 recommendations were identified in 29 guidelines, of which 1,282 (71.9%) were based on low‐quality or low‐consistency evidence (low‐level evidence), including “case reports or clinical experience only” (18.9%). A substantial proportion (31/143, 21.7%) of category 1 (the highest level) recommendations were based on low‐level evidence. The majority of authors (87.1%) received payments from industry. However, no association was found between the prevalence of payments among authors and the percentage of recommendations developed from low‐level evidence per guideline.

The evidence certainly highlights the need for more high quality evidence to support these decisions. Also interesting that industry payments did not influence the quality of evidence considered.

A more recent paper by Noy et al. (2022) looks at the quality of evidence available for different for radiation-related NCCN guidelines. They find that:

Among all radiation therapy recommendations, the proportions of category I, IIA, IIB, and III CE were 9.7%, 80.6%, 8.4%, and 1.3%, respectively. When analyzed by disease site, cervix and breast cancer had the highest portion of category I CE (33% and 31%, respectively). 

One interesting study to do would be to have AI–particularly an AI that cites its sources (like Perplexity)–make treatment recommendations for the same disease categorization that NCCN uses . One could see to what extent (i) the AI-generated recommendations mimic NCCN guidelines, (ii) whether the AI-generated recommendations relied on more or less high-quality evidence when making these recommendations, and (iii) how frequently the AI ‘hallucinated’ and cited a source that was not at all relevant (or was made up!). An interesting research project for someone to take up!