Home > News > Sharpen That Needle? No, I Don’t Think So

Sharpen That Needle? No, I Don’t Think So

Scott Beale/Laughing Squid

A study out in Archives of Neurology this week has generated a lot of interest. For example, the New York Times had a front-page story with the subhead “100% Accuracy in Test for Alzheimer’s” or something similar. The “100% Accurate” part is what caught my eye, and others too. I doubt that a “10% Accurate” headline would have made it to the front page or that the story would have been covered on the major networks. But is the headline true? My first thought was that no, of course not, otherwise the headline would have been “First-Ever 100%-Accurate Medical Test Developed.” 100% is a number we don’t see too much in medicine, particularly in the ambiguous world of neurological diagnosis. A test like that would be an enormously huge deal.

Does the study live up to the hype? Sadly, no. And it certainly doesn’t justify the an editorial entitled “Sharpen That Needle” which appeared in the same issue, written by two doctors who should know better. There is a 100% result found within the study results, but it isn’t the accuracy of the test exactly. It’s the sensitivity. And not the overall sensitivity, but the sensitive among patients with mild cognitive impairment who went on to develop Alzheimer’s. Now in the next paragraph I’ll explain sensitivity, so those of you who don’t want or need a (very) basic statistics lesson can skip over it.

New York Times Front Page 8/10/10

The “sensitivity” of a test tells us how good it is at finding true cases of what it’s looking for. Of all the people who have a certain disease, say, how many will test positive on our proposed test? If 5% of people with the disease test negative (false negatives) then our test is 95% sensitive. The other basic quality of a test is “specificity.” It tells us how many of the people who come up positive on our test actually have the disease we’re testing for (true positives). So let’s say that 10% of the time our test is positive, it’s wrong. Then we have a 90% specificity. You can’t judge a test based on just one of these elements. As an extreme example, let’s say we check if people are breathing to see whether they have Alzheimer’s. Now since we only test living people, all of the people with Alzheimer’s will have a positive test result: 100% Sensitivity, hooray! But if only 3% of the people we test have Alzheimer’s, then we have a 3% Specificity. Boo! Now the test proposed in the article did a lot better than that, but you get the idea.

If you guessed that the problem with the test is in the specificity, you’d be right. About 36% of the normals in the study also tested positive. So our specificity is down around 64%. This is a problem. It’s a problem because if you use this as a screening test in your office, about 1/3 of the people you inform that they should prepare for Alzheimer’s won’t actually get it. Not good. The other problem is that the “accuracy” of the test is better described by the positive predictive value, which depends on the sensitivity/specificity but also the prevalence of the disease. About 20% of people 75-85 have Alzheimer’s. If we use that as the prevalence, I get a 38% PPV for the test. (This is where my statistics starts to get hazy, so please send corrections if needed). That means that if you test positive on this test, and you live past 75, there’s about a 40% chance you’ll get Alzheimer’s. There are just too many false positives for this test to be a good screen.

andrewacomb via flickr

And the biggest problem with this test for most people won’t even be its accuracy. The big problem will be that it requires a spinal tap. Let me tell you firsthand that lots of people are deathly afraid of taps. They’re really not so bad, but if you think it’s hard to get people in for colonoscopy, forget about spinal taps.

Not to mention the biggest problem of all: if you do get Alzheimer’s there’s not a lot we can do for you anyway. Most doctors would or should question the ethics of testing for an untreatable illness. Now AD isn’t exactly untreatable, but there’s no evidence that the meager treatments we do have change the course of the illness in the long run, or that starting them earlier makes much difference.

If this did become a standard screening test, I’d certainly stand to benefit: as a neurologist I would start a dementia clinic and tap everybody and send lots of big bills to Medicare. And if I thought it would do my patients any good, I’d do just that. In fact, I’d team up with a GI doc and offer a combined colonoscopy/lumbar puncture. It’d be perfect: you’re already lying on your side, sedated, and you won’t feel or remember a thing. Turning 50? Come on in!

But until we have something to offer our Alzheimer’s patients, I’ll postpone sharpening that needle, thanks.

  1. September 19, 2010 at 11:45 am

    Excellent post on statistics. By my back-of-the envelope calculations, the situation might be even worse than you predict: if we take a .1 incidence in AD among 60-year-olds (http://www.ncbi.nlm.nih.gov/pmc/articles/PMC1509089/?tool=pmcentrez, reasonable because this study was examining those with mild cognitive impairment). Then, in a sample of 1000 55 year olds, 100 will get AD by age 60 and 900 will not. All 100 AD patients will test positive, as will 324 of the non-AD patients. Therefore, if you are 55 and get a positive test result, there is only a ~24% chance (100/424) that you will get AD in 5 years.

    • September 19, 2010 at 10:59 pm

      Thanks for commenting. Yes, if you look at the prevalence of AD at 60, and are trying to use this test to predict that, then the results are even worse, just as you describe. I was being generous and using a higher prevalence, more like what you see at 85. Of course many 55-year-olds won’t make it to 85 for a variety of reasons, so my estimate was high, but even in that case the numbers don’t look very good for this supposed screening test.

  1. September 18, 2010 at 1:19 pm
  2. September 24, 2010 at 9:15 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: