Critique of Study: More BSNs equal better pt outcomes

IMustBeCrazy · Jun 14, 2006

I wasn't aware that I needed to 'google' my opinion. Darn that independent thinking!

indigo girl · Jun 14, 2006

Fascinating thread. And it touches upon an aspect of nursing I've wanted to ask for ages but never found the right opportunity to do so.
If this question really detracts from the thread, please feel free to ignore it.
My question:
My Father has often told me that this whole business of graduating from school and then having to pay someone to sit for the NCLEX so that the State can grant you a 'professional' lisence - then having to go to a job and undergo orientation/training anyways makes no sense. He says that the whole lisencing process as it stands today is BS - just another way for the State and/or the 'professiona' bodies to make more money by selling the bunkum that some lisence is required by taking a particular exam and that this is somehow more required that that initial orientation/training.
Is he mistaken? Is there some grain of truth to this?
Thanks

Just an aside from this thread:

Maybe your dad is right. Many libertarians feel this way about state licenses including marriage licenses, car registrations, driver's licenses etc.

It is like living in an alternate reality. It can be done, but it's really, really difficult. The Amish are doing a version of it.

pickledpepperRN · Jun 14, 2006

Thank you all for your opinions. Timothy did an excellent critique.

When I read the original study in JAMA it rang true.

The second study disturbed me. The logical conclusions of the first were absent in the second.

I didn't bother to analize why. Was I lacking in logic or research skills? After all i dropped out of grad school.

Clearly the ability to assess and prevent a crisis is a combination of education and experience.

This is a great subject.

weirdRN, RN · Jun 14, 2006

I have an idea for a study design.

Let's have ONLY six units of Nurses participate in a year long study.

Unit One: All ADNs with less than Five years experience.

Unit Two: All BSNs with less than Five years experience.

Unit Three: All ADNs with more than five but less than thirty years experience.

Unit Four: All BSNs with more than five but less than thrity years experience.

Unit Five: MIX of ADN and BSNs with MIXED levels of experience.

Unit Six: Control (can anyone think of what a control group might look like?)

We would look at specific criteria similar to the ones used in the study.

The six units would have identical policies and procedures that all the nurses participating in the study would be inserviced on.

The kinds of clients being admitted to our units would be general med-surg patients.

Hmmm I am fading I have to go think about this a little longer.

ZASHAGALKA, RN · Aug 24, 2007

A recap on why the Aiken study regarding mortality and education is seriously flawed:

1. Significant Data Ignored

The authors used the data they had gathered for their previous study, on mortality vs patient ratios. It was not a good fit for this new study, and as a result this study left out military and VA hospitals, and small hospitals. In other words, the hospitals MOST likely to have high and low percentages of BSNs were left out of a study regarding the effects of BSN on care. You cannot just ignore significant population segments within a study and then claim that your observations could apply generally to the population.

For example, if I were studying speeding as a relationship to vehicles that people drive, and I left all sports cars out of the study because my data didn't include information on sports cars - and then I came to the conclusion that most drivers do not speed in their vehicles - how accurate would that study be?

2. Degree Bias

Imagine just how monumental this study should be: is there a difference in outcomes between degrees? Now, look at THIS statement, in the body of the study: "Because there is no evidence that the relative proportions of nurses holding diplomas and associate degrees affect the patient outcomes studied, these two categories of nurses were collapsed into a single category."

Hold on there for just a second! There are two very important things about this above statement.

First, how can the authors blithely dismiss the potential differences between ADN and Diploma degrees in a study about the differences between degrees? What methodology did they use to reach this conclusion? Wouldn’t such a finding merit AT LEAST AS MUCH attn as the study itself? Isn’t there a high level of hubris involved in dismissing the differences between ADN and Diploma in a study about the comparative value of degrees?

Second, what this means is that the authors simply didn’t study BSN vs. ADN, or BSN vs. Diploma. By themselves, maybe ADN degrees would outperform BSN. Or maybe, Diploma would. Or maybe, both degrees would outperform BSN, if measured individually. We’ll never know, as it wasn’t a part of this study.

3. Author Bias

Instead of examining real life situations, the authors looked at ICD-9 codes on State reporting records from hospitals in Penn in 1999. They compared these to death records. If there was a death AND a subsequent ICD-9 code (a code entered after the original, admission code(s)), then the authors determined if this was a new event, a 'failure to rescue', or a "co-morbidity". The authors, “in their own expertise”, decided which of the subsequent ICD-9 codes that they examined equated to ‘failure to rescue’ and which were 'co-morbidities'. Any event they deemed 'co-morbidity' was eliminated from further consideration, as they were looking for events deemed to be, 'failure to rescue'.

In the real world of science, for a study to be valid, there must be some level of ‘double-blindness’. There must be a separation between the data that goes in and the results it produces. Otherwise, if you were so inclined, you could just manipulate the data going into a study until the results become what you want. It’s the old cliché, ‘garbage in, garbage out’.

Manipulating the data that went into the study is JUST what the authors did. IN THEIR EXPERTISE, they sorted which data made it into the study, and which data didn’t. It doesn't matter if they were intellectually honest, or not. It is the POTENTIAL for abuse here that completely undermines this study.

Now, do you think it matters that this study was funded by the AACN – an organization known to strongly promote BSN-entry education of nurses? Why would this be important? Am I just shooting the messenger here? In logical fallacy terms, am I simply engaging in an ad hominem attack?

If I’m a pollster and I produce a poll that includes only the data that I see fit to include, and excludes all other data, and my poll results show that 'America loves Hillary', do you think it would be important to also know that her campaign paid for my polling? More importantly, how would you think knowing my sponsor and knowing that I manipulated the data that went into the study could affect my credibility? Exactly.

Or maybe, it's just a happy coincidence that a study not disciplined by objective scientific controls happened to EXACTLY match the opinions of the backers of the study. It is BECAUSE the authors subjectively sorted the data prior to inclusion into their statistical models that it becomes important to know if they have a particular stake in the outcome. At least their backers got their money's worth.

Or maybe, real science is much more stringent in protecting against the potential of such biases. MAYBE. . .

4. Multi-variable Regressions and Collinearity

You might think that the study compared hospitals with higher numbers of BSNs against those with lessor percentages. You’d be wrong. After manipulating the data 133 different ways, the authors used a ‘logistics regression model’ to examine education – and only education – as a factor in mortality, and then projected that factor into hypothetical situations with hypothetically higher and lower percentages of BSNs.

Sorry, but this is fundamentally impossible. Let me explain why.

First, multi-variable regression studies are limited by increasing the number of factors you attempt to distinguish between. The higher the number of independent variables, the less reliable the results become. In addition, there is a significant issue of collinearity: factors too close to be distinguished from each other invalidate any attempt to segregate them for the purposes of study.

Let’s take multi-variable regression first. Steven Levitt, in his book, Freakonomics, gives a wonderful explanation of logistics regression models. Imagine a bank of switches in which one master switch turns on a light, and a whole bank of other switches are left in the ‘on’ position. If you want to know which switch (or switches), like the master switch, could also turn on the light, then you have to – one by one – turn off all switches but the one you wish to test. Logistics regression is about just that: turning off all other switches, or, accounting for all other possible explanations for why a thing happens – except the one you want to study.

As it turns out, the authors of the study noted 133 ways in which other factors might affect mortality besides education alone. Then, the authors suggest that they turned off all those switches except the one – education. The problem is that the more variables you introduce, the more likely even very small miscalculations could ultimately have exponential affects on the outcome.

How do you determine the difference in education, and education alone, between Fred, an ADN RN in a county hospital with 8 patients, and Sally, a BSN in a city teaching hospital, with 4 patients? How? You use a regression model to eliminate all those differences except education. Except, the more 'risk adjustments' that must be made to equalize these situations, the more likely for error to be introduced.

It is just impossible to suggest that the authors accurately accounted for 133 different variables and were able to switch them all ‘off’ in sync. This however, also gives the authors a different way to manipulate the outcome of the study. Tweaking any of those different variables (risk adjustments) this way or that, by even a small amount, once compounded 133 ways, could produce changes in the results in the lines of orders of magnitude.

This violation of methodology is referred to as overfitting a statistical model.

Then there is the factor of collinearity. In order to perform a regression model, different situations cannot be similar; they cannot lie on the same line, or be collinear to each other. The problem here is that education and experience are essentially collinear. In fact, it is accurate to say that experience itself IS a form of education.

The only way to conduct a study that would note differences in education absent the effect of experience is to eliminate experience as a factor, altogether. Amazingly, this is JUST what the authors found. From the study, “Nurses' years of experience were not found to be a significant predictor of mortality or failure to rescue in the full models.” Re-read that and UNDERSTAND the implications of what it means. Wouldn’t you think that, if the authors truly discovered, as a result of this study, that experience has no impact on mortality, that this fact would be MORE IMPORTANT than the object of the study itself????? Who knew that, on average, a nurse with 3 weeks experience is just as safe in practice as a nurse with 30 years experience?! Well apparently, the authors of the Aiken study knew it.

Can you see how the conclusion that experience is not a factor in mortality is a necessary construct to eliminate the collinearity problems presented in trying to evaluate education alone? More to the point, can you see how this "discovery", that there is no relationship between experience and mortality, simply fails to pass the smell test? Do you believe it? If the authors truly discovered that experience is not a factor in mortality, THAT would have merited mention, high in the abstract and in the headlines this study made. If however, this is just a necessary construct to justify using a regression model, then it would be buried in the small print, which, it was.

Amazing that such a concession, a necessary concession to use a regression model in this study, is also just the concession these authors discovered! Just one thing: it would have been very nice to have available the methodology of how they reached this epochal discovery. I have to tell you, the potential discovery that experience is irrelevant in nursing BLOWS AWAY the conclusions the authors reached, at least by comparison.

5. Standardization

Even after all this, the results didn’t ‘fit’ with ‘clusters of data’ that had to be run through direct standardization models. In other words, the sky wasn’t green like the study suggested, so the study needed a filter. The authors admit, in the small print, deep in the study, that in order to standardize the data, they had to use 'robust estimations'.

'Robust estimations'?! I’m sorry, that’s like saying that except for the fact that she’s had sex, this girl is a virgin! I estimate that this study is wrong. In that, since I am only estimating, maybe even 'robustly', I stand on equal scientific footing and I am being at least as scientifically valid as the study, itself.

6 My Conclusion

Saying that education is a bigger factor than experience in ANYTHING is the same as saying that nurture is a bigger factor than nature in ANYTHING. The relationships are so intricately linked as to be inseparable. As a result, these types of arguments rise to the level of philosophy.

This study claims the ability to make such distinctions, using incomplete and highly manipulated (133 ways by its own admission) data and applying that data only to hypothetical situations.

This is not science; it's propaganda.

Simply put, this flawed and un-reproducible study is worthless as anything BUT propaganda. And that's the bottom line.

~faith,

Timothy.