How objective are studies anyways?

Imagine that you want data about ETOH and opioids influence on driving. You want this data, ideally, to be applicable to any dude walking/driving down the street (external validity). You also want this data to be done correctly so that nobody could say "hi, look, the study population was 97% of White, obese, middle-aded males and 76% of them had diabetes and nobody checked their kets and sugars - the oxycodone, which is made by our drug company, is abdolutely safe and effective, all those simulated car crashes happened because of hypoglycemias, heart attacks or because the subjects couldn't comfortably fit themselves in sim sits!" (internal valudity).

What you will have to do to avoid these two pitfalls at once?

You would like to start from recruiting as many different dudes as possible. Your stat guy will calculate expected power of study, or number of subjects to be recruited to gather enough data points for getting valid results. You will add at least 50% to this number - to account for those who get second thought and refuse, will vomit for hours after oxycodone, get sick, have to move across the country, miss appointment because they got to stay in line for new Iphone X, etc., etc. Then, you'll start recruiting. While doing so, you will pay special attention to categories of subjects which are known to be difficult to recruit and whose inclusion adds greatly to BOTH internal and external validity, such as females and minorities. At this stage, you won't be worried about any of the recruited dudes being alcoholics or drug users or patients who take no oxycodone but only Norco for chronic pain. After you finally recruit enough of them, you will run your study population for demographic data vetting and see how many of them are going to move your ideal population either way (and so introduce biases to your study design). Depending on what you get, you either change your protocol or vet the subjects' pool.

And only then the hopefuls - not all of them - will be called for their free shot of Jack Daniels and a "good pill".

That is how it should be done. In reality, one can see "research" written on the level of "the moon goes up once the sun goes down - it clearly suggests that these two are connected by a sort of a tight but very strong rope, and the Earth must be hung right in the middle of it. But we need more quality RCTs to have more evidence of it" in NEJM and JAMA. The infamous "study" of A. Wakefield was accepted and published by Lancet, which is one of the most prestigious peer-reviewed journals in the World.

Ok, got detailed consent. I'll make $300 for participating! Woo hoo!

"Do not trust any statistic you did not fake yourself."

(attributed to Winston Churchill)

I've found almost every methodology will have its biases and weaknesses. What strengthens the study is when they acknowledge the weaknesses and assess how that could effect the study. It's also our job as critical readers to take the studies and analyze the methodology/ results.

Today I received an email saying that they've recalculated my score from my in-person interview and disqualified me. I thought that strange, since the in-person interview hasn't occurred yet, only a short phone screening.

I have severe doubts, now, regarding studies. These people seem disorganized and not methodical. Are they trying to steer the study in a particular direction? Are they confused? I was told that the 4 hrs session was my in-person interview.