Monday, September 26, 2011

How to move forward - published results don't replicate

Powering better drug discovery research(ers):
Most people working in and around the pharmaceutical industry agree that it must become more efficient, more effective, and less costly.  How to do this is still being debated. One prominent view is to reduce internal research within larger companies, relying on academia and start-up companies to provide grist for the development mill.  Another view is that stand-alone pharmaceutical companies must retain strong internal research groups (see John LaMattina).  Both sides agree that a primary goal is to lower the cost of products going to the market.
There is little doubt that novel drug discovery must be grounded in unfettered academic-type research.  The question is how best to apply those findings to drug discovery.  I’ve written previously that a primary hurdle lies in the evaluation of which early stage findings to follow up on.  Academic research has no incentive to provide replication of findings or presumes that other researchers will provide for replication.  Those of us in pharmaceutical research know that replication of published biological results is not common.  This leads pharmaceutical researchers to using gut instincts, read subjective preferences, to decide which new ideas to follow up.  It also leads to hard feelings among academics about pharmaceutical companies. This real concern has recently been given quantitation (James Choi, The .Plan: A Quasi-Blog)); fully 2/3 of published findings could not be replicated! 
The situation makes it impossible for pharmaceutical companies to efficiently, cost-effectively, start drug discovery programs.  With the goal of containing the final cost of new drugs, dependent on the amount of money the company selling the product spent to develop it, pharmaceutical development companies simply cannot follow up on one-shot publication discoveries.  The hurdle posed by the irreproducibility of findings is compounded by the fact that non-replications, i.e. negative results, are extremely difficult to publish.  Information that a result cannot be replicated is primarily carried word-of-mouth and is not directly verifiable.  The presumption by academics that replication will occur elsewhere is, in reality, nowhere.
This reality means that the view that pharmaceutical companies should cut internal research and rely on discoveries from academia and start-ups can only come from people who have never attempted to discover a novel drug.  It is a dead-end approach.  It is true that development and sales companies cannot afford to add to the consumer price of drugs by footing large, permanent research groups.  However, they must have experienced pharmaceutical scientist-consultants who can, in their own labs or by contract, efficiently validate (replicate and more) novel ideas and findings.  However, even these groups will be overloaded by the number of novel ideas that ought to be evaluated.
Still, the bigger issue is how to breech the grand canyon between novel ideas and exploratory drug discovery projects.  One way novel ideas progress to a stage that they can be assessed by companies is for an individual scientist to start a company to gather more information about their idea.  Much money and other resources have been made available to facilitate this path.  There are at least two gigantic faults with this approach.  One, many excellent scientists simply have no interest in taking this route, preferring to continue basic research, teaching, etc.  Their ideas may simply be lost.  The other fault is that the academic scientists who do form a start-up must now convert themselves into pharmaceutical scientists, without training.  Notwithstanding that these scientists are all the smartest people in the room, the conversion to pharmaceutical researcher has as much to do with weltanschauung and experience as intelligence.  I conclude, simplistically, but truthfully, that this is primary a waste of time and money.  The graveyard of genomics start-ups is a testament to this opinion.  It is only the fact that these costs are not directly factored into the overall price of a drug that makes this a viable path.
Moreover, this approach distracts academics from their role of digging up new ideas.  And, the breech between academic ideas and pharmaceutical exploratory programs is, very often, a matter of whether to believe results are reproducible and generalizable.  Replication of results needs to become a standard expectation from academic laboratories.  Results should, first of all, be repeated by the original investigator/postdoc, but in a blinded fashion.  The dividends this will pay in the education of new investigators are enormous, if initially painful.  Secondly, someone else in the same lab (graduate student, junior postdoc) should repeat the findings, in a blinded fashion.  Again, this provides specific training to these less experienced researchers, as well as experience with equipment and procedures used in the lab.  If these two levels of replication cannot be met, the results should not be published. This scenario obviously leads to young investigators staying longer in their training lab, but the benefits that can occur – more time to learn how to write successful grant applications, taking on the role of lab manager and learning how to manage a lab, etc. – will pay off greatly in terms of new faculty better able to adjust to having their own position, their own lab, etc.
The gap between one-publication new ideas and exploratory drug projects must be bridged!  Pharmaceutical companies and/or NIH could form institutes whose goal is to replicate published results.  However, the multitude of new publications every day would quickly overwhelm such efforts.  Much better that young investigators and their lab chiefs, universities and journals, take on the responsibility of replicating their own results.  Huge benefits would accrue to the development of new drugs, and in increased trust and communication between companies and academic investigators.  In addition, I strongly believe that the lessons learned, and new wisdom developed, from the experience of non-replication will lead to tremendous progress in all of biological research.


No comments:

Post a Comment