Trouble Brewing around NBER Paper on Indian Pharma – Part I

We’ve recently been informed of two events, one flowing from the other, both of which I would term as very disturbing in their own way.

trouble brewingFirst, is this NBER Working Paper that concludes that it is likely that Indian pharmaceutical firms differentiate drug quality based on destination of consumption, to the detriment of African countries.
Secondly, is the follow up development that the Indian government may be taking legal action against the authors of the above paper. Needless to say – both of these developments call for urgent attention.

In this post, I shall be dealing with the first of these developments: the paper itself. In a follow up post, (available here) Gautam Bhatia, a free speech scholar will be examining the second of these developments. Since I won’t be discussing that second issue, I’d like to state up front – regardless of the content and quality of this paper – I think it is an extremely distressing state of affairs if academia needs to start worrying about defamation (or any other) law suits. Exponentially more so if the action comes from the government!! – Not to mention, I’m not sure how the government can claim defamation when products of (unnamed) companies are supposedly being defamed. Anyhow, I trust Gautam will discuss that issue in suitable depth. Moving on to the first issue: I will first give the basic thrust of the paper, followed by my own observations and comments, followed by the reaction from Indian pharmaceutical companies. Relevant links are found at the end of the page. Long post ahead.

Basic info on the paper: 

The paper titled “Poor Quality Drugs and Global Trade: A Pilot Study” was published in September 2014, and authored by 1. Roger Bate (of the thinktank ‘American Enterprise Institute’), 2. Ginger Zhe Jin (of University of Maryland, Dept of Econ), 3. Aparna Mathur (also of ‘American Enterprise Institute’), and 4. Amir Attaran (of University of Ottawa, Faculty of Law).

As the paper states, they examined 1470 samples of antiobiotics that claim to be made in India, distributed across 22 cities spread over 18 low to middle income countries across the world. The samples included two broad-spectrum antibiotics (ciprofloxacin and erythromycin) and two first line antibiotics for tuberculosis (isoniazad and rifampicin).

They found that 3.9% of samples contain no Active Pharmaceutical Ingredient (API), and 7% of samples contain less than 80% API. They also state that these bad samples tend to be unregistered products that are disproportionately found in the African cities they’ve sampled, and that it is likely that Indian manufacturers intentionally send bad samples to African countries as it is cost effective.

Analysis and comments: 

Given the nature of this claim, it is natural that it be taken very seriously. And to be perfectly frank, I wouldn’t find it surprising that not only Indian companies, but that all pharmaceutical companies try to cut costs by exploiting weaker regulatory regimes. For that reason alone, I commend the taking up of this study. That being said, I’ve only had a chance to take a preliminary look at their paper and have noticed some problematic parts. Given that this is a ‘working’ paper, I hope that these gaps are taken note of and addressed.

1. Country selection & grouping bias?

The paper bases much of its conclusions on ‘comparing’ distributions between different groups of countries. It categorises them into three groups: India, African countries (12 of them) and Non-African countries (5 of them). It doesn’t however explain the selection of these countries. As the study itself notes, there is a high correlation between GDP and ‘bad samples’. Therefore its worth noting (see Table 4 of their paper) that the “Non-African group” consists of 5 countries, all with 5-digit city GDP per capita, while all the other countries in the list, with the exception of Egypt, have a 4-digit city GDP per capita (or less). Therefore, first off, this doesn’t seem to be a comparison of equals.  The study notes that outside factors have been controlled for – but given the very small sample size for non-African countries, I don’t know how useful this ‘control’ is. But regardless of that –  simply the selection and grouping of countries, I think, leaves a lot to be desired. For example, I would be very interested in seeing whether Egypt (being the only African country with a GDP comparable to the Non-African countries), sees similar rates of bad samples as the rest of Africa. Or how much DRC (as the only country with a 3-digit city GDP per capita) brings down the average for African countries. Or if non-African poor countries such as Nepal, Bangladesh, Afghanistan, Haiti, etc also get similarly bad samples. Currently, simply stating a discrimination against “African” countries seems a tad bit more sensational grouping, rather than useful information. [It goes without saying that low GDP or not, no manufacturer ought to be intentionally sending low quality drugs anywhere].

2. Sampling bias?

I’ve drawn up a chart of the samples they’ve taken based on their data provided in the tables at the end of the paper: (Fake = 0% API; Substandard = < 80% API)

Samples India Africa Non-Africa Total
All 956 430 84 1470
Ciprofloxacin 456 151 84 691
(Fake) (SubStandard) (4.8%) (3.3%) (3.3%) (8.6%) (4.8%) (0.0%) (4.5%)(4.1%)
Erythromycin 167 119 0 286
(Fake) (SubStandard) (1.8%) (10.8%) (7.6%) (11.8%) 0 (4.2%)(11.2%)
Isoniazad 166 57 0 223
(Fake) (SubStandard) (1.8%) (4.8%) (8.8%)(7.0%) 0 (3.6%)(5.4%)
Rifampicin 167 103 0 270
(Fake) (SubStandard) (2.4%) (7.8%) (1.9%) (17.5%) 0 (2.2%)(11.5%)
Total (3.3%)(5.6%) (4.9%)(11.4%) (4.8%) (0.0%) (3.9%)(7.0%)

The first thing one notices is that there were no samples taken of 3 of the 4 drugs, from Non-African countries! It’s worth noting that this breakdown of samples is not mentioned in the main text of the paper; it is only found in the tables after the end notes. The only reason I found mentioned for differences in quantities of samples from different countries was this line: “Because drug availability varies across purchase countries (in part because the targeted disease varies)”.

A few quick points jump out:

1) that this once again sharply demonstrates the problem with comparing these countries – since clearly more than 50% of the drugs compared aren’t even required/weren’t sampled in one of the two export groups – yet the claim of being discriminated against, is being made for the sole export group.

2) that their given reason doesn’t actually explain why samples of the other broad-spectrum antibiotic Erythromycin weren’t taken in Non-African countries. [Admittedly, I am making an assumption that where one broad-spectrum Indian manufactured antibiotic is available, the other one also should be.

3) In the only comparison where all three groups are covered, African countries actually have the least percentage of fake drugs.

4) In two out of the four drugs, India has a higher percentage of fake drugs than do the African countries. However, in a statement on page 7, they say: “However, drugs purchased from Africa are more likely to fail the TLC test than the same type of drugs in the Indian domestic group.”

This statement would only be true if patients were willing to substitute one drug with another (which obviously doesn’t apply here), or if more types of drugs were likely to be fake in Africa, than in India. However, as their own data shows, 2 types are more likely to be fake in Africa, while the other 2 types are more likely to be fake in India. And clearly a sample of only 4 types of drugs isn’t a big enough one to be making broader generalizations on.

5) With regard to drug quality, they also state, “In comparison, drugs from Non-Africa have a greater passing rate than the Indian domestic ones within the same drug type.”  While technically true – this is misleading, as only one type of drug has actually been compared between Non-Africa and Indian domestic – and even in this one drug – there was an equal percentage (4.8%) of failed drugs in both.

Only in the substandard test, did Indian domestic (3.3%) have a worse rate than Non-Africa (0%). This is also to be kept in context of 456 samples being taken from India, while only 84 samples were taken from across 5 countries for Non-Africa.

6) Of the two drugs that India has more fake drugs than Africa, it is curious to note that one drug has a ridiculously high percentage of sub-standard availability in African countries (Rifampcin: 17.5%). As I’ve often anecdotally heard that TB medicines often get degraded in storage, I decided to do a preliminary check and found that Sanofi’s rifampcin capsulestablets as well as infusion solutions are required to be kept under 25C, and away from light and moisture. For comparison with another drug – Bayer’s ciprofloxacin tablets don’t require any special storage conditions. Thus – I have to ask – Has this properly been controlled for, in making the determination that the drug was not degraded due to storage and handling? I am presuming most of Africa does not remain at 25 degrees C. Therefore questions of storage and handling must be properly looked into.

[Edit: Madhulika has brought some studies to my attention, on when Rifampcin may degrade in quality. They have been mentioned in the comments below]

There were a few more minor (presumably non-consequential) discrepancies I came across, such as the authors making a counting mistake in stating that 9 Indian pharmaceutical companies exported more than 75% of their net sales, while I could count only 8 in the chart they referred to. (footnote 4). Similarly, they state that they used Ciprofloxacin data from 3 of their previous papers – But when I took a look at the latest one of these papers, I found that there they had found 683 samples from Indian manufacturers, while this latest study has 691 samples. This latest addition of 8 samples isn’t explained.

Indian Industry Reaction

IBEFAs this post has already gone on for quite a length, I will only briefly mention the main points of issue that the Indian industry have brought up. I will also upload their statements for our readers to view it on their own.

The Indian industry has chosen to respond via the India Brand Equity Foundation (IBEF) “on behalf of the Government of India”. IBEF is a Trust established by the Ministry of Commerce and Industry, Government of India. [Comment: This is curious on its own – that the government is choosing to respond to a working paper that discusses 4 drugs made by private manufacturers based in India.]

Calling it a smear campaign, the IBEF has raised several points of contention, including:

1. The NBER Paper has not been peer-reviewed, nor been subject to a review by the NBER Board – yet it is using NBER’s well established name to gain credibility. 

2. The methodology and treatment of samples make the conclusions suspect. Degradation by transportation and storage conditions, though mentioned, haven’t adequately been discussed. Similarly, counterfeit products (made elsewhere but with a “Made in India” label) have not adequately been accounted for. 

3. The samples were collected over 3 years but no mention of where and when the testing was done, and why it took 2 years to write this report. [Comment: It is true that the report is only 25 pages, of which 16 are text. However, there can be any number of reasons for taking 2 years to write this report – I’m not inclined to see this as ‘necessary’ to explain. However, I believe this 2 year question is brought up more as it makes it difficult/impossible to cross-check the samples after so long.]

4. The authors should have given the Indian manufacturers a chance to comment on the test observations. Pharmexcil’s Director General, Dr P.V. Appaji states that there are at least 8 technical issues with the sampling and data techniques.

5. Due to the above reasons, the report has no value and is written for the purpose of maligning India’s image in the international market.

6. Realising the serious damage that such an irresponsible report is trying to make, the pharma industry has asked the Government of India to initiate legal proceedings against all concerned.


As the one of the largest suppliers of generics, and the oft touted tag “pharmacy of the developing world”, it is certainly terrific that Indian manufacturers are being scrutinized.

And if one were to assume the paper’s conclusion as a valid one, this is naturally a very disturbing thought – that Indian manufacturers are intentionally exploiting poorer countries with weak regulatory oversight. And quite frankly, given anecdotal evidence, I personally would not be surprised by this from any manufacturer, not just Indian. And given the volume and necessity of Indian generics, quality should always be maintained as a top-priority. After all, the majority of the sick – the poor of the world – are not going to be able to afford the large branded drug prices.

However, anecdotes aside, real world studies must be undertaken with appropriate rigour and from my admittedly limited perspective, I believe this present study has certain important gaps it is required to fill, in order to be taken seriously. In the given context of India’s pharma regime being constantly scrutinized and attacked, unfairly or inadequately reporting on Indian manufacturers could be devastating on patient populations everywhere. [Indian Pharmaceutical Association has also listed some comments on the paper. Linked below]

All that being said – I see absolutely no reason for the Government of India to institute legal proceedings against the authors / publishers of the paper! Even if one were to assume that the authors were out to malign India – The only proper response is to appropriately deconstruct and de-credit that paper. It is an absolutely terrible signal to send out – that the Government of India is willing to sue academics! This would certainly have a chilling effect on any critical study done on the pharma industry – something that NO sector should be free from, especially not one as crucial as the pharma industry. And even strategically, this would only serve to further allegations of the Indian Government protecting its domestic manufacturers – for that’s exactly what they would be doing here. And giving any strength to that allegation would be devastating for India’s pharma patent regime.

Readers can look forward to a further examination of this legal threat in Part II – coming up soon. [Edit: Part II is now available here]

Relevant Links: 
1. Poor Quality Drugs and Global Trade: A Pilot Study (The paper in question)
2. IBEF Press release (The “official response”) 
3. IPA Comments on AEI study (Further comments by IPA)

Tags: , , , ,

About The Author

5 thoughts on “Trouble Brewing around NBER Paper on Indian Pharma – Part I”

  1. Vijaylakshmi Vardan

    This is an article which came in THE ECONOMIC TIMES of 6 October 2014:-
    [Editor’s note: Thanks for this comment. However, as the comment text is a reproduction of the article in the link provided, I am clipping the text out and leaving only the title and link here. Thanks]

    Oct 06 2014 : The Economic Times (Delhi)
    India’s Legal Threat Shocking Development, Feel US Researchers
    Soma Das and Dilasha Seth

  2. Thank you for your detailed analysis of our paper. We welcome such intelligent discourse.
    I’ll address some of the points you raise, and some IBEF raised, and my coauthors may feel inclined to address others.
    Sampling, timing and research questions
    1. We have been sampling medicines since 2007 and only with the aim of assessing problems with products in various markets, we never targeted any manufacturing nation in our samplings. After speaking with Ranbaxy whistleblower Dinesh Thakur abut practices at some Indian companies, we decided to investigate whether the same products performed worse in different locations – the NBER working paper is the result.
    2. Since the original research question was never about individual companies the sampling is far from perfect to that end. We investigated ciprofloxacin in more detail than other products hence the larger sample sizes, and in some instnaces products were not available in markets where we expected them to be. In others there is more restricted access (to TB meds) so sampling was more limited. My colleagues deployed the best statistical methods possible to limit and even nullify these biases – they can explain further if there is interest.
    3. Much of the quality data has appeared in the scientific peer review literature prior to the NBER working paper. These papers can be found on our website It is also worth appreciating a difference in the socio-economic publishing environment and the scientific/medical one. In the latter peer review is fast and hence working papers are less important, in socio-econ publishing can take 18-24 months, so working papers are used to garner feedback. We will take on board all the feedback we get and improve the paper before it appears in the peer review literature.
    4. We started work on this paper months before the Indian election and aimed to publish it to coincide with a Capitol Hill event we co-organized, featuring the heads of various donor/drug buying agencies (GFATM, PMI, OGAC), and we executed that plan last month. The release was not intentionally timed with Prime Minister Modi’s visit, although we certainly appreciated that his visit would increase possible coverage of the findings.
    5. We generally contacted companies about problems with their products. This effort was most significant in 2009/2010. When it comes to Indian companies the vast majority never replied. Those that did said that since we were not regulators or clients they would provide no information. We had only useful information from one Indian company. VS International confirmed to us a fake of its ciprofloxacin product sold in Nigeria, called Ciprotab – the product had incorrect batch numbering for the manufacturing date and had no API. We appreciated VSI responding to us and helping with the research. Indian companies (are not alone in) complaining about a lack of consultation when data are made public, but did very little to assist in data analysis prior to publication.
    6. Product degradation. It is possible that products identified as substandard had in fact degraded. But it is unlikely since NONE of the products (including rifampicin) made in EU/US in the same product categories failed quality analysis. There may be more subtle explanations – to do with transportation conditions from India to Africa, or inferior stability of some Indian products, which mean they are fine in perfect storage conditions and degrading rapidly in poor conditions. But I’d suggest that this is a problem if you intend on selling such products into environments where storage maybe imperfect.
    7. We accept all legitimate criticisms of our paper, which is why it is called a pilot. We want feedback, and most importantly, we need far more detailed and extensive studies of products (wherever made) on sale. The international community has done too little to measure this problem and we had hoped, still hope, that our paper will drive more detailed analysis, identify any problems and hopefully have them resolved. Litigation (or merely its threat) will undermine that effort.
    8. India is the pharmacy to the developing world. It makes the cheapest products on the planet (although pressures from Vietnam and China are increasing competition at the low cost end), and from what we’ve analyzed the vast majority are fine. But it has its faults, as you document on this excellent blog. Substandard products risk patient life and increase the threat of drug resistance at a population level. Drug resistance is a particular worry for infectious disease medicines – it is a natural process and doesn’t need any help from cost/corner cutting substandard production.

  3. Well, in an ideal world, an academics’ political affiliations would have no bearing on the research that they carried out, but since this is a non-ideal world we are living in, I wouldn’t put much stock into any paper coming from an author associated with the American Enterprise Institute; they are mostly right-wing shills

  4. Madhulika has helpfully sent across a few studies which indicate that rifampin degradation is often reported in due to elevated temperatures and exposure to air:

    Rifampin degradation has often been reported and considered to be due to oxidative degradation by air or elevated temperature. It seems to occur preferentially in aqueous solutions.
    (Lecaillon JB, Febvre N, Metayer JP, et al. Quantitative assay of rifampicin and three of its metabolites in human plasma, urine and saliva by high-performance liquid chromatography. J Chromatrogr 1978;145:319-24; Woo J, Wong CL, Teoh R, et al. Liquid chromatographic assay for the simultaneous determination of pyrazinamide and rifampicin in serum samples from patients with tuberculous meningitis. J Chromatogr 1987;420:73-80)

    A study conducted to evaluate rifampin stability in intravenous solutions showed a marked decrease in actual concentration for several hours.
    (Pearson SD, Trissel LA. Stability and compatibility of minocycline hydrochloride and rifampin in intravenous solutions at various temperatures. Am J Hosp Pharm1993;50:698-702.) ]

Leave a Comment

Scroll to Top