The topics are grouped in main sections, within each section there is a series of contributions set in ‘accordion’ (expanding when you click on the item’s headline). Within a contribution you will find links either to other URL or to pdf’ed texts for download. For larger writings (e.g. ‘manuals’) you may be asked to contact us to obtain access. This is a continuous project that will be updated regularly, also based on your feedback, comments and suggestions.
{01.Aug.2023 U: 26.Jun.2024 | ACPS-CdM}
GENERAL
WORLDWIDE – Generics
USA
EU
EU National
AUSTRALIA
PHILIPPINES
WHO
PHARMACOVIGILANCE
EU (central & national)
SPECIAL (SAFETY) CONDITIONS
{06.Jun.2020 – U: 05.Sep.2024 | ACPS-CdM}
EU [EMA]
US [FDA]
{14.Feb.2024 | ACPS-CdM}
Excellent platforms to search disease specifications (epidemiology, etiology, diagnosis, treatment, prognosis, etc.):
{ACPS-CdM: 15.Sep.2020 | U: 27.Mar.2022}
Developmental Toxicology
Drug-induced Liver Injury
Comparative Toxicogenomics
Drugs & Pregnancy
Drug Passage to Milk
Consumer Product Information Database
Occupational Exposure to Chemicals
{ACPS-CdM: 03.Oct.2022 | U: 07.Feb.2024}
{ACPS-CdM: 03.Oct.2022}
EU Pharmacovigilance
US Pharmacovigilance
eHEALTHMed Database
Using real-world big data and innovative AI/ML algorithms, eHealthMe provides a platform for everyone to run phase IV trial to see how drugs do to themselves individually. The studies are personalized to gender, age (0-99+), drugs and conditions, and are continuously updated.
SPECIAL (SAFETY) CONDITIONS
{ACPS-CdM: 11.Apr.2023}
Collections:
Obesity research:
Paediatrics:
Partitioning of body water (relative to body weight)
Carboplatin dose (AUC) calculator
{27.Mar.2018 U: 17.Sep.2020 | ACPS-CdM}
Organ specific
ICU focused
Disease severity on admission | and use it to predict outcome
Shock
Trauma Severity
Scores that assess the presence and severity of organ dysfunction
Scores that assess nursing workload use
Traumatic Brain Injury (TBI)
{12.May.2020 | ACPS-CdM}
{under construction} soon to com!
{12.May.2020 | ACPS-CdM}
Renal function:
CKD: Chronic Kidney Disease
AKI: Acute Kidney Injury
{27.Mar.2018 U: 10.May.2022 | ACPS-CdM}
EEA | EMA
General provisions
Fixed Dose Combinations
Modified Release Products – General
Oral Modified-Release products
Transdermal patches
LaLa: Locally Applied, Locally Acting Products
LaLa: Locally Applied, Locally Acting Products: Gastrointestinal tract
LaLa: Locally Applied, Locally Acting Products: Nasal & Orally Inhaled Products (OIP)
LaLa: Locally Applied, Locally Acting Products: Topical (skin) products
GCP-Inspections (BABE)
US | FDA
General – General principles of therapeutic equivalence
General – CFR
General – Statistical Approaches
General – Reserve & Retention Samples
Early Pharmacokinetics – General Considerations
BA & BE for Oral Products – General Considerations
BA (& BE) for Oral Products – NDAs or INDs
BE for Oral Products – ANDAs
Product-specific BE Guidance
Biowaiver
Combination Products
Modified-release Products
See also: Wang YL, Chang YT, Yang SY, Chang YW, Kuan MH, Tu CL, Hong HC, Lai IC, Gau CS, Hsu LF. Approval of modified-release products by FDA without clinical efficacy/safety studies: A retrospective survey from 2008 to 2017. Regul Toxicol Pharmacol. 2019 Apr;103:174-180.
Topical – Locally applied, Locally acting
LaLa – Dermatological
LaLa – Nasal
Biosimilarity
Therapeutic Proteins
Drug-device Combinations: Bridging
ICH
WHO
ASEAN
AUSTRALIA
BRASIL
CANADA
EGYPT
INDIA
JAPAN
NEW ZEALAND
RUSSIA
THAILAND: see ASEAN
{27.Mar.2018 | Update:11.Aug.2024 | ACPS-CdM}
BABE-reporting templates including BA-specification:
There is no GLP mandate for the bioanalytical (BA) determinations of the PK-levels; the planning, conduct, quality management and reporting of the BA work must be GCP-compliant; however, it is advisable that BA work is set up, executed and quality managed in accordance with GLP-principles. The methods must be well characterised, fully validated and documented to yield reliable and valid results. These procedures are subject to explicit regulatory guidance:
In general, the bioanalytical methods are to be validated prior to the study with regard to the 1) stability of the stock solutions and of the analyte(s) in the biological matrix under processing conditions and during the entire period of storage, 2) specificity, 3) accuracy, 4) precision, 5) limit of quantitation and 6) response function (acc. EMA). FDA-Guidance requires documentation of stability along with 1) accuracy, 2) precision, 3) sensitivity, 4) specificity, 5) linearity, and 6) reproducibility. During the assays for the study itself, stability, accuracy and precision are to be surveyed further.
All procedures should be performed according to pre-established Standard Operating Procedures (SOPs). All relevant procedures used to validate the method should be submitted to the authorities as part of the study documentation; any modification of the method during the study may require adequate revalidation.
Attention is drawn on the need to report specific BA-method information in the EU Summary Sheet of BA/BE-studies (EMA/CHMP/600958/2010/Corr. Appendix IV of the Guideline on the Investigation on Bioequivalence: Presentation of Biopharmaceutical and Bioanalytical Data in Module 2.7.1 (Nov.2011).
{27.Mar.2018 – U: 18.Mar.2023 | ACPS-CdM}
Background – Recommended Reading
(Regulatory) Guidance
BCS Monographs
{ACPS-CdM: 20.Oct.2020}
Background & rationale
An information guide for patients published by the European Commission explains in a clear, unbiased way what biosimilar medicines are, how they are developed and approved in the EU and what patients can expect in terms of availability and safety.
Information guide for health care providers providing reference on the science and regulation underpinning the use of biosimilar medicines.
Regulatory Guidelines
List of centrally approved biosimilars
For the list of biosimilar medicines approved via the EMA centralised procedure,
see: Centrally authorised biosimilar medicines
{01.Sep.2023 U: 16.Feb.2024 | ACPS-CdM}
Establishing and maintaining blinding throughout the complex process of capturing and analysing data may be challenging, but avoids compromising objectivity. Bioanalytical (BA) determinations and pharmacokinetic (PK) analyses are at risk of being result-driven unless blinded. In the EU, BABE Guidance specifically instructs that bioanalysis should be conducted without information on treatment. However, most bioanalysts will be aware of the subject and period ID; no such restrictions are imposed with regard to the PK-analysts.
Blinding could be established by:
We consider these steps to be highly recommendable; however, we are not aware of any guideline that encourages/enforces such blinding, at PK-analyst level, in particular.
{27.Mar.2018 U: 25.Oct.2020| ACPS-CdM}
Most regulatory pharmacokinetic (PK) arguments rely on summary values to describe and compare the time courses of the blood/plasma concentrations (the “PK-profile”) across treatments (e.g. formulations, doses, co-medications, co-morbidities, etc.).
To this purpose, the descriptors of the PK-profile should be chosen such that they are sufficient to identify meaningful differences in systemic exposure that might be relevant for the treatment outcome in terms of efficacy and safety.
This is generally accepted to be well covered by non-compartmental analysis (NCA). NCA relies on very basic non-pretentious mathematics and it does not impose model-assumptions; therefore NCA is unlikely to be confounded by whether the chosen mechanistic interpretation of the PK-profile is ‘correct’ or not.
While relying on very basic mathematics it can be easily processed with unsophisticated software. There is a large range of PK software tools and packages, most of which also containing NCA: see for instance PharmPK-List of PK-Software.
Whatever software is chosen for PK-analysis (± statistics), it is crucial responsibility i) to keep the software up-to-date, ii) to check for differences between new and former versions of the same package, and iii) to crosscheck with alternative software. This help avoiding embarrassing revokes by the regulatory authorities who obviously also keep a close eye on the software used (see Morales-Alcelay S et al. On the Incorrect Statistical Calculations of the Kinetica Software Package in Imbalanced Designs. AAPS J. 2015 Jul;17(4):1033-4.)
Among the many options available, we have excellent experience with PCModfit, which operates with all analytical steps well documented within a single platform; this is important for documenting the selection of the data points used to derive the apparent terminal log-linear disposition rate constant and half-life. This tool operates reliably, and we have documented excellent agreement of its results with other software tools (incl. WinNonlin). Also, PCModfit has some really outstanding options for model-specific and model-unrelated estimation of the time courses of the plasma concentrations on repeated dosing (allowing for asymmetric dose units and dosage intervals) and for nonlinear model-fitting.
There is no regulatory ruling on using specific PK-software; however, it is a well-established urban myth that regulators would only accept analyses processed with the NCA-subroutines of expensive pharmacometric software platform packages such as Phoenix WinNonlin. Surely this package is reasonably claimed to be “The Industry Standard for PK/PD Modeling and Simulation”; however, there is no reason to consider it the standard for NCA; also, the cost is high and out of proportion for just doing NCA.
{27.Mar.2018 U: 22.Mar.2021 | ACPS-CdM}
Most regulatory pharmacokinetic (PK) arguments rely on summary values to describe and compare the time courses of the blood/plasma concentrations (the “PK-profile”) across treatments (e.g. formulations, doses, co-medications, co-morbidities, etc.). This is generally accepted to be well covered by non-compartmental analysis (NCA). NCA relies on very basic non-pretentious mathematics and it does not impose model-assumptions; therefore is unlikely to be confounded by whether the chosen mechanistic interpretation of the PK-profile is ‘correct’ or not.
NCA derives simple descriptors of the PK-profile: maximum observed concentrations (Cmax), time of occurrence of Cmax after dosing (tmax), quantifiable i.e. truncated area under the time course of the concentrations (AUC_tz), total i.e. extrapolated AUC (AUC_∞), apparent terminal disposition rate constant (λ), apparent terminal disposition half-life (t½), the quantifiable and total area under the statistical first-moment curve (AUMC), and mean residence time (MRT).
Among the NCA-criteria, Cmax and AUC_∞ are of prime relevance while representing the peak and total (area) systemic exposure; the MRT is a highly robust expression of the timely distribution of the AUC (since it relies on all profiling points).
Cmax/AUC_∞ is useful since it allows qualifying differences in the rate of systemic bioavailability while reflecting changes in Cmax disproportionate to changes in the amount of bioavailability.
For good reason, the AUC_∞ (=F.Dose/CL) is a prime characteristic since it allows quantifying differences in the fractional amount of systemic bioavailability (F) across treatments if the clearance (CL) can be accepted to be treatment-independent.
Since the AUC_∞ expresses the balance between the amount of systemically bioavailable drug (F.Dose) and the disposition thereof (CL), the latter can be calculated from the AUC if F is known: CL = F.Dose/AUC_∞. F is known only for intravascular dosing (iv-bolus or iv-infusion); in contrast, for extravascular dosing F is generally incomplete to an unknown extent.
Nevertheless, NCA-programs often calculate Dose/AUC_∞ and report it as CL/F. This so-called ‘apparent oral clearance’ is not meaningful since it results from two ‘unknowns’ (F and CL) and most certainly does not reflect actual CL or changes thereof in any meaningful fashion. If anything, CL/F is nothing but the reciprocal of the dose-normalised AUC_∞.
Another such oddity is that NCA-programs often report Vd/F (‘apparent oral distribution volume’), which can only be used as an estimate of the distribution volume (Vd) if F were known (Vd/F = CL/λ or Vd/F = CL.MRT). Although it is no measure of volume, Vd/F serves some mechanistic role since it represents the composite dilutional effect on how a given dose results in the observed concentrations depending on a) the fraction of the ‘dose’ actually reaching in the systemic circulation and b) how this residual is then diluted by the impact of all distribution compartments.
We advocate not using CL/F or Vd/F unless F is reliably known. This is particularly important in SmPC-descriptions of a drug’s PK-behaviour since the terms are often used without clearly specifying them as ‘apparent’ in this context.
———————–
This is how we would have liked to leave things until recently. To our surprise, CL/F and Vd/F are mentioned in the draft 2019 FDA Guidance on FED studies as criteria that should be reported (FDA Guidance for Industry – Assessing the Effects of Food on Drugs in INDs and NDAs – Clinical Pharmacology Considerations – Draft – Feb.2019). Possibly, this stems from the same desk that lists CL/F as “physiologically plausible structural element” of a population PK model (FDA Guidance to Industry – Population Pharmacokinetics – Jul.2019). Rarely seen a better illustration of the intellectual constraints of model-driven pharmacometrics.
{09.May.2018 – Update:22.Oct.2020 | ACPS-CdM }
Well-groomed regulatory rules are in place to facilitate the development and authorization of multi-origin, i.e. generic drugs. These rules are focused on developing mere copies of the originator and securing their prescribability by evidencing average bioequivalence (ABE). The rulings do not secure switchability (unless constraints are imposed that limit the asymmetry of the GMR point estimates). Little is left from the exciting biostatistical debate in the 1990’s about alternative approaches such as individual and population BE but for the now well-established practice of replication designs and reference-scaling for high-variability drugs and drug products. Also, several limitations become apparent when conventional ABE-rulings (as conceived for oral immediate-release products) are amended to regulate other regulatory scenarios (e.g. modified-release products and fixed-combination products).
In the following, several such topics are discussed. For each topic, references are presented that guide further reading.
Marketing authorization of multi-origin (generic) drugs is based on the argument of equivalence vs the originator. To this purpose, therapeutic equivalence studies can be waived if pharmacokinetic equivalence can be evidenced (“bioequivalence” [BE]). This is generally based on the testing of average bioequivalence (ABE), i.e. on comparing the 90% confidence interval estimates of the true ratios of the treatment means of the peak and area exposure levels for test:reference relative to a predefined default tolerance zone of 80.00 to 125.00%. Such ABE is the well-established quality mark of in vivo biopharmaceutical performance that is accepted to suffice for the regulatory authorization of generics (prescribability).
Since the early 1990s, the concern is commonly expressed that ABE is not well-suited to evidence comparability between (different) formulations in terms of within-subject switchability (switching R to T rather than repeating R). Other approaches such as individual and population BE might be better suited – biostatistically – to assess interchangeability of multi-origin products within the target population.
In the late 1990s, the FDA published several draft guidance documents favouring such alternative approaches. Individual bioequivalence is particularly well-suited for certain drugs by addressing within-subject variabilities, allowing for scaling of the tolerance zone to the reference variability, and by assessing subject-by-formulation interaction.
In spite of their biostatistical elegance and mathematical sophistication, these alternatives have never been a real challenge to ABE as the decisive quality mark for the authorization of generics. However, these discussions helped to endorse the use of replication designs and reference scaling (possibly with a point-estimate constraint) for ABE of high-variability drugs (HVD); others might argue that high-variability itself is a sufficient reason not to use ABE (even with replication) for the validation of generic products.
There is no upper limit on the sample size for ABE. Therefore, it is possible to force the interval estimate of the relative exposure levels within the tolerance zone in spite of a large asymmetry of the point estimate by using a very large sample. This asymmetry increases the risk of poor switchability: although generic A (BE point estimate 0.9) and generic B (BE point estimate 1.10) are each equivalents to the reference originator, they are not equivalent among themselves (BE point estimate: 1.22 for B to A).
This phenomenon has been referred to as bioavailability or bioequivalence “drifting”.
This issue and the need to resolve possible concerns about it are not addressed in the respective guidelines for the testing of bioequivalence of immediate release or modified release formulations. By the intervention of the Czech regulatory authorities (see Clinical Aspects Of The Development Of Fixed Dose Combination Products by Jiří Haman & Marina Feřtek (SÚKL)), it comes up as a rather enigmatic comment in the guideline on fixed-dose combination with possibly quite broad implications (since little is known about how to resolve this issue).
Inadequacy of the common ABE practice to resolve concerns about switchability is not just a biostatistical issue. To an important extent, treatment adherence relies on how the patient relates to his/her medication. This depends on the product’s name, package, form, size, smell, taste, etc. These aspects ought to be taken into account when considering switching a patient on chronic therapy to an alternative product. In the EU, this is particularly important, since a generic product and a reference product may be considered to have the same pharmaceutical form if they have the same form of administration as defined by the Pharmacopoeia; furthermore, Article 10(2)(b) of the amended Directive 2001/83/EC [Community Code] provides that the various immediate-release oral forms, i.e. tablets, capsules, oral solutions, and suspensions, are considered to be the same pharmaceutical form for the purposes of Article 10 (see also Eudralex Volume 2A Notice to Applicants – Section 5.3.2.1).
REFERENCES
Prescribability and Switchability
Switchability and BA/BE-drifting
Individual BE
{ACPS-CdM: 08.Oct.2020}
Replication Designs and Reference Scaling for High-Variability Drugs
In the EU, fixed-dose combinations (FDC) are regulated by EMA/CHMP/158268/2017 (“Guideline on clinical development of fixed combination medicinal products” – Mar.2017); these should best be read in conjunction with the public review comments when the guideline was drafted (EMA/757184/2015 “Submission of comments on ‘Guideline on Clinical Development of Fixed Combination Medicinal Products’ – Oct.2018). In comparison, in the US, there is an elaborate Guidance for newly developed FDC (FDA Guidance for Industry – Codevelopment Of Two Or More New Investigational Drugs For Use In Combination – Jun.2013), but not for FDC under an ANDA.
Any FDC-product, in the EU, is expected to meet the following three basic scientific requirements:
Three therapeutic scenarios are foreseen: add-on treatment, substitution therapy and initial combination treatment; each requires its own type of evidence.
For generic-FDC, the documentation expected to be presented largely depends on i) the scenario of development and authorisation of the reference-FDC (“add-on” | “substitution” – see below) and ii) the extent of pivotal documentation on the efficacy and safety of the combination available for the reference-FDC (or for the loose combination upon which the authorisation of the reference-FDC took reference). Also, the Guideline has introduced several calls for caution (raising concerns that require extra work) that lack precision and distinction (also depending on what may be expected for an “add-on”-FDC differently from a “substitution”-FDC.
The need for studies and the type of studies for generic FDC-products is regulated by FDC-Guideline (EMA/CHMP/158268/2017 “Guideline on clinical development of fixed combination medicinal products” – Mar.2017). Only a relatively short Paragraph (section 4.5) is dedicated to generic-FCDs. Understanding this section and its implications is a complex challenge since it is full of riddles.
In order to understand the implications, you need to clarify upfront:
With this in mind, the next step is to read carefully what is specified in the pertinent section of the Guideline:
EMA/CHMP/158268/2017 – Section 4.5 Generic medicinal products
» The development of a generic medicinal product is based on demonstrating bioequivalence with the reference fixed combination medicinal product (see also first paragraph section 4.6) «
See below regarding possible implications of Section 4.6
» This should be demonstrated for all active substances in the fixed combination medicinal product according to the relevant guidelines that are mentioned in section 3. «
In this case, this means that CPMP/EWP/QWP/1401/98 Rev. 1 (“Guideline On The Investigation Of Bioequivalence” – Jan.2010) needs to be complied with.
However, it is our understanding that a straightforward BE-comparison of the generic-FDC with its originator-FDC will only suffice if pivotal phase-3 studies were carried out with the originator-FDC or with the loose combination upon which the authorisation of the originator-FDC took reference. In the absence of such data, the applicant for marketing authorisation of the generic-FDC may be expected to present extra data; however, it is unclear what data this ought to be.
» …. Also, for generic fixed combination medicinal products it needs to be verified that the evidence base that may have been generated for the reference product with individual active substances (rather than with the fixed combination medicinal product, to which reference is being made) applies to the generic fixed combination medicinal product. In this case, two pharmacokinetics bridges may need to be built, one between the reference fixed combination medicinal product and its active substances and one between the generic and reference fixed combination medicinal product. «
For FDCs that are positioned as “substitution” this would mean “two pharmacokinetics bridges may need to be built, one between the reference fixed combination medicinal product and its active substances taken simultaneously …” (i.e. testing for equivalence or absence of formulation effect).
In contrast, for FDCs developed as “add-on” the requirement would need to mean “two pharmacokinetics bridges may need to be built, one between the reference fixed combination medicinal product and its active substances taken separately …“. (i.e. testing for interaction).
In both cases, the requirement imposed on the generic is odd since it implies, doing, repeating or extending the work that ought to have been done by the originator in the first place.
The last sentence of section 4.5 raises even more questions:
» A justification should be provided why ‘drifting’ of bioavailability is not considered relevant and hence why the original demonstration of efficacy and safety is relevant to the generic «
This sentence “hangs” without context. In order to understand this, a look needs to be taken on EMA/757184/2015 (“Submission of comments on ‘Guideline on Clinical Development of Fixed Combination Medicinal Products’ (EMA/CHMP/ 281825/2015)” – Oct.2018), which reports this concern extensively. It relates to shifts in the between-treatment contrasts of complex bioequivalence series; in simplest terms: generic-A is equivalent with R (GMR for A/R: 0.90), generic-B is equivalent with R (GMR for B/R: 1.10); since both are equivalent with the reference, they are “prescribable”; however, they are obviously not “switchable” (GMR fort B/A: 1.22 – see also Section 5.4.2).
Applied in the present context, this might mean:
i) reference-FDC{AB} is equivalent with {A}+{B}(the loose combination of both components given simultaneously)
ii) test-FDC{AB} is equivalent with reference-FDC{AB}
→ can this be accepted to mean that test-FDC{AB} is equivalent with {A}+{B}?
It is our understanding that this might (only) be resolved without an extra study (like the one recommended in Section 4.6 of the FDC-Guideline)
if the contrasts for i) and ii) are not too asymmetric (i.e. their GMR Point Estimates are close to 1.00).
However in some PAR we found the awkward statement that BA/BE drfiting was not a problem since pivotal studies had been undertaken with the reference (we wonder how the assessor came to this intriguing conclusion).
This hurdle is repeated in the next section of the FDC-Guideline:
EMA/CHMP/158268/2017 – 4.6. Bridging the evidence base to the fixed combination medicinal product
» Clinical data establishing the contribution of each active substance and the positive benefit-risk are often obtained from the combined use of individual active substances. In this case demonstration of similar pharmacokinetics (usually through demonstrating bioequivalence) of the fixed combination medicinal product versus its individual active substances taken simultaneously is required. This is to satisfy the third basic requirement for an MAA for fixed combination medicinal products «
Irrespective of the type of FDC (“add-on” or “substitution”) this means that the test-FDC{AB} needs to be tested for equivalence (i.e. absence of formulation effects) with the loose combination of the components {A} and {B} that founded the authorisation of the originator administered simultaneously. Obviously, it may prove very difficult to identify the precise products that were used to this purpose. Furthermore, when taking reference on the safety of {A} or {B}, this cannot be confined to data on the combined use of {A}+{B} alone; unavoidably, reference will also need to be taken on the use of {A} or {B} alone.
In consequence, a generic-FDC needs to be tested for bioequivalence vs the originator-FDC. Extra studies may be demanded if no pivotal efficacy and safety studies were carried out with the originator-FDC product or with the loose combination of its components upon which the authorisation of the originator-FDC took reference. This could mean:
We think that both sections 4.5 and 4.6 of the FDC-Guideline ought to have been less confused & confusing. Applicants and sponsors would have benefited from better guidance.
{ACPS-CdM: 10.Oct.2020 | U: 25.Jan.2021}
In Applied Clinical Pharmacology we work with data. Within a GCP-framework, we install and manage risk-based Quality Management Systems, and operate within such RB-QMS to ensure that trial data is “credible” (per ICH GCP E6(R2)), i.e. fit to serve a tangible worthwhile (scientific, medical/ethical) purpose. Accordingly, we have a first-line involvement on how data are selected, weighted, collected, recorded, verified, processed and reported. Even if important parts of this responsibility reside with the sponsor and the sponsor may choose to outsource such services, a CP-investigator may be expected to have a sound working knowledge of statistics in order to communicate and collaborate efficiently with these expert support groups and/or statistical co-investigators.
In the following, we present a series of statistical topics that have implications on how we, as non-statisticians, communicate/collaborate with statisticians. Unavoidably, one of the hot issues in this domain is the “tyranny of statistical significance” and the “need for a ban on p-values!” This ought to be high on the list of our priorities also in the perspective of whether and how these issues might be resolved by estimation (confidence intervals) rather than hypothesis testing.
This is a life project and more topics will follow (noninferiority, staged designs, multiplicity and hierarchy of outcome criteria, etc..).
The topic is a bit ‘nerdy’. But we try to lighten this up by using collapsable text items. In the list of references “recommended for reading”, we have added a click button to each reference that allows you to access the abstract offline; in addition, we have added links to abstract or full paper pdfs online.
{10.Nov.2023 | ACPS-CdM}
General statistical resources
Samples size calculators
FARTSSIE: “Free Analysis Research Tool for Sample Size Iterative Estimation”.
{if used for BE trial planning, take also a look at H. Schütz (BEBAC): Sample Site Estimation (Bucharest 2013 Lecture)}
Statistical fragility Index
The fragility index is a measure of the robustness (or fragility) of the results of a clinical trial. The fragility index is a number indicating how many patients would be required to convert a trial from being statistically significant to not significant (p ≥ 0.05). The larger the fragility index the better (more robust) a trial’s data are. The intent of the fragility index is to be used in conjunction with the P-value, 95% confidence interval, and various measures describing benefit or risk (relative risk reduction, absolute risk reduction, etc): ClinCalc.com | Fragility Index
False Positive Riks (FPR) acc. Colquhoun (2019)
Colquhoun (2019) proposes continuing the use of continuous p-values, but only in conjunction with the “false positive risk (FPR).” The FPR answers the question, “If you observe a ‘significant’ p-value after doing a single unbiased experiment, what is the probability that your result is a false positive?” It tells you what most people mistakenly still think the p-value does, Colquhoun says. The problem, however, is that to calculate the FPR you need to specify the prior probability that an effect is real, and it’s rare to know this. Colquhoun suggests that the FPR could be calculated with a prior probability of 0.5, the largest value reasonable to assume in the absence of hard prior data. The FPR found this way is in a sense the minimum false positive risk (mFPR); less plausible hypotheses (prior probabilities below 0.5) would give even bigger FPRs, Colquhoun says, but the mFPR would be a big improvement on reporting a p-value alone. He points out that p-values near 0.05 are, under a variety of assumptions, associated with minimum false positive risks of 20–30%, which should stop a researcher from making too big a claim about the “statistical significance” of such a result:
Calculator for False Positive Risk (FPR) (version for unequal sample sizes)
Calculating and Reporting Effect Sizes to Facilitate Cumulative Science
Reference is set to the paper of Daniel Lakens (Lakens D. Calculating and reporting effect sizes to facilitate cumulative science: A practical primer for t-tests and ANOVAs. Frontiers in Psychology 2013;4:863). There had been some misspellings in the equations, but most of these have been resolved in the worksheets and the precious R-programs that have been developed in parallel.
{27.Mar.2018 U: 22.Feb.2022 | ACPS-CdM}
I keep six honest serving-men (They taught me all I knew). Their names are What and Why and When and How and Where and Who. Rudyard Kipling (1902)- “The Elephant’s Child” from Just So Stories
In experimental clinical pharmacology, we usually generate rather than test (verify, i.e. falsify) hypotheses. We are not often directly exposed to what Stang et al. 2010 termed the “tyranny of statistical significance”, which they described as:
» It has been stated that the P-value is perhaps the most misunderstood statistical concept in clinical research. As in the social sciences, the tyranny of SST is still highly prevalent in the biomedical literature even after decades of warnings against statistical significance testing (SST). The ubiquitous misuse and tyranny of SST threatens scientific discoveries and may even impede scientific progress. In the worst case, misuse of significance testing may even harm patients who eventually are incorrectly treated because of improper handling of P-values. «
Growing out and beyond the relatively secure professional environment of a research job but entering the privileged charade of decision making in pharmaceutical drug development, I became increasingly exposed to this “tyranny” although challenging it most stubbornly. I soon discovered that whatever my seniority, my fight against idle p-values was very much in vain. P-values mattered with stakeholders and stockholders; they mattered to, and with regulators; they were crucial for investigators who wanted to publish their trial data, etc. A P value on the right side of whatever critical level, makes and breaks drugs, efforts, and people.
» The p-value quantifies the probability of observing results at least as extreme as the ones observed given that the null hypothesis is true. It is then compared against a pre-determined significance level (α). If the reported p-value is smaller than α the result is considered statistically significant. « {Vidgen B, Yasseri T. P-Values: Misunderstood and Misused. Frontiers Physics 2016}
» Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g. the sample mean difference between two compared groups) would be equal to or more extreme than its observed value «{Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician 2016}
Note: unsure whether you should write “P value”, “p value”, “P value”, or “P-value”? – See: GraphPad KNOWLEDGEBASE – ARTICLE #1885
Thus, P-values mean something else than what most users and readers might think. It is easier to understand the issue by discussing what P isn’t and how it ought not to be used, as per Greenland
Some of the most severe distortions of the scientific literature produced by statistical testing involve erroneous comparison and synthesis of results from different studies or study subgroups. Among the worst are:
When the same hypothesis is tested in different studies and none or a minority of the tests are statistically significant (all P> 05), the overall evidence supports the hypothesis. No! - Tick for more details
This belief is often used to claim that a literature supports no effect when the opposite is case. It reflects a tendency of researchers to “overestimate the power of most research”. In reality, every study could fail to reach statistical significance and yet when combined show a statistically significant association and persuasive evidence of an effect. For example, if there were five studies each with P = 0.10, none would be significant at 0.05 level; but when these P-values are combined using the Fisher formula, the overall P-value would be 0.01. There are many real examples of persuasive evidence for important effects when few studies or even no study reported “statistically significant” associations. Thus, lack of statistical significance of individual studies should not be taken as implying that the totality of evidence supports no effect.
When the same hypothesis is tested in two different populations and the resulting P-values are on opposite sides of 0.05, the results are conflicting. No! - Tick for more details
Statistical tests are sensitive to many differences between study populations that are irrelevant to whether their results are in agreement, such as the sizes of compared groups in each population. As a consequence, two studies may provide very different P-values for the same test hypothesis and yet be in perfect agreement (e.g., may show identical observed associations). For example, suppose we had two randomized trials A and B of a treatment, identical except that trial A had a known standard error of 2 for the mean difference between treatment groups whereas trial B had a known standard error of 1 for the difference. If both trials observed a difference between treatment groups of exactly 3, the usual normal test would produce P = 0.13 in A but P = 0.003 in B. Despite their difference in P-values, the test of the hypothesis of no difference in effect across studies would have P = 1, reflecting the perfect agreement of the observed mean differences from the studies. Differences between results must be evaluated directly, for example by estimating and testing those differences to produce a confidence interval and a P-value comparing the results (often called analysis of heterogeneity, interaction, or modification).
When the same hypothesis is tested in two different populations and the same P-values are obtained, the results are in agreement. No! - Tick for more details
Again, tests are sensitive to many differences between populations that are irrelevant to whether their results are in agreement. Two different studies may even exhibit identical P-values for testing the same hypothesis yet also exhibit clearly different observed associations. For example, suppose randomized experiment A observed a mean difference between treatment groups of 3.00 with standard error 1.00, while B observed a mean difference of 12.00 with standard error 4.00. Then the standard normal test would produce P= 0.003 in both; yet the test of the hypothesis of no difference in effect across studies gives P = 0.03, reflecting the large difference (12.00 − 3.00 = 9.00) between the mean differences.
If one observes a small P-value, there is a good chance that the next study will produce a P-value at least as small for the same hypothesis. No! - Tick for more details
This is false even under the ideal condition that both studies are independent and all assumptions including the test hypothesis are correct in both studies. In that case, if (say) one observes P= 0.03, the chance that the new study will show P ≤ 0.03 is only 3 %; thus the chance the new study will show a P-value as small or smaller (the “replication probability”) is exactly the observed P-value! If on the other hand the small P-value arose solely because the true effect exactly equaled its observed estimate, there would be a 50 % chance that a repeat experiment of identical design would have a larger P-value. In general, the size of the new P-value will be extremely sensitive to the study size and the extent to which the test hypothesis or other assumptions are violated in the new study; in particular, P may be very small or very large depending on whether the study and the violations are large or small.
Caution is also required with regard to misinterpretations of confidence intervals as providing sharp answers when none are warranted. The hypothesis which says the point estimate is the correct effect will have the largest P-value (P = 1 in most cases), and hypotheses inside a confidence interval will have higher P-values than hypotheses outside the interval. The P-values will vary greatly, however, among hypotheses inside the interval, as well as among hypotheses on the outside. Also, two hypotheses may have nearly equal P-values even though one of the hypotheses is inside the interval and the other is outside. Thus, if we use P-values to measure compatibility of hypotheses with data and wish to compare hypotheses with this measure, we need to examine their P-values directly, not simply ask whether the hypotheses are inside or outside the interval. This need is particularly acute when (as usual) one of the hypotheses under scrutiny is a null hypothesis.
The specific 95% confidence interval presented by a study has a 95% chance of containing the true effect size. No! - Tick for more information
A reported confidence interval is a range between two numbers. The frequency with which an observed interval (e.g., 0.72–2.88) contains the true effect is either 100% if the true effect is within the interval or 0% if not; the 95 % refers only to how often 95% confidence intervals computed from very many studies would contain the true size if all the assumptions used to compute the intervals were correct. It is possible to compute an interval that can be interpreted as having 95% probability of containing the true value; nonetheless, such computations require not only the assumptions used to compute the confidence interval, but also further assumptions about the size of effects in the model. These further assumptions are summarized in what is called a prior distribution, and the resulting intervals are usually called Bayesian posterior (or credible) intervals to distinguish them from confidence intervals.
An effect size outside the 95% confidence interval has been refuted (or excluded) by the data. No! - Tick for more information
As with the P-value, the confidence interval is computed from many assumptions, the violation of which may have led to the results. Thus it is the combination of the data with the assumptions, along with the arbitrary 95 % criterion, that are needed to declare an effect size outside the interval is in some way incompatible with the observations. Even then, judgements as extreme as saying the effect size has been refuted or excluded will require even stronger conditions.
If two confidence intervals overlap, the difference between two estimates or studies is not significant. No! - Tick for more information
The 95 % confidence intervals from two subgroups or studies may overlap substantially and yet the test for difference between them may still produce P< 0.05. Suppose for example, two 95 % confidence intervals for means from normal populations with known variances are (1.04, 4.96) and (4.16, 19.84); these intervals overlap, yet the test of the hypothesis of no difference in effect across studies gives P = 0.03. As with P-values, comparison between groups requires statistics that directly test and estimate the differences across groups. It can, however, be noted that if the two 95 % confidence intervals fail to overlap, then when using the same assumptions used to compute the confidence intervals we will find P < 0.05 for the difference; and if one of the 95 % intervals contains the point estimate from the other group or study, we will find P > 0.05 for the difference.
An observed 95% confidence interval predicts that 95 % of the estimates from future studies will fall inside the observed interval. No! - Tick for more information
This statement is wrong in several ways. Most importantly, under the model, 95 % is the frequency with which other unobserved intervals will contain the true effect, not how frequently the one interval being presented will contain future estimates. In fact, even under ideal conditions the chance that a future estimate will fall within the current interval will usually be much less than 95 %. For example, if two independent studies of the same quantity provide unbiased normal point estimates with the same standard errors, the chance that the 95 % confidence interval for the first study contains the point estimate from the second is 83 % (which is the chance that the difference between the two estimates is less than 1.96 standard errors). Again, an observed interval either does or does not contain the true effect; the 95 % refers only to how often 95 % confidence intervals computed from very many studies would contain the true effect if all the assumptions used to compute the intervals were correct.
If one 95% confidence interval includes the null value and another excludes that value, the interval excluding the null is the more precise one. No! - Tick for more information
When the model is correct, precision of statistical estimation is measured directly by confidence interval width (measured on the appropriate scale). It is not a matter of inclusion or exclusion of the null or any other value. Consider two 95 % confidence intervals for a difference in means, one with limits of 5 and 40, the other with limits of −5 and 10. The first interval excludes the null value of 0, but is 30 units wide. The second includes the null value, but is half as wide and therefore much more precise.
Also, 95 % confidence intervals force the 0.05-level cut-off on the reader, lumping together all effect sizes with P > 0.05, and in this way are as bad as presenting P-values as dichotomies. Nonetheless, many authors agree that confidence intervals are superior to tests and P-values because they allow one to shift focus away from the null hypothesis, toward the full range of effect sizes compatible with the data—a shift recommended by many authors and a growing number of journals.
Call for “détente cordiale” between null hypothesis significance testing (NHST) and Bayesian a posteriori approaches
Ruberg SJ. Détente: A Practical Understanding of P values and Bayesian Posterior Probabilities. Clin Pharmacol Ther. 2021 Jun;109(6):1489-1498. {ABS-Link} Click for Details » Null hypothesis significance testing (NHST) with its benchmark P value < 0.05 has long been a stalwart of scientific reporting and such statistically significant findings have been used to imply scientifically or clinically significant findings. Challenges to this approach have arisen over the past 6 decades, but they have largely been unheeded. There is a growing movement for using Bayesian statistical inference to quantify the probability that a scientific finding is credible. There have been differences of opinion between the frequentist (i.e., NHST) and Bayesian schools of inference, and warnings about the use or misuse of P values have come from both schools of thought spanning many decades. Controversies in this arena have been heightened by the American Statistical Association statement on P values and the further denouncement of the term “statistical significance” by others. My experience has been that many scientists, including many statisticians, do not have a sound conceptual grasp of the fundamental differences in these approaches, thereby creating even greater confusion and acrimony. If we let A represent the observed data, and B represent the hypothesis of interest, then the fundamental distinction between these two approaches can be described as the frequentist approach using the conditional probability pr(A | B) (i.e., the P value), and the Bayesian approach using pr(B | A) (the posterior probability). This paper will further explain the fundamental differences in NHST and Bayesian approaches and demonstrate how they can co-exist harmoniously to guide clinical trial design and inference. «
On the need for caution with P-values and SST
Stang A, Poole C, Kuss O. The ongoing tyranny of statistical significance testing in biomedical research. Eur J Epidemiol. 2010 Apr;25(4):225-30. {ABS-Link} Click for Details » Since its introduction into the biomedical literature, statistical significance testing (abbreviated as SST) caused much debate. The aim of this perspective article is to review frequent fallacies and misuses of SST in the biomedical field and to review a potential way out of the fallacies and misuses associated with SSTs. Two frequentist schools of statistical inference merged to form SST as it is practised nowadays: the Fisher and the Neyman-Pearson school. The P-value is both reported quantitatively and checked against the alpha-level to produce a qualitative dichotomous measure (significant/nonsignificant). However, a P-value mixes the estimated effect size with its estimated precision. Obviously, it is not possible to measure these two things with one single number. For the valid interpretation of SSTs, a variety of presumptions and requirements have to be met. We point here to four of them: study size, correct statistical model, correct causal model, and absence of bias and confounding. It has been stated that the P-value is perhaps the most misunderstood statistical concept in clinical research. As in the social sciences, the tyranny of SST is still highly prevalent in the biomedical literature even after decades of warnings against SST. The ubiquitous misuse and tyranny of SST threatens scientific discoveries and may even impede scientific progress. In the worst case, misuse of significance testing may even harm patients who eventually are incorrectly treated because of improper handling of P-values. For a proper interpretation of study results, both estimated effect size and estimated precision are necessary ingredients «
Greenland S, Senn SJ, Rothman KJ, Carlin JB, Poole C, Goodman SN, Altman DG. Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations. Eur J Epidemiol. 2016; 31: 337–350 {TXT-Link} Click for Details » Misinterpretation and abuse of statistical tests, confidence intervals, and statistical power have been decried for decades, yet remain rampant. A key problem is that there are no interpretations of these concepts that are at once simple, intuitive, correct, and foolproof. Instead, correct use and interpretation of these statistics requires an attention to detail which seems to tax the patience of working scientists. This high cognitive demand has led to an epidemic of shortcut definitions and interpretations that are simply wrong, sometimes disastrously so—and yet these misinterpretations dominate much of the scientific literature. In light of this problem, we provide definitions and a discussion of basic statistics that are more general and critical than typically found in traditional introductory expositions. Our goal is to provide a resource for instructors, researchers, and consumers of statistics whose knowledge of statistical theory and technique may be limited but who wish to avoid and spot misinterpretations. We emphasize how violation of often unstated analysis protocols (such as selecting analyses for presentation based on the P values they produce) can lead to small P values even if the declared test hypothesis is correct, and can lead to large P values even if that hypothesis is incorrect. We then provide an explanatory list of 25 misinterpretations of P values, confidence intervals, and power. We conclude with guidelines for improving statistical interpretation and reporting «
Chavalarias D, Wallach JD, Li AH, Ioannidis JP. Evolution of Reporting P Values in the Biomedical Literature, 1990-2015. JAMA. 2016 Mar 15;315(11):1141-8 {TXT-Link} Click for Details » In this analysis of P values reported in MEDLINE abstracts and in PMC articles from 1990-2015, more MEDLINE abstracts and articles reported P values over time, almost all abstracts and articles with P values reported statistically significant results, and, in a subgroup analysis, few articles included confidence intervals, Bayes factors, or effect sizes. Rather than reporting isolated P values, articles should include effect sizes and uncertainty metrics «
Vidgen B, Yasseri T. P-Values: Misunderstood and Misused. Frontiers Physics 2016; 4: Article 8 {TXT-Link} Click for Details » P-values are widely used in both the social and natural sciences to quantify the statistical significance of observed results. The recent surge of big data research has made the p-value an even more popular tool to test the significance of a study. However, substantial literature has been produced critiquing how p-values are used and understood. In this paper we review this recent critical literature, much of which is routed in the life sciences, and consider its implications for social scientific research. We provide a coherent picture of what the main criticisms are, and draw together and disambiguate common themes. In particular, we explain how the False Discovery Rate is calculated, and how this differs from a p-value. We also make explicit the Bayesian nature of many recent criticisms, a dimension that is often underplayed or ignored. We conclude by identifying practical steps to help remediate some of the concerns identified. We recommend that (i) far lower significance levels are used, such as 0.01 or 0.001, and (ii) p-values are interpreted contextually, and situated within both the findings of the individual study and the broader field of inquiry (through, for example, meta-analyses) « » The p-value quantifies the probability of observing results at least as extreme as the ones observed given that the null hypothesis is true. It is then compared against a pre-determined significance level (α). If the reported p-value is smaller than α the result is considered statistically significant. «
Wasserstein RL, Lazar NA. The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician 2016;70: 129-133 {TXT-Link} Click for Details » Informally, a p-value is the probability under a specified statistical model that a statistical summary of the data (e.g. the sample mean difference between two compared groups) would be equal to or more extreme than its observed value «
Kang J, Hong J, Esie P, Bernstein KT, Aral S. An Illustration of Errors in Using the P Value to Indicate Clinical Significance or Epidemiological Importance of a Study Finding. Sex Transm Dis. 2017 Aug; 44(8): 495–497 {TXT-Link} Click for Details » A simulation study was carried out to illustrate that P values can suggest but not confirm statistical significance; and they may not indicate epidemiological significance (importance). We recommend that researchers consider reporting effect sizes as P values in conjunction with confidence intervals or point estimates with standard errors to indicate precision (uncertainty) «
Amrhein V, Korner-Nievergelt F, Roth T. The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research. Peer J. 2017; 5: e3544 {TXT-Link} Click for Details » The widespread use of ‘statistical significance’ as a license for making a claim of a scientific finding leads to considerable distortion of the scientific process (according to the American Statistical Association). We review why degrading p-values into ‘significant’ and ‘nonsignificant’ contributes to making studies irreproducible, or to making them seem irreproducible. A major problem is that we tend to take small p values at face value, but mistrust results with larger p-values. In either case, p-values tell little about reliability of research, because they are hardly replicable even if an alternative hypothesis is true. Also significance (p≤0.05) is hardly replicable: at a good statistical power of 80%, two studies will be ‘conflicting’, meaning that one is significant and the other is not, in one third of the cases if there is a true effect. A replication can therefore not be interpreted as having failed only because it is nonsignificant. Many apparent replication failures may thus reflect faulty judgment based on significance thresholds rather than a crisis of unreplicable research. Reliable conclusions on replicability and practical importance of a finding can only be drawn using cumulative evidence from multiple independent studies. However, applying significance thresholds makes cumulative knowledge unreliable. One reason is that with anything but ideal statistical power, significant effect sizes will be biased upwards. Interpreting inflated significant results while ignoring nonsignificant results will thus lead to wrong conclusions. But current incentives to hunt for significance lead to selective reporting and to publication bias against nonsignificant findings. Data dredging, p-hacking, and publication bias should be addressed by removing fixed significance thresholds. Consistent with the recommendations of the late Ronald Fisher, p-values should be interpreted as graded measures of the strength of evidence against the null hypothesis. Also larger p-values offer some evidence against the null hypothesis, and they cannot be interpreted as supporting the null hypothesis, falsely concluding that ‘there is no effect’. Information on possible true effect sizes that are compatible with the data must be obtained from the point estimate, e.g., from a sample average, and from the interval estimate, such as a confidence interval. We review how confusion about interpretation of larger p values can be traced back to historical disputes among the founders of modern statistics. We further discuss potential arguments against removing significance thresholds, for example that decision rules should rather be more stringent, that sample sizes could decrease, or that p-values should better be completely abandoned. We conclude that whatever method of statistical inference we use, dichotomous threshold thinking must give way to non-automated informed judgment «
Vetter TR. Descriptive Statistics: Reporting the Answers to the 5 Basic Questions of Who, What, Why, When, Where, and a Sixth, So What? Anesth Analg. 2017 Nov;125(5):1797-1802 {TXT-Link} Click for Details » Descriptive statistics are specific methods basically used to calculate, describe, and summarize collected research data in a logical, meaningful, and efficient way. Descriptive statistics are reported numerically in the manuscript text and/or in its tables, or graphically in its figures. This basic statistical tutorial discusses a series of fundamental concepts about descriptive statistics and their reporting. The mean, median, and mode are 3 measures of the center or central tendency of a set of data. In addition to a measure of its central tendency (mean, median, or mode), another important characteristic of a research data set is its variability or dispersion (ie, spread). In simplest terms, variability is how much the individual recorded scores or observed values differ from one another. The range, standard deviation, and interquartile range are 3 measures of variability or dispersion. The standard deviation is typically reported for a mean, and the interquartile range for a median. Testing for statistical significance, along with calculating the observed treatment effect (or the strength of the association between an exposure and an outcome), and generating a corresponding confidence interval are 3 tools commonly used by researchers (and their collaborating biostatistician or epidemiologist) to validly make inferences and more generalized conclusions from their collected data and descriptive statistics. A number of journals, including Anesthesia & Analgesia, strongly encourage or require the reporting of pertinent confidence intervals. A confidence interval can be calculated for virtually any variable or outcome measure in an experimental, quasi-experimental, or observational research study design. Generally speaking, in a clinical trial, the confidence interval is the range of values within which the true treatment effect in the population likely resides. In an observational study, the confidence interval is the range of values within which the true strength of the association between the exposure and the outcome (eg, the risk ratio or odds ratio) in the population likely resides. There are many possible ways to graphically display or illustrate different types of data. While there is often latitude as to the choice of format, ultimately, the simplest and most comprehensible format is preferred. Common examples include a histogram, bar chart, line chart or line graph, pie chart, scatterplot, and box-and-whisker plot. Valid and reliable descriptive statistics can answer basic yet important questions about a research data set, namely: “Who, What, Why, When, Where, How, How Much?” «
Kennedy-Shaffer L. When the Alpha is the Omega: P-Values, “Substantial Evidence,” and the 0.05 Standard at FDA. Food Drug Law J. 2017; 72(4): 595–635 {TXT-Link} Click for Details » A test provides ranges of that test statistic for which the null hypothesis will be accepted or rejected. Within this hypothesis testing framework, the p-value, a number between 0 and 1, can be defined in several equivalent ways. The formulation most commonly used in the medical literature defines the p-value as the “probability, under the assumption of no effect or no difference, (the null hypothesis), of obtaining a result equal to or more extreme than what was actually observed.” If an event is only of interest if it is more extreme in the same direction as the observed results (compared to the null hypothesis), then we use only that one-sided probability. More commonly, however, a two-sided probability is calculated that is agnostic to whether the more extreme event is in the same or opposite direction as the observed results. A (one-sided or two-sided) p-value is generally then compared to some pre-specified alpha level or significance level. If it is below the alpha level, the null hypothesis is rejected; if it is above the alpha level, the null hypothesis is not rejected. One can equivalently define the p-value, then, as the value of alpha for which the data would be on the border between rejecting and not rejecting the null hypothesis «
Leek J, McShane BB, Gelman A, Colquhoun D, Nuijten MB, Goodman SN. Five ways to fix statistics. Nature 2017;551:557-559 {TXT-Link}
van Rijn MHC, Bech A, Bouyer J, van den Brand JAJG. Statistical significance versus clinical relevance. Nephrol Dial Transplant. 2017 Apr 1;32(suppl-2):ii6-ii12{TXT-Link} Click for Details » In March this year, the American Statistical Association (ASA) posted a statement on the correct use of P-values, in response to a growing concern that the P-value is commonly misused and misinterpreted. We aim to translate these warnings given by the ASA into a language more easily understood by clinicians and researchers without a deep background in statistics. Moreover, we intend to illustrate the limitations of P-values, even when used and interpreted correctly, and bring more attention to the clinical relevance of study findings using two recently reported studies as examples. We argue that P-values are often misinterpreted. A common mistake is saying that P < 0.05 means that the null hypothesis is false, and P ≥0.05 means that the null hypothesis is true. The correct interpretation of a P-value of 0.05 is that if the null hypothesis were indeed true, a similar or more extreme result would occur 5% of the times upon repeating the study in a similar sample. In other words, the P-value informs about the likelihood of the data given the null hypothesis and not the other way around. A possible alternative related to the P-value is the confidence interval (CI). It provides more information on the magnitude of an effect and the imprecision with which that effect was estimated. However, there is no magic bullet to replace P-values and stop erroneous interpretation of scientific results. Scientists and readers alike should make themselves familiar with the correct, nuanced interpretation of statistical tests, P-values and CIs. «
Wellek S. A critical evaluation of the current “p-value controversy”. Biom J. 2017 Sep;59(5):854-872 {ABS-Link} Click for Details » This article has been triggered by the initiative launched in March 2016 by the Board of Directors of the American Statistical Association (ASA) to counteract the current p‐value focus of statistical research practices that allegedly “have contributed to a reproducibility crisis in science.” It is pointed out that in the very wide field of statistics applied to medicine, many of the problems raised in the ASA statement are not as severe as in the areas the authors may have primarily in mind, although several of them are well‐known experts in biostatistics and epidemiology. This is mainly due to the fact that a large proportion of medical research falls under the realm of a well developed body of regulatory rules banning the most frequently occurring misuses of p‐values. Furthermore, it is argued that reducing the statistical hypotheses tests nowadays available to the class of procedures based on p‐values calculated under a traditional one‐point null hypothesis amounts to ignoring important developments having taken place and going on within the statistical sciences. Although hypotheses testing is still an indispensable part of the statistical methodology required in medical and other areas of empirical research, there is a large repertoire of methods based on different paradigms of inference that provide ample options for supplementing and enhancing the methods of data analysis blamed in the ASA statement for causing a crisis «
Gagnier JJ, Morgenstern H. Misconceptions, Misuses, and Misinterpretations of P Values and Significance Testing. J Bone Joint Surg Am. 2017 Sep 20;99(18):1598-1603 {ABS-Link} Click for Details » The interpretation and reporting of p values and significance testing in biomedical research are fraught with misconceptions and inaccuracies. Publications of peer-reviewed research in orthopaedics are not immune to such problems. The American Statistical Association (ASA) recently published an official statement on the use, misuse, and misinterpretation of statistical testing and p values in applied research. The ASA statement discussed 6 principles: (1) “P-values can indicate how incompatible the data are with a specified statistical model.” (2) “P-values do not measure the probability that the studied hypothesis is true, or the probability that the data were produced by random chance alone.” (3) “Scientific conclusions and business or policy decisions should not be based only on whether a p-value passes a specific threshold.” (4) “Proper inference requires full reporting and transparency.” (5) “A p-value, or statistical significance, does not measure the size of an effect or the importance of a result.” (6) “By itself, a p-value does not provide a good measure of evidence regarding a model or hypothesis.” The purpose of this article was to discuss these principles. We make several recommendations for moving forward: (1) Authors should avoid statements such as “statistically significant” or “statistically nonsignificant.” (2) Investigators should report the magnitude of effect of all outcomes together with the appropriate measure of precision or variation. (3) Orthopaedic residents and surgeons must be educated in biostatistics, the ASA principles, and clinical epidemiology. (4) Journal editors and reviewers need to be familiar with and enforce the ASA principles «
Katz JN, Losina E. Uses and Misuses of the P Value in Reporting Results of Orthopaedic Research Studies. J Bone Joint Surg Am. 2017 Sep 20;99(18):1507-1508 {TXT-Link} Click for Details » We provide context for the ASA report and the article by Gagnier and Morgenstern in the form of 2 (fictitious) research abstracts that base their conclusions solely on a p value «
Blume JD, D’Agostino McGowan L, Dupont WD, Greevy, Jr RA. Second-generation p-values: Improved rigor, reproducibility, & transparency in statistical analyses. PLoS One. 2018; 13(3): e0188299 {TXT-Link} Click for Details » Verifying that a statistically significant result is scientifically meaningful is not only good scientific practice, it is a natural way to control the Type I error rate. Here we introduce a novel extension of the p-value—a second-generation p-value (pδ)–that formally accounts for scientific relevance and leverages this natural Type I Error control. The approach relies on a pre-specified interval null hypothesis that represents the collection of effect sizes that are scientifically uninteresting or are practically null. The second-generation p-value is the proportion of data-supported hypotheses that are also null hypotheses. As such, second-generation p-values indicate when the data are compatible with null hypotheses (pδ = 1), or with alternative hypotheses (pδ = 0), or when the data are inconclusive (0<pδ<1). Moreover, second-generation p-values provide a proper scientific adjustment for multiple comparisons and reduce false discovery rates. This is an advance for environments rich in data, where traditional p-value adjustments are needlessly punitive. Second-generation p-values promote transparency, rigor and reproducibility of scientific results by a priori specifying which candidate hypotheses are practically meaningful and by providing a more reliable statistical summary of when the data are compatible with alternative or null hypotheses «
Schober P, Bossers SM, Schwarte LA. Statistical Significance Versus Clinical Importance of Observed Effect Sizes: What Do P Values and Confidence Intervals Really Represent? Anesth Analg. 2018 Mar;126(3):1068-1072 {TXT-Link} Click for Details » Effect size measures are used to quantify treatment effects or associations between variables. Such measures, of which >70 have been described in the literature, include unstandardized and standardized differences in means, risk differences, risk ratios, odds ratios, or correlations. While null hypothesis significance testing is the predominant approach to statistical inference on effect sizes, results of such tests are often misinterpreted, provide no information on the magnitude of the estimate, and tell us nothing about the clinically importance of an effect. Hence, researchers should not merely focus on statistical significance but should also report the observed effect size. However, all samples are to some degree affected by randomness, such that there is a certain uncertainty on how well the observed effect size represents the actual magnitude and direction of the effect in the population. Therefore, point estimates of effect sizes should be accompanied by the entire range of plausible values to quantify this uncertainty. This facilitates assessment of how large or small the observed effect could actually be in the population of interest, and hence how clinically important it could be. This tutorial reviews different effect size measures and describes how confidence intervals can be used to address not only the statistical significance but also the clinical significance of the observed effect or association. Moreover, we discuss what P values actually represent, and how they provide supplemental information about the significant versus nonsignificant dichotomy. This tutorial intentionally focuses on an intuitive explanation of concepts and interpretation of results, rather than on the underlying mathematical theory or concepts «
Gates S, Ealing E. Reporting and interpretation of results from clinical trials that did not claim a treatment difference: survey of four general medical journals. BMJ Open. 2019; 9(9): e024785 {TXT-Link}Click for Details » The majority of trials (54.2%) inappropriately interpreted a result that was not statistically significant as indicating no treatment benefit. Very few studies interpreted the result as indicating a lack of evidence against the null hypothesis of zero difference between the trial arms « « »
Wasserstein RL, Schirm AL, Lazar NA. Moving to a World Beyond “p<0.05”. The American Statistician, 2019; 73(sup1): 1-19 {TXT-Link} Click for Details Don’t’s Do’s
Amrhein V, Greenland S, McShane. Retire statistical significance. Nature 2019;567:305-307 {TXT-Link} Click for Details » Our call to retire statistical significance and to use confidence intervals as compatibility intervals is not a panacea. Although it will eliminate many bad practices, it could well introduce new ones. Thus, monitoring the literature for statistical abuses should be an ongoing priority for the scientific community. But eradicating categorization will help to halt overconfident claims, unwarranted declarations of ‘no difference’ and absurd statements about ‘replication failure’ when the results from the original and replication studies are highly compatible. The misuse of statistical significance has done much harm to the scientific community and those who rely on scientific advice. P values, intervals and other statistical measures all have their place, but it’s time for statistical significance to go «
Dunkler D, Haller M, Oberbauer R, Heinze G. To test or to estimate? P-values versus effect sizes. Transpl Int. 2020 Jan;33(1):50-55 {TXT-Link} Click for Details » The P‐value measures the compatibility of the observed data with the null hypothesis. Technically, it expresses the probability with which, given the null hypothesis was true, data with an effect size as extreme as the observed one or more extreme than the observed one can be obtained. The P‐value cannot separate implausibility of the null hypothesis from implausibility of any of the assumptions: A small P‐value gives evidence that the data are not compatible with the specified model – encompassing the null hypothesis and all assumptions. Hence, the P‐value should be viewed as a continuous measure of compatibility of the data to the model ranging from 0 (complete incompatibility) to 1 (complete compatibility). Consequently, precise P‐values should be presented (e.g., P = 0.07 and not P = NS or P > 0.05). «
Calin-Jageman RJ, Cumming G. The New Statistics for Better Science: Ask How Much, How Uncertain, and What Else is Known. Am Stat. 2019 ; 73(Suppl 1): 271–280 {TXT-Link} Click for Details » The “New Statistics” emphasizes effect sizes, confidence intervals, meta-analysis, and the use of Open Science practices. We present 3 specific ways in which a New Statistics approach can help improve scientific practice: by reducing over-confidence in small samples, by reducing confirmation bias, and by fostering more cautious judgments of consistency. We illustrate these points through consideration of the literature on oxytocin and human trust, a research area that typifies some of the endemic problems that arise with poor statistical practice «
{18.Mai.2020 U: 06.Sep.2023 | ACPS-CdM}