1
$\begingroup$

In a total population of 4,200 patients, the Q1-Q4 groups are divided based on a continuous variable, tyg. During the baseline analysis, the age of the total population is a skewed continuous variable, with a median of 58.0 (25th percentile: 52.0, 75th percentile: 64.0). The median age (with 25th and 75th percentiles) for each group is as follows:

Group 1: 60.0 (55.0, 67.0) Group 2: 58.0 (52.0, 65.0) Group 3: 57.0 (51.0, 63.0) Group 4: 57.0 (50.0, 62.8) When performing a Kruskal-Wallis H test to compare the groups, a p-value less than 0.001 was obtained. However, the reviewer is questioning why the differences in age between the four groups are not large, yet the p-value is less than 0.001.

I have rechecked the data and the statistical method (Kruskal-Wallis H test) and found that the result is still p < 0.001. How should I interpret the reviewer's question and respond? Or is my statistical method incorrect?

Improve this question
$\endgroup$
3
  • $\begingroup$ By "total population" do you mean the number of your observations (normally referred to as sample size)? $\endgroup$ Commented Nov 9, 2024 at 13:57
  • 2
    $\begingroup$ Exact duplicate of stats.stackexchange.com/questions/656984/… ... please don't just repost a question a few hours later $\endgroup$ Commented Nov 9, 2024 at 15:31
  • $\begingroup$ Okay, I will pay attention to it, and I have already removed the duplicates $\endgroup$ Commented Nov 10, 2024 at 9:03

2 Answers 2

Reset to default
4
$\begingroup$

I assume that you have 4200 observations (which we'd normally call "sample size" rather than "total population"). This is a pretty large sample size. Generally, with large sample sizes, tests can easily be significant with fairly small differences between the groups. The bigger your sample size, the smaller the group difference needs to be in order to give you a low p-value, because with large samples you can distinguish even small differences from the null hypothesis of equality. You can observe differences more precisely. so to say.

Note that the p-value is not a measurement of effect size, i.e., low p-value doesn't need to mean that the differences between groups are big, at least not with large samples. Chances are your reviewer doesn't understand this.

One thing you can do is that in your interpretation you say that, despite significant, differences between the groups are rather small. Of course I have no idea about the subject matter background, so I can't tell whether the observed differences are really small, but your reviewer may be right about this. But they need to accept that this is not in contradiction with a low p-value.

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Remember you can accept the answer if it helped you answer your query. $\endgroup$ Commented Nov 9, 2024 at 15:31
4
$\begingroup$

The main reason why the reviewer is surprised is because the Kruskal-Wallis test (K-W) is not a test of the medians (just like, btw, the Mann-Whitney U test is not a test of medians). So yes, the medians could be exactly identical, and you could have a very low p-value. This post, or this one present exactly such real world cases.
K-W is a test of stochastic superiority (or, equivalently, of stochastic equivalence). Rejecting the null implies that at least 1 population is stochastically superior to at least another 1 (E.g. from Wikipedia, "A significant Kruskal–Wallis test indicates that at least one sample stochastically dominates one other sample".)
It is sad that your reviewer does not know this, and sadder than many textbooks, university blogs (e.g. here), even the R documentation of K-W, etc. say that it is. It is a test of the median iff the 3 or more samples come from exactly the same distribution, except for a shift, which is untestable, and practically speaking implausible. And btw, under this very restrictive condition, it is also a test of the mean, of the quartiles, of any percentile you care to chose.
And then, when you report your significant result, please do not report it as a difference of medians...

Improve this answer
$\endgroup$

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.