3
$\begingroup$

I am running n event study like DID analysis where my dependent variable is expressed in logs. The variable is the aggregate amount of sales from a given firm to a specific type of partner over a year, and I am interested in studying what happens once a firm is exposed to a specific type of shock. Given that the dependent variable is heavily skewed, I log transformed it, given that (i) sales are approximately log-normally distributed; (ii) it is a common procedure in the econ literature.

Since my findings (which are robust in log terms) point to a large drop of sales to this specific type of partner, I have been asked to see what happens leaving the variable in levels. However, the results do not match. For instance, where I had a clear and significant drop in sales following the shock when I use the log, I have no significant impact when the variable is expressed in levels.

Besides the usual absolute vs percentage change, how can I explain this (hopefully only apparently) incoherent result? This post suggests that the two specifications reflect the different weights assigned to large or small observations, but even removing outliers does not solve the issue. I am also including a few control variable to rely on the matching between treated and control units, but their inclusion or removal does not affect the results remarkably (with the log specification still pointing to a significant effect and the level specification pointing to no effect at all.)

For reference, I am conducting my analysis in STATA using Callaway and Sant'anna (2021) Doubly robust DID estimator (csdid)

Improve this question
New contributor
Filippo Santi is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
2
  • 2
    $\begingroup$ Could you please explain what "leaving the variable in levels" means? $\endgroup$ Commented yesterday
  • 1
    $\begingroup$ Apologize if I was not precise. I meant the non-transformed variable $\endgroup$ Commented 22 hours ago

3 Answers 3

Reset to default
4
$\begingroup$

I really like this question, because it provides a great example of the ills and perils of data transformation.

The OP says that he transformed the data because the DV “is heavily skewed...(approximately log-normal)” and “it is a common procedure in the econ literature”. These are about some of the poorest reasons to transform data. Let’s take then 1 by 1:
The data is heavily skewed? Ok, so where is the problem; that is the reality of the data. Why torture the data to erase this reality? Let’s instead deal with it. The knee-jerk reaction of “oh, my data does not look good, so let’s change it” is simply poor practice.
Everyone else does it? I will simply quote rear admiral Grace Hopper “The most dangerous phrase in the language is: We’ve always done it this way.”, and leave it at that.

The only valid reason to transform a DV is because you are in fact not interested in the untransformed DV (in other words, you were going to transform it even before seeing what the data looked like, and regardless of what others do, or not). A good example of this scenario is the Beer-Lambert law, where we measure the intensity of a radiation (light, sound, etc.), but are really interested in the distance it travelled; we measure intensity because it is practical, pragmatic, and then transform it to get to the distance (because the distance itself is not directly measurable). Or measuring the performance of an amplifier in dB (where we care about relative amplification).

So, if the OP really is interested in the rate of change in sales, then the DV should have been transformed (regardless of skewness, or “tradition”). But since he is now asked to look at the absolute change, it does not seem that the rate of change was really of interest, or maybe his audience is unclear about what they are interested in...

Now, testing on the log transformed sales will allow to test the geometric mean (against 1), but testing on the actual sales will allow you to test the arithmetic mean (against 0). So there is nothing inconsistent about the different results; you are testing 2 different null hypotheses, on 2 different data sets, so indeed you could get 2 different results...

Why could this be? Hard to say for sure w/o seeing the data, but a likely reason would be the change in standard deviation. With the skewed original data, the sd is likely quite large; by log transforming the data, you have greatly compacted your data range (e.g. instead of ranging from 1 to 100, it now ranges from 0 to 2), considerably reducing the value of the sd. So it should be no surprise that a test on the log transformed data could be significant, while one on the untransformed data would not be. Just examine the values of the sd...

Morale of the story? Decide what is of practical importance for you (and your audience). The net average change in sales (difference in arithmetic means)? Then do not transform anything. Or the mean relative change in sales (difference in geometric means)? Then log transform away, and tell your audience not to worry about what the arithmetic means say.

Or maybe both? Then a) ask your audience to make up their mind b) handle the mixed result you got (e.g., how non-significant was the result on untransformed sales? $p=.06$, or $p=.5$?. Did you use a single sided or double sided test?. Etc...)

Last, a caveat about relative changes. As xkcd shows us, a large relative change on a small quantity is still a small quantity. So examine your (untransformed) data to see the magnitudes of your before and after sales; are there a lot of rather small values, or are they all very substantial numbers? That is, is your data mostly like the one xkcd deals with (in which case the geometric mean is distorting the reality; large number of large relative changes but on values which have small business importance), or not (large relative changes on values which are of business importance).

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think both points are important, though we were most interested in the transformed version: how the shock affects a firm performance, and this performance, in our idea, related to its growth. The point is that our audience found an increase up to 200% (log of 2) to be excessive, but I guess it makes sense if we show that small firms are the ones driving the result when taking the log, while are large firms on the right tail to drive the result when we take the variable without transforming. Thanks for the lecture :-) $\endgroup$ Commented 21 hours ago
4
$\begingroup$

There are already a couple of good answers, but sometimes a concrete example helps as well.

Consider the case where the "control" group changes from a mean of 50 to a mean of 40 (difference of 10, reduction by 20%). The "treatment" group changes from a mean of 40 to a mean of 30 (difference of 10, reduction by 25%). In this case looking at the difference of (untransformed) means should result in "not significant" because both groups changed by the exact same amount, should show "significant" (given enough data) on the log scale because the percent reduction is different.

Going the other way, if a 2nd "treatment" group goes from a mean of 40 to 32 (difference of 8, reduction by 20%), then the difference (untransformed) should be "significant", while the logged difference should be "not significant" when compared to the "control" group above.

The moral of the story is to not be surprised when the results for different null hypotheses are different.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for the reply, and for the clarifying example. I think it makes perfect sense what you are saying, but from a purely speculative POV, finding incoherent results is never pleasant when someone has to defend a the analysis. Our point is indeed on the rate of growth of the exposed firms, but the large coefficient from the transformed DV specification "spooked" a few commentators, who asked about what happens when we do not transform the variable...it is hard to justify we were not p-hacking our way when results are so different and the reader point to another RQ! $\endgroup$ Commented 21 hours ago
3
$\begingroup$

First, simply saying "significant" vs. "not significant" is not very meaningful. See Andrew Gelman's article "The difference between 'significant' and 'not significant' is not, itself, significant".

Second, even if you look at effect sizes (appropriately modified), why would you expect them to stay the same? Indeed, if they were going to stay the same, there would be no point to the transformation.

Third your question:

Besides the usual absolute vs percentage change, how can I explain this (hopefully only apparently) incoherent result?

That's the difference.There's nothing incoherent here. Change in number vs. change in percent are fundamentally different things. This is especially true when the DV is skewed - and skewness is related to, but different from, having outliers.

Fourth, you should choose the DV based on substantive reasons. Here, logged values seem to make sense. "Did sales double?" is a good question. "Did sales go up by 400 units?" is like to be less so, if the numbers vary by a lot (which skewness says is what happens).

Finally, if you write this up, it might make a good case for explaining why all the above are true.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks @PeterFlom for your comment. It's not that I was expecting the results to stay the same, but at least to be coherent. And they do not (if you read my comments to the previous answers, you can see why I raised the question). As for significant vs not significant, I agree I might have not being super precise and that significance per se says little given the two different question a transformation allow to answer to, but I also think that the difference can sometimes be significant - when it is the ground on which some results are accepted or not (for what that is worth, anyway!) $\endgroup$ Commented 21 hours ago

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.