3
$\begingroup$

Part of my job is measuring the effect of marketing interventions using experiments when possible, or estimating their effect when it's not possible to experiment.

I know relatively little about causal inference, beyond the fact that it's a very dense area of applied statistics, and I typically use Google's Causal Impact package to infer causal effect sizes from observational data.

However, one thing that I do know is that all methods of causal inference - even seemingly simple aggregate-level Diff-in-Diff approaches - require adherence to strong assumptions in order to draw valid results.

A (much more senior) colleague has recently proposed a method of estimating the effect of an upcoming intervention using a simple forecast-based approach. Essentially, they intend to forecast the behaviour of a response time series from the point of intervention using the historical (pre-intervention) behaviour of that time series and one or two covariates corresponding to marketing-promotions and holidays. They will then infer the effect by looking at the difference between observed data and the predicted counterfactual (what the response would have done if we'd not made an intervention).

This is all very familiar to anyone who has worked with causal impact or other extensions of DiD, however the part that is troubling me is the specification of the model. In the reading around the subject that I've done, I've never seen such a naively simple approach; synthetic controls/counterfactuals are always carefully constructed using contemporaneous covariates that seem to impart causal estimation power to the model.

Quite frankly, if this is valid, then why bother going to the extent of other causal inference approaches? Why not just build a brilliantly accurate predictive model and forecast a counterfactual without concern? Alternatively, if there are key considerations being overlooked here then can someone please help with identifying them as I feel there is something amiss but I lack the knowledge to know what.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Such attitude has no scientific value, because none of its properties can be proven. Prediction can be anything, it may, or it may not correspond with the real values in the future. Even if you can show quality of the prediction on the training data, you can not prove its properties after the intervention, unless you have reliable knowledge from outside the data (which will be strong assumption, as you described in your description). This is the reason why you will not find such approach in causal inference papers.

However, little companies seem to expect real causal inference. No one can tell if the result of this "predicted causal effect" will be close or not to actual causal effect of intervention. It depends on the quality of the prediction and hope, that no other intervention happened at the same time (structural parameters did not change). It may be ok, and company may be on quite accurate track, or not.

Also, this approach seems to be somehow connected to ITS (Interrupted Time Series Designs). There is a lot of extensions of the simple ITS model, and maybe some of them are accurate to your setting. Take a look at McDowall et al. (2019).

McDowall, David, Richard McCleary, and Bradley J. Bartos. Interrupted time series analysis. Oxford University Press, 2019.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for your response, and for pointing me in the direction of ITS models. I think the proposed quasi-experiment design is not valid because - as you say - the model says nothing about the effect of the intervention. The standard form of the most basic ITS estimator can be written as Yt = β0 + ��1T + β2Xt + β3TXt where the effect of the policy change is described by β2 and β3. Whereas the proposed forecast model does not incorporate any terms for the intervention directly so to infer anything from it is unfounded. Does this seem like a correct interpretation? $\endgroup$ Commented Jan 28, 2021 at 10:12
  • $\begingroup$ Models in general do not have to incorporate anything directly to be valid. Take the most basic experimental design as an example. In the proposed approach, if prediction was 100% accurate, the causal reasoning could be also 100% accurate. However there is no possibility to prove properties of any prediction for the future, mostly because structural parameters change. And there starts the part, where assumptions and beliefs are needed. There are some ways to make them more convincing, but all TS-based causal reasonings base on very strong assumption, that structural parameters did not change. $\endgroup$ Commented Jan 29, 2021 at 10:13

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.