6
$\begingroup$

I am currently writing my bachelor’s thesis and have only a limited background in statistics.

I have four groups with four observations each, except for one group which contains only three observations. I am aware that this is a very small sample size for conducting a two-way ANOVA with two main effects and their interaction; however, this approach was suggested by my supervisor.

I therefore have three questions:

I initially fitted a Type I ANOVA (Type I sums of squares) including two main effects and their interaction. After realizing that one group contains a missing observation, I found that the interaction term was not significant. Because the design is unbalanced, I subsequently fitted a Type II ANOVA model with the same two main effects and their interaction. Is this procedure appropriate?

Due to a significant Shapiro–Wilk test (I am aware that this test has limitations for small sample sizes, but it was required by my supervisor), I applied a squared transformation to the dependent variable.

My supervisor suggested reporting an effect size. Which effect size measure would be appropriate in this case, and do I need to back-transform the variable before calculating it?

The current table in my bachelor thesis looks like this:

enter image description here

Freiheitsgrade = df Quadratsumme = SS

Am I missing something?

$\endgroup$

1 Answer 1

7
$\begingroup$

From the linked table, it looks like your model isn't "significantly different" from a null model. As you are new to statistics and your questions would be important with "significant" results, here are some answers.

Because the design is unbalanced, I subsequently fitted a Type II ANOVA model with the same two main effects and their interaction. Is this procedure appropriate?

With an unbalanced design, results with Type I ANOVA depend on the order in which the predictors are entered into the calculation. See this page and its links. I would argue that it's generally inappropriate to use Type I ANOVA with unbalanced data. Your use of Type II ANOVA with your unbalanced data is certainly appropriate.

One caution: don't let p-values/significance of prior calculations determine later choices of tests. In your situation, your choice of Type II ANOVA could be made without having done the prior Type I ANOVA, based on the unbalanced design.

Due to a significant Shapiro–Wilk test (I am aware that this test has limitations for small sample sizes, but it was required by my supervisor), I applied a squared transformation to the dependent variable.

There's nothing wrong in principle with transforming dependent variables. If you have reason to believe that the predictors are linearly associated with the square (or the log, or the square-root, or...) of the dependent variable, that's fine. Just recognize that you are then modeling the mean of the square (or of the log, or of the square-root, or ...) instead of the mean of the variable itself.

As you learn more about statistics, look into other types of models that might be more directly related to the questions you are asking. For example, generalized linear models can be more useful than simple transformation of dependent variables. Ordinal regression doesn't make assumptions about distributions of outcomes or error terms, and with modern computational tools it can now be applied in practice in many situations that would previously have been analyzed with ANOVA or multiple linear regression.

My supervisor suggested reporting an effect size. Which effect size measure would be appropriate in this case, and do I need to back-transform the variable before calculating it?

The proportion of variance explained by each factor is the basis for frequently used measures of "effect size" in ANOVA. This page explains how to use tools in the R effectsize package to perform several types of such calculations. You have to be careful here with an unbalanced design; make sure to calculate in a way that uses your Type II ANOVA result, as explained on that page.

As you are modeling the square of the dependent variable, the "proportion of variance explained" from your Type II ANOVA will be with respect to the variance among the squared values. That's OK to report if you are clear about that. It just might not be as intuitively easy to understand as if you had performed ANOVA on the original outcome scale.

As you learn more about statistics, be very careful with the term "effect size." It has many different meanings. When you see someone use that term (or you use it yourself), make sure that you know just which "effect size" is in question.

$\endgroup$
2
  • $\begingroup$ thank you. But why not type III ANOVA since I have an interaction term included in my model? I hear all the time "use type II if there is no inteaction term". I actually just did the transforamtion of the DV because of the significant shapiro-Wilk test. And is it useful to calculate effect sizes with not-significant ANOVA results? $\endgroup$ Commented Jan 10 at 8:30
  • $\begingroup$ @Faith my head hurts when I think about Type III ANOVA. Arguments about Type II versus Type III are covered in many pages on this site. An advantage of reporting effect sizes for a "not-significant" ANOVA is that others than can include your effect sizes in meta-analyses combining multiple studies, while not suffering from the publication bias that comes from only evaluating "significant" studies. Whether "variance explained" effect sizes are most useful for meta-analysis is another question. $\endgroup$ Commented Jan 10 at 13:56

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.