2
$\begingroup$

I've been using an interpretable machine learning package for binary decision trees known as IAI.

Long story short: the core method here is known as an 'optimal classifier' (it does not use greedy heuristics such as random forest or XGBoost. But instead evaluates all trees in combination to obtain global optimisation).

So, given the opposing merits of these three models, I would like to compare AUCs on one graph. Suppose then that I had three stored plots:

Optimal classifier:

x <- iai::roc_curve(grid, test_X, test_y, positive_label = 1)

Random forest:

y <- iai::roc_curve(grid, test_X, test_y, positive_label = 1)

XGBoost

z <- iai::roc_curve(grid, test_X, test_y, positive_label = 1)

Is it possible to combine these in one plot? I've tried pROC and "add true" arguments. But I haven't had any luck.

I've attached the source of my code, in case that is helpful. Would truly appreciate some help.

Optimal Classifier AUC:

https://docs.interpretable.ai/stable/IAI-R/quickstart/ot_classification/

Greedy Methods AUC

https://docs.interpretable.ai/stable/IAI-R/quickstart/heur_classification/

Improve this question
$\endgroup$
6
  • $\begingroup$ You can extract the data from the pROC code and plot them yourself with a plot call and a couple of lines calls, just like how you'd plot any other three curves. An alternative is to extract the data, create a data frame containing said data, and use ggplot2. $\endgroup$ Commented Sep 23, 2021 at 16:07
  • $\begingroup$ Hi Dave, thanks for the feedback. When you say extract the data, do you mean setting each of the ROCs equal to something (say x, y, z, like above), and then creating a data frame? $\endgroup$ Commented Sep 23, 2021 at 16:15
  • $\begingroup$ I don't remember the exact syntax, but the pROC package has a function like roc or auc that determines the points that make it to the plot. Take those points and plot them yourself, as you would plot any other two variables. $\endgroup$ Commented Sep 23, 2021 at 16:18
  • $\begingroup$ Thanks Dave. I tried pROC, but I'll keep trying. Thanks for your feedback $\endgroup$ Commented Sep 23, 2021 at 16:26
  • $\begingroup$ When you are saying one plot, are you implying that they should be plotted on the same axis? Or the three should be side by side in one plot? $\endgroup$ Commented Sep 23, 2021 at 17:38

1 Answer 1

Reset to default
1
$\begingroup$

The roc function in the pROC package allows you to extract the sensitivity and specificity values. I will give an example below. Keep in mind that the $y$-axis is sensitivity, but the $x$-axis is $1 - specificity$.

library(pROC)
set.seed(2021)
N <- 1000
x1 <- rnorm(N)
x2 <- rnorm(N)
x3 <- rnorm(N)
z <- x1 + x2 + x3
pr <- 1/(1 + exp(-z))
y <- rbinom(N, 1, pr)
L1 <- glm(y ~ x1, family = binomial)
L2 <- glm(y ~ x1 + x2, family = binomial)
L3 <- glm(y ~ x1 + x2 + x3, family = binomial)
pred1 <- 1/(1 + exp(-predict(L1)))
pred2 <- 1/(1 + exp(-predict(L2)))
pred3 <- 1/(1 + exp(-predict(L3)))
roc1 <- pROC::roc(y, pred1)
roc2 <- pROC::roc(y, pred2)
roc3 <- pROC::roc(y, pred3)
plot(1 - roc1$specificities, roc1$sensitivities, col = 'black')
points(1 - roc2$specificities, roc2$sensitivities, col = 'red')
points(1 - roc3$specificities, roc3$sensitivities, col = 'blu

Keep in mind that statisticians do not necessarily like ROC curves.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Hi Dave, thanks again for your time. The protocol I am working in already provides a way in which the AUC is extracted with 'true positive' and 'false positive' on the respective axis. I guess I am just struggling to combine those three AUCs that I have. $\endgroup$ Commented Sep 23, 2021 at 16:46
  • $\begingroup$ If all you have is the plot and no ability to extract the sensitivity and specificity values, I think you're stuck unless you want to go through and examine by hand/eyeball what the coordinates are of the points on your three ROC curves. However, I would be flabbergasted if your software didn't give the points needed to construct the ROC curves the way that pROC::roc does. // If you have the probability values predicted by your model, you can construct the ROC curves yourself, either in pROC or by hand, picking a sequence of thresholds like seq(0, 1, 0.001). $\endgroup$ Commented Sep 23, 2021 at 20:34
  • $\begingroup$ If I have the curves, is it possible just to combine them in ggplot2? Is that a thing? $\endgroup$ Commented Sep 23, 2021 at 20:56
  • 1
    $\begingroup$ Do you mean that you have graphs that you can see or that you have the points that make up those curves (vectors, data frames, lists...something like that)? I doubt ggplot2 can combine multiple graphs made in a different package if you lack the raw data. // Why not use software like pROC that can do what you want to do? $\endgroup$ Commented Sep 23, 2021 at 20:57
  • $\begingroup$ Yeah, pROC seems like the optimal plan. But tbh, I just have three sets of commands which print out an AUC curve for me, for three models. It seems unecessarily difficult to combine these in pROC somehow with the code I have unfortunately. Yet, I know it can be done, because the authors routinely evaluate their models against others with multiple AUC plots $\endgroup$ Commented Sep 23, 2021 at 21:01

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.