0
$\begingroup$

How do we evaluate the performance of a regression model with a certain RMSE given that a domain knowledge performance metric is not present?

Maybe MAPE is one way of comparing the performance of my model on my dataset with another model on another but a similar dataset.

Is there any way to evaluate the performance of my trained model given summary statistics and RMSE values?

To illustrate, here are the summary statistics of the target value:

Mean = 3.33
Median = 2
STD = 3.24
Max = 92 (no data error)
RMSE:1.97

My RMSE value is significantly lower than the standard deviation. should this indicate that my model is "good enough" ? I mean how can I be sure that my model does not underfit the data ?

Improve this question
$\endgroup$
5
  • 1
    $\begingroup$ Can you compare with a couple different models? That can be a benchmark, $\endgroup$ Commented Mar 30, 2023 at 11:51
  • $\begingroup$ yes. but let's say I compared it with other models and I got a lower score such as 1.8. It is again me who does the modelling. I can not be the benchmark. can I? :) I'd suggest maybe a better data scientist would get a better score with better handling of the hyperparameters and feature engineering. $\endgroup$ Commented Mar 30, 2023 at 11:54
  • $\begingroup$ If not you then who? Do you have other people that are willing to model this? $\endgroup$ Commented Mar 30, 2023 at 11:55
  • $\begingroup$ I don't. that is why I am looking for a mathematical way of proving that my model is good enough. $\endgroup$ Commented Mar 30, 2023 at 11:57
  • $\begingroup$ No such thing, mathematics won't help you here. $\endgroup$ Commented Mar 30, 2023 at 11:59

1 Answer 1

Reset to default
0
$\begingroup$

Under parameterize the regression model by starting with a model that is just an intercept. Then perform forward selection, add coefficients one at a time to find a model that is not under-fit.

Model comparison is the best way to evaluate a model. Does one model perform better than another model?

Improve this answer
$\endgroup$
2
  • $\begingroup$ That's almost surely going to overfit, so not a very useful metric. $\endgroup$ Commented Mar 30, 2023 at 13:34
  • $\begingroup$ You are correct. I misread the question. I have revised my answer. $\endgroup$ Commented Mar 30, 2023 at 13:48

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.