Questions tagged [calibration]
Calibration can refer to adjustment of measurements to agree with value of some standard; to transform classifier scores into class membership probabilities; etc. Do not use for predicting an explanatory variable from an observation of the dependent variable, for that use the tag inverse-prediction.
343 questions
0
votes
0
answers
35
views
Tolerance inference of instrument B from instrument A
I have two instruments A and B and I have a known tolerance on A. The instruments use different underlying processeses to measure the same quantity. I measure a quantity simultaneously with both ...
- 1
1
vote
0
answers
13
views
Stereo calibration fail for no apparent reason
I am working on a stereo calibration of 2 thermal cameras, mounted on a 4m high apparatus and are about 4m apart. Bottom line: Fail to achieve good calibration. I get baseline lengths ~6m and high ...
5
votes
1
answer
120
views
How can I best compare two Bayesian posteriors from simulated datasets with common true parameter values?
I would like to evaluate how well two experimental designs perform with the goal of parameter estimation.
To do this, I'm generating 1000 sets of the 3 model parameters ('true values'), simulating ...
- 22.5k
2
votes
1
answer
46
views
Tune a Model on Calibrated or Uncalibrated Probabilties?
I report model performance using log loss on calibrated probabilities, where calibration is temperature scaling fitted on train-only out-of-fold (OOF) predictions.
For hyperparameter tuning, should ...
5
votes
2
answers
107
views
Paradoxical effect of right censoring on D-calibration in survival analysis
I am examining the effect of (right-)censoring on the D-calibration of survival data sets. The data sets are completely synthetic, generated by R package coxed. I use my own code (not the package's ...
3
votes
0
answers
108
views
Running the Breusch-Pagan test manually in R assuming a weighted linear regression
I am trying to run the Breusch-Pagan test manually in RStudio from a weighted linear model (wi = 1/x^2). I need help verifying whether the following rationale is correct:
What I did:
WLS and residuals
...
- 31
11
votes
3
answers
796
views
How can calibration plots for my model's predictions look good while the standard metrics (ROC AUC, F-score, etc.) look poor?
Background
I trained an XGBoost model to predict a dichotomous outcome, which has a base rate of about 55% in the overall sample. This model will not be used to classify, however: It will be used ...
- 11.7k
0
votes
0
answers
30
views
Calibration in Cox regression
I am running a Cox regression model to examine the predictive performance of a model using one variable (a risk assessment score) to predict days to crime. As part of the modeling, we are examining ...
1
vote
0
answers
104
views
Raking survey weights of specific subgroups
The Question: what are the best practices for raking the survey weights of a specific subsample while leaving the remaining sample (relatively) unchanged for later comparative analyses?
The Situation: ...
- 11
2
votes
0
answers
115
views
Logistic model giving mixed results
I am currently creating two logistic regression models (one with forward selection and one with LASSO) using R to predict whether a patient has a malignant or benign breast cancer from this dataset: ...
- 43
1
vote
1
answer
106
views
A correct approach to validate/correct readings from similar sensors?
I am looking to apply a calibration/correction approach on a set of sensors and I just wanted to know that the approach I am going to use is statistically correct and acceptable.
I am using a set of ...
- 157
1
vote
0
answers
88
views
Calibration with all data: data-poor scenarios [closed]
I’m working on species distribution modeling with binary data (presence / absence, 1 / 0). My target species is extremely rare (prevalence ~0.014), so my dataset is almost all zeros and just a handful ...
- 19
2
votes
1
answer
129
views
Calibration of score derived from model
I am looking to externally validate the calibration of a model someone else developed using a different dataset.
The original model was developed by using linear regression, and then the weights were ...
0
votes
0
answers
98
views
Interpreting Multicollinear Models with SHAP: Challenges with XGBoost and Isotonic Regression
I am familiar with SHAP and often use it when developing or assessing ML models. I want to use SHAP in a new context. I'm working on a project that relies on an XGBoost Classifier, which outputs ...
- 1
1
vote
1
answer
62
views
How to obtain the same AUC using isotonic regression in R?
I am trying to calibrate the predicted probabilities using isotonic regression for binary outcome model in R. I know that calibrating probabilities should not change the AUC. But the following R ...
- 163