26  Lab: Evaluate model performance

26.1 Required software

It is assumed you already installed the packages from computer lab in chapter 3.

26.2 Model assessment SOC prediction for Ponui island

In the computer lab on spatial machine learning – regression you computed predictions for topsoil SOC content for Ponui island. We did not yet fully inspect the model performance.

Download the CSV file with the prediction for the validation samples
CSV download.

TipTask

Create a validation scatterplot (predicted on the x-axis, observed on the y-axis) with a lowess scatterplot smoother.

CautionQuestion 26-1

Given this plot, which statements are correct? (select all that apply)

  1. The points are located nearly on the 1:1-line, this indicates very good model performance.

  2. The lowess scatterplot smoothing line follows more or less the 1:1-line. This indicates that there is no conditional bias (like for example systematic over-estimation, but only for small values).

  3. There are five very strong outliers. This is suspicious, we should check if there was a data preparation error or if the model could be improved for this prediction situation.

  4. There is a correlation visible between observed and predicted. But, some of the points are located quite faraway from the 1:1-line. This indicates only medium model performance.

TipTask 1

Compute validation metrics to characterize the model performance. Chose metrics as you see fit.

CautionQuestion 26-2

Which statements do you support? (select all that apply)

  1. The model systematically underestimates SOC content. But, given an observed value range of 2.7 to 8.2 %, this bias is only small.

  2. The percentage of correct predictions is about 42 %. This is a medium performance as expected from the scatterplot.

  3. A good metric is the kappa statistic, which is around 0.75 and thus larger than expected from the scatterplot.

  4. It can be confirmed that model performance is only medium. Only about 42 % of the variance could be explained by the model.

  5. We should not compute any validation metrics. The validation locations were not chosen well and any of those statistics will be biased and not give us any information on the actual map accuracy.