Optimization and Complexity: Assessing Model Accuracy [Systems thinking & modelling series]

This is part 58 of a series of articles featuring the book Beyond Connecting the Dots, Modeling for Meaningful Results.

We first import our historical data into a converter primitive. We then assess the qualitative and quantitative accuracy of the model. To assess how well our model fits the historical data qualitatively, we plot the simulated and historical data series next to each other. Ideally, they will match closely but if they do not we should pay close attention to how they differ.

If they have the same general shape (except for a vertical or horizontal displacement) that is good news, as it indicates that most likely the general dynamics of your model are correct and you may just need to fine-tune the relationships and parameter values. If the results look considerably different you may have more work to do in improving the model.

You can also assess the accuracy of models quantitatively. A standard tool used to assess the accuracy of a model is the R² metric¹. R² is the fraction of the squared error explained by the model compared to the “null” model. It ranges from 0 (the model basically provides no predictive power) to 1 (the model predicts perfectly). Mathematically, R² is calculated like so:

Naively used, R² has a number of issues that we will discuss later in this section. However, it is still a useful tool that many people use and with which they are familiar. It is also relatively straightforward to calculate. The following code calculates an R² for a model fit. This is code written in JavaScript and can be placed as the Action for a button primitive in Insight Maker². The code assumes two primitives: a converter Historical Hamsters containing historical population sizes and a stock Hamsters containing simulated population sizes. You can edit the code to reference the actual names of the primitives in your model.

var simulated = findName(“Hamsters“); // Replace with your primitive name

var historical = findName(“Historical Hamsters“); // Replace with your primitive name

var results = runModel({silent: true});

var sum = 0;
for(var t = 0; t < results.periods; t++){
sum += results.value(historical)[t];
}

var average = sum/results.periods;

var nullError = 0;
var simulatedError = 0;
for(var t = 0; t < results.periods; t++){
nullError += Math.pow(results.value(historical)[t] – average, 2);
simulatedError += Math.pow(results.value(historical)[t] – results.value(simulated)[t], 2);
}

showMessage(“Pseudo R^2: “+((nullError-simulatedError)/nullError));

Next edition: Optimization and Complexity: Calibrating the Model.

Article sources: Beyond Connecting the Dots, Insight Maker. Reproduced by permission.

Notes: