Explore: Profile

Suggest edits
Documentation > Explore

Contents

Profile parameters ๐Ÿ”—

The profile (or calibration profile) method is designed to test the sensitivity of the input parameters in a calibration context. Calibration profile algorithm differs from traditional sensitivity analysis: it captures the full effect of a parameter variation on the model fitness, every other input being calibrated to optimize the calibration criterion.

In the following we will use the term of fitness to denote the calibration error function, or calibration criterion. Several types of evaluation functions can be used as fitness. Most of the time, they are either some kind of distance between model dynamics and data, or statistical measures performed on the output dynamics.

Method scores ๐Ÿ”—

image/svg+xml Output Exploration Input Exploration Sensitivity Optimisation

The calibration profile method is suited to reveal a model's sensitivity regarding its parameter, hence the highest score possible in sensitivity. However, it does not retrieve information about the input space nor the output space structures, as it focus on one parameter/input, every other input being let free (to be calibrated). As the studied parameter varies, the other parameter are calibrated, so this method scores very well regarding calibration. For the same reason, it can handle stochasticity. Finally, the profile method realizes calibrations on the other inputs for each interval of the input under study, so the more inputs, the more sensitivity to dimensionality of input space.

"image/svg+xml" For each i 's value, j and k are calibratedto minimize the objective i k j i Model

Run

Given a fitness function, the profile of a selected parameter i is constructed by dividing its interval into subintervals of equal size. For each subinterval, i is fixed, and the model is calibrated to minimise the fitness, similarly to Calibration. The optimisation is performed over the other parameters of the model.

As an example, let's consider a model with 3 parameters i, j and k, each taking real values between 1 and 10. The profile of the parameter i is made by splitting the [1,10] interval into 9 intervals of size 1. Then, Calibration is performed in parallel within each subinterval of i. At the end of the calibration, we obtain sets of i,j and k values minimising the fitness, with i taking values in each subinterval. By plotting the fitness against the values of i , one can visually determine if the model, within each of i subintervals, is able to produce acceptable dynamics. An important nuance is that for each point of this fitness vs. i values plot, fitness values has been obtained by "adjusting" j and k to counterbalance the effect of i's values on the model dynamics. This means that j and k values vary for each point of the i parameter profile plot.

Typed signature ๐Ÿ”—

The profile method can be typed likewise:

Profile : ((๐“ง1,...,๐“งk,) โ†’ {1,...,k} โ†’ [(๐“ง,Y)]
such that : Profile(M)(i) = [(๐’™i1,y1), ... , (๐’™in, yn)],
โˆ€ j โˆˆ [1;n], yj =min๐’™โˆˆX-i(M(๐’™))
X-i={๐’™โˆˆ๐“ง1 x ... x ๐“งi-1 x {๐’™i} x ๐“งi+1 x ... x ๐“งk}

With M, the model, X , the Input space, Y, the output space, ๐“Ÿ(X) the power set of X ( i.e. every subset of X , including X and โˆ… )
In other words : this function takes a model M ( whose signature is (Xโ†’Y) ) , an element y of Y (y is the list of criterion value to reach) and find a list of elements of X (noted x) such that, M(x) are Pareto dominant compared to every image of other elements of X by M , regarding criterion y)

Profile within OpenMOLE ๐Ÿ”—

The ProfileEvolution method takes the following arguments:
  • evaluation: the model task, that has to be previously declared in your script.
  • genome: a list of the model input parameters with their values interval declared with the in operator
  • objective: the fitness, an output variable defined in the OpenMOLE script that is used to evaluate the model dynamics, to be minimized
  • x: the parameter to profile
  • nX: the size of the subintervals in x domain
  • stochastic: (optional) the seed provider, mandatory if your model contains randomness
  • parallelism: (optional) number of parallel calibration performed within i subinterval, default is 1
  • termination: a "quantity" of model evaluation allocated to the profile task. Can be given in number of evaluations (e.g. 20000) or in computation time units (e.g. 230 minutes, 2 days, ...)
  • distribution: (optional): computation distribution strategy, default is "steadyState"


To build a profile task, use the ProfileEvolution constructor. Here is an example :
val param1 = Val[Double]
val param2 = Val[Double]
val fitness = Val[Double]

ProfileEvolution(
  evaluation = modelTask,
  parallelism = 10,
  termination = 100,
  x = param1,
  nX = 20,
  genome = Seq(
      param1 in (0.0, 99.0),
      param2 in (0.0, 99.0)
      ),
  objective = fitness,
  stochastic = Stochastic(seed = seed, replications = 100)
  parallelism = 500,
  termination = 200000,
  distribution = Island(10 minutes)


Interpretation guide ๐Ÿ”—

A calibration profile is a 2D plot with the value of the profiled parameter represented on the X-axis and the best possible calibration error (i.e. fitness value) on the Y-axis. To ease the interpretation of the profiles, we define an acceptance threshold on the calibration error (i.e fitness). Under this acceptance threshold the calibration error is considered sufficiently satisfying and the dynamics exposed by the model sufficiently acceptable. Over this acceptance threshold the calibration error is considered too high and the dynamics exposed by the model are considered unacceptable, or at least non-pertinent.
The computed calibration profiles may take very diverse shapes depending on the effect of the parameter of the model dynamics, however some of this shapes are recurrent. The most typical shapes are shown on the figure bellow.


  • Shape 1 occurs when a parameter is constraining the dynamic of the model (with respect to the fitness calibration criterion). The model is able to produce acceptable dynamics only for a single specific range of the parameter. In this case a connected (i.e. "one piece" domain) validity interval can be established for the parameter.
  • Shape 2 occurs when a parameter is constraining the dynamic of the model (with respect to the fitness calibration criterion), but the validity domain of the parameter is not connected. It might mean that several qualitatively different dynamics of the model meet the fitness requirement. In this case, model dynamics should be observed directly to determine if every kind of dynamics is suitable or if the fitness should be revised.
  • Shape 3 occurs when the model "resists" to calibration. The profile does not expose any acceptable dynamic according to the calibration criterion. In this case, the model should be improved or the calibration criterion should be revised.
  • Shape 4 occurs when a parameter does not constrain sufficiently the model dynamics (with respect to the fitness calibration criterion). The model can always be calibrated whatever the value of the profiled parameter. In this case this parameter constitute a superfluous degree of liberty for the model : its effect is compensated by a variation of the other parameters. In general it means that either this parameter should be fixed, or that a mechanism of the model should be removed, or that the model should be reduced by expressing the value of this parameter as a function of the other parameters.


The calibration profile algorithm has been published in the following paper: Romain Reuillon, Clara Schmitt, Ricardo De Aldama, and Jean-Baptiste Mouret, ยซA New Method to Evaluate Simulation Models: The Calibration Profile (CP) Algorithm ยป published in Journal of Artificial Societies and Social Simulation (JASSS) , Vol 18, Issue 1, 2015.
[online version] [bibteX]

Stochastic models ๐Ÿ”—

You can check additional options to profile stochastic models on this page.