Estimation of Model Parameters: Houston, there was a problem

as3 = ((1+((p1/v3_Kd)^v3_h))-1)/((1+((p1/v3_Kd)^v3_h)));

cod3 = pro3_strength * (( as3) *(rs4));

rs4 = 1.0/((1+((p2/v4_Kd)^v4_h)));

In this case, the repression terms (rs4) was updated after the rate term (cod3) which depends on rs4. This means that a temporal shift is introduced during the ODE integration, as the protein concentration of the previous integration step is used to calculate the repression term. The scoring and data given for purchase were generated using the unintended (i.e., wrong order) order of differential equations.

Participants using the SBML model definitions with "intelligent" integrators, or those that explored the equations by eye and spotted the error, integrated the equations correctly. However, they were unintentionally penalized in the process of fitting the model parameters because, even though their model was correct, the data purchased and the predictions were generated with the incorrect order of assignments. his error had different effects in different models. Plots for data offered for purchase given by DREAM and the corrected integration scheme can be seen in www.fdmold.uni-freiburg.de/~ckreutz/ComparisonWithoutNoise.zip. Some analysis show that in model 1 and model 2 the error introduced is not considerable, but this is not the case for model 3.The exact effect of this error in the fitting depends on the order the teams purchased the data, and is hard to disentangle.

Some Analysis

We studied the correlation of between the final scores and the average square difference between data purchased and data correctly simulated with the right parameters for the different teams. We saw no correlation, which in principle indicates that the variance due to different fitting ability is comparable to the effect of the errors in integration. We also requested participants to tell us what integration method was used. In principle, people using Matlab (as given in the challenge), should have had an advantage over people using the corrected integration scheme. However, this did not result in an enrichment of MATLAB users amongst the best performing teams.

Are the Scores right or wrong?

The strict ordering of the scores is incorrect. However, there are a number of conclusions that can be extracted from the challenge as is:

1) The team Orangeballs was the best performer. They independently discovered that the provided Matlab code updated repression coefficients one time step later than activation coefficients, and decided to used this "wrong" dynamics in their fitting given that it seemed to yield consistent results. Therefore their best performance can be considered as justified. They were able to fit the the parameters correctly using the dynamics used by the organizers.

2) The second best performing team, team crux, is the team that discovered the error. Despite the fact that they "corrected" the order of integration, they obtained a second place. But if they had used the same dynamics as the one used to produced the data for purchase, they could have obtained the best performance. Therefore, we cannot say that they are in second place. We will call Orangeballs and crux best performers, without distinction of 1st place or 2nd place.

3) The difference in dynamics using different order of assignments in the odes in model 1 and 2 are smaller than those in model 3. So, in a way, a re-ranking of performance according to models 1 and 2 may seem reasonable. However, the resulting order could have changed if the teams had purchased their data in a different order.

4) Another alternative is to separate the teams as those whose model is the same as the one used to generate the dynamics given for purchase (such as team orangeballs), and those who were trying to fit that data with a slightly wrong model (as is the case in real biological applications). We could analyze each set separately, as they represent almost 2 different challenges. We could use the comparison between both lists as an example of what happens when our starting model assumptions are wrong (in this case there was a delay in system). We can get reasonably good fits and parameters even when your assumptions are not fully correct. Clearly in this reframing of the challenge there would still be two best performers: Orangeballs and crux, which also justifies our assignment of these two teams as best performers.

A few lessons

There were many lessons to learn from this challenge despite the mistakes.

1) We should have given the exact code for integration we used, including the error models.

2) SBML specification should include the order of the differential equations, as this problem can make for serious divergence of results intended to be the same.

3) We should have specified the time steps, and integrator used for this challenge.

We hope we will do this challenge again next year, and next time we hope to fix all these wrinkles. Raise your hand if you would participate again?

Thanks and Apologies

 We want to thank all the effort that was put by participants in this challenge. We are truly and deeply sorry that the clear success of this challenge was somehow deterred by an error in coding.

The DREAM organizers