Estimation of Model Parameters Challenge

ch_no: 
D6C2

Definitions

P-value for Parameter Predictions

Call Np the number of parameters to predict. Each model has a different number of parameters.  Models 1, 2 and 3 have N= 34, 41 and 57 parameters to predict. Let’s denote as vipred and vireal the predicted and actual parameters used in the simulations where i runs between 1 and Np. The “distance” between predicted and real parameters can be taken to be:

image

A null model was be created from the distance between estimated and known parameters, based on the predictions of all the participants. If there are M participants, we chose at random one of the M predictions for v1pred, then at random one of the M predictions for v2pred, …, and finally one of the M predictions for vNppred. The resulting value of Djparam represents one possible random choice of predictions amongst all the participants. By doing the same process a large number of times, we generated a distribution of Djparam between known and estimated parameters, for which a p-value can be estimated for the actual predictions. That p-value denoted as pjparam for model j is tabulated for each team for models 1, 2 and 3 in the the table below. 

P-value for Time-Course Predictions

Let’s denote as pkpred(ti) and pksim(ti) the predicted and simulated levels of protein k at times ti. Because the initial conditions were given, the real challenging predictions take place after some time has elapsed from t=0. We took that time to be 5. Therefore the squared distance between predicted and measured protein abundances for model j was taken to be:

image

Note that the squared difference terms are normalized with the model of noise that was implemented in the data provided.A null model was created from this distance, based on the predictions of all the participants. If there are M participants, we chose at random one of the M predictions for pkpred(t11), then at random one of the M predictions for pkpred(t12), …, and finally one of the M predictions for pkpred(t40). The resulting value of Djprot represents one possible random choice of predictions amongst all the participants. By doing the same process a large number of times, we generated a distribution of distance squares, for which a p-value can be estimated for the actual predictions. That p-value denoted as pjprot for model j is tabulated for each team for models 1, 2 and 3 in the the table below. 

Overall score

For each of the three models, each team obtained a p-value for the time-course predictions and a p-value for the parameter predictions. The overall score is the product of all the p-values:

image

Teams that participated in the challenge can login to de-anonymize their identity.


p-value  for parameter predictions
p-value  for protein time-couse predictions

OverallScore

Team

Model 1

Model 2

Model 3

Model 1

Model 2

Model 3

All Models

orangeballs

3.7E-03

3.7E-07

5.1E-04

1.0E-23

2.8E-08

1.1E-08

116.6

crux

2.8E-05

3.4E-03

8.4E-02

3.9E-22

1.2E-01

5.0E-02

73.1

Team #71

4.8E-03

1.2E-03

1.2E-02

3.0E-15

6.4E-04

1.2E-01

59.4

Team #136

2.7E-03

1.2E-01

5.6E-02

1.3E-06

6.6E-08

3.6E-06

53.5

Team #105

6.6E-03

2.7E-04

6.0E-02

7.0E-07

6.4E-08

1.0E+00

46.8

Team #76

2.6E-02

1.6E-01

8.2E-01

7.9E-15

8.7E-01

1.0E+00

38.3

Team #39

2.5E-01

5.6E-01

5.3E-01

1.3E-08

2.3E-06

9.8E-01

33.7

Team #150

2.6E-02

9.6E-01

1.0E+00

1.8E-04

1.0E+00

4.0E-06

24.7

Team #194

2.2E-01

2.1E-02

1.1E-01

1.0E+00

1.6E-03

6.7E-05

23.6

Team #228

8.4E-04

1.3E-01

4.4E-02

2.2E-04

1.0E+00

1.0E+00

20.7

Team #109

1.0E+00

9.9E-01

9.9E-01

1.0E+00

4.4E-04

1.6E-05

18.8

Team #23

4.4E-01

3.4E-02

3.7E-01

1.0E+00

2.3E-01

1.3E-05

17.9

Team #199

1.9E-03

2.4E-01

2.9E-02

5.2E-03

4.4E-01

1.0E+00

17.3

Team #77

2.2E-01

4.9E-01

7.9E-03

1.0E+00

1.0E+00

9.9E-04

14.0

Team #135

1.0E+00

9.4E-01

6.7E-01

1.0E+00

7.3E-01

7.6E-05

10.3

Team #157

2.6E-01

3.3E-01

5.2E-01

1.0E+00

5.8E-02

8.0E-01

6.2

Team #89

4.1E-01

2.5E-01

2.0E-01

1.0E+00

4.0E-01

1.0E+00

4.8

Team #129

7.0E-01

1.0E+00

9.3E-01

1.0E+00

1.0E+00

8.9E-02

2.8

Team #88

8.7E-01

5.6E-01

2.6E-01

1.0E+00

1.0E+00

8.8E-01

2.2