Benchmark timings for saem.mmkin
Johannes Ranke
Last change 17 February 2023 (rebuilt 2023-11-16)
Source:vignettes/web_only/saem_benchmarks.rmd
saem_benchmarks.rmdEach system is characterized by operating system type, CPU type, mkin version, saemix version and R version. A compiler was available, so if no analytical solution was available, compiled ODE models are used.
Every fit is only performed once, so the accuracy of the benchmarks is limited.
For the initial mmkin fits, we use all available cores.
n_cores <- parallel::detectCores()Test data
Please refer to the vignette dimethenamid_2018 for an
explanation of the following preprocessing.
dmta_ds <- lapply(1:7, function(i) {
ds_i <- dimethenamid_2018$ds[[i]]$data
ds_i[ds_i$name == "DMTAP", "name"] <- "DMTA"
ds_i$time <- ds_i$time * dimethenamid_2018$f_time_norm[i]
ds_i
})
names(dmta_ds) <- sapply(dimethenamid_2018$ds, function(ds) ds$title)
dmta_ds[["Elliot"]] <- rbind(dmta_ds[["Elliot 1"]], dmta_ds[["Elliot 2"]])
dmta_ds[["Elliot 1"]] <- NULL
dmta_ds[["Elliot 2"]] <- NULLTest cases
Parent only
parent_mods <- c("SFO", "DFOP", "SFORB", "HS")
parent_sep_const <- mmkin(parent_mods, dmta_ds, quiet = TRUE, cores = n_cores)
parent_sep_tc <- update(parent_sep_const, error_model = "tc")
t1 <- system.time(sfo_const <- saem(parent_sep_const["SFO", ]))[["elapsed"]]
t2 <- system.time(dfop_const <- saem(parent_sep_const["DFOP", ]))[["elapsed"]]
t3 <- system.time(sforb_const <- saem(parent_sep_const["SFORB", ]))[["elapsed"]]
t4 <- system.time(hs_const <- saem(parent_sep_const["HS", ]))[["elapsed"]]
t5 <- system.time(sfo_tc <- saem(parent_sep_tc["SFO", ]))[["elapsed"]]
t6 <- system.time(dfop_tc <- saem(parent_sep_tc["DFOP", ]))[["elapsed"]]
t7 <- system.time(sforb_tc <- saem(parent_sep_tc["SFORB", ]))[["elapsed"]]
t8 <- system.time(hs_tc <- saem(parent_sep_tc["HS", ]))[["elapsed"]]
anova(
sfo_const, dfop_const, sforb_const, hs_const,
sfo_tc, dfop_tc, sforb_tc, hs_tc) |> kable(, digits = 1)| npar | AIC | BIC | Lik | |
|---|---|---|---|---|
| sfo_const | 5 | 796.3 | 795.3 | -393.2 |
| sfo_tc | 6 | 798.3 | 797.1 | -393.2 |
| dfop_const | 9 | 709.4 | 707.5 | -345.7 |
| sforb_const | 9 | 710.0 | 708.1 | -346.0 |
| hs_const | 9 | 713.7 | 711.8 | -347.8 |
| dfop_tc | 10 | 670.1 | 668.0 | -325.0 |
| sforb_tc | 10 | 662.9 | 660.8 | -321.4 |
| hs_tc | 10 | 667.2 | 665.1 | -323.6 |
The above model comparison suggests to use the SFORB model with two-component error. For comparison, we keep the DFOP model with two-component error, as it competes with SFORB for biphasic curves.
illparms(dfop_tc)## [1] "sd(log_k2)"
illparms(sforb_tc)## [1] "sd(log_k_DMTA_bound_free)"For these two models, random effects for the transformed parameters
k2 and k_DMTA_bound_free could not be
quantified.
One metabolite
We remove parameters that were found to be ill-defined in the parent only fits.
one_met_mods <- list(
DFOP_SFO = mkinmod(
DMTA = mkinsub("DFOP", "M23"),
M23 = mkinsub("SFO")),
SFORB_SFO = mkinmod(
DMTA = mkinsub("SFORB", "M23"),
M23 = mkinsub("SFO")))
one_met_sep_const <- mmkin(one_met_mods, dmta_ds, error_model = "const",
cores = n_cores, quiet = TRUE)
one_met_sep_tc <- mmkin(one_met_mods, dmta_ds, error_model = "tc",
cores = n_cores, quiet = TRUE)
t9 <- system.time(dfop_sfo_tc <- saem(one_met_sep_tc["DFOP_SFO", ],
no_random_effect = "log_k2"))[["elapsed"]]
t10 <- system.time(sforb_sfo_tc <- saem(one_met_sep_tc["SFORB_SFO", ],
no_random_effect = "log_k_DMTA_bound_free"))[["elapsed"]]Three metabolites
For the case of three metabolites, we only keep the SFORB model in order to limit the time for compiling this vignette, and as fitting in parallel may disturb the benchmark. Again, we do not include random effects that were ill-defined in previous fits of subsets of the degradation model.
illparms(sforb_sfo_tc)
three_met_mods <- list(
SFORB_SFO3_plus = mkinmod(
DMTA = mkinsub("SFORB", c("M23", "M27", "M31")),
M23 = mkinsub("SFO"),
M27 = mkinsub("SFO"),
M31 = mkinsub("SFO", "M27", sink = FALSE)))
three_met_sep_tc <- mmkin(three_met_mods, dmta_ds, error_model = "tc",
cores = n_cores, quiet = TRUE)
t11 <- system.time(sforb_sfo3_plus_const <- saem(three_met_sep_tc["SFORB_SFO3_plus", ],
no_random_effect = "log_k_DMTA_bound_free"))[["elapsed"]]Results
Benchmarks for all available error models are shown. They are intended for improving mkin, not for comparing CPUs or operating systems. All trademarks belong to their respective owners.
Parent only
Constant variance for SFO, DFOP, SFORB and HS.
| CPU | OS | mkin | saemix | t1 | t2 | t3 | t4 |
|---|---|---|---|---|---|---|---|
| Ryzen 7 1700 | Linux | 1.2.0 | 3.2 | 2.140 | 4.626 | 4.328 | 4.998 |
| Ryzen 7 1700 | Linux | 1.2.2 | 3.2 | 2.427 | 4.550 | 4.217 | 4.851 |
| Ryzen 9 7950X | Linux | 1.2.1 | 3.2 | 1.352 | 2.813 | 2.401 | 2.074 |
| Ryzen 9 7950X | Linux | 1.2.2 | 3.2 | 1.328 | 2.738 | 2.336 | 2.023 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 1.118 | 2.036 | 2.010 | 2.088 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 1.419 | 2.374 | 1.926 | 2.398 |
| Ryzen 9 7950X | Linux | 1.2.4 | 3.2 | 0.972 | 2.550 | 1.987 | 2.055 |
| Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz | Linux | 1.2.6 | 3.2 | 2.998 | 6.523 | 6.126 | 4.721 |
| Ryzen 9 7950X | Linux | 1.2.6 | 3.2 | 1.135 | 2.025 | 2.406 | 2.478 |
Two-component error fits for SFO, DFOP, SFORB and HS.
| CPU | OS | mkin | saemix | t5 | t6 | t7 | t8 |
|---|---|---|---|---|---|---|---|
| Ryzen 7 1700 | Linux | 1.2.0 | 3.2 | 5.678 | 7.441 | 8.000 | 7.980 |
| Ryzen 7 1700 | Linux | 1.2.2 | 3.2 | 5.352 | 7.201 | 8.174 | 8.401 |
| Ryzen 9 7950X | Linux | 1.2.1 | 3.2 | 2.388 | 3.033 | 3.532 | 3.310 |
| Ryzen 9 7950X | Linux | 1.2.2 | 3.2 | 2.341 | 2.968 | 3.465 | 3.341 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 2.159 | 3.584 | 3.307 | 3.460 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 2.348 | 3.134 | 3.253 | 3.530 |
| Ryzen 9 7950X | Linux | 1.2.4 | 3.2 | 2.127 | 3.587 | 3.433 | 3.595 |
| Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz | Linux | 1.2.6 | 3.2 | 5.070 | 8.464 | 8.525 | 7.599 |
| Ryzen 9 7950X | Linux | 1.2.6 | 3.2 | 2.161 | 3.325 | 3.669 | 3.153 |
One metabolite
Two-component error for DFOP-SFO and SFORB-SFO.
| CPU | OS | mkin | saemix | t9 | t10 |
|---|---|---|---|---|---|
| Ryzen 7 1700 | Linux | 1.2.0 | 3.2 | 24.465 | 800.266 |
| Ryzen 7 1700 | Linux | 1.2.2 | 3.2 | 25.193 | 798.580 |
| Ryzen 9 7950X | Linux | 1.2.1 | 3.2 | 11.247 | 285.216 |
| Ryzen 9 7950X | Linux | 1.2.2 | 3.2 | 11.242 | 284.258 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 11.796 | 216.012 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 12.841 | 292.688 |
| Ryzen 9 7950X | Linux | 1.2.4 | 3.2 | 12.160 | 265.934 |
| Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz | Linux | 1.2.6 | 3.2 | 30.168 | 748.675 |
| Ryzen 9 7950X | Linux | 1.2.6 | 3.2 | 12.007 | 286.757 |
Three metabolites
Two-component error for SFORB-SFO3-plus
| CPU | OS | mkin | saemix | t11 |
|---|---|---|---|---|
| Ryzen 7 1700 | Linux | 1.2.0 | 3.2 | 1289.198 |
| Ryzen 7 1700 | Linux | 1.2.2 | 3.2 | 1312.445 |
| Ryzen 9 7950X | Linux | 1.2.1 | 3.2 | 489.939 |
| Ryzen 9 7950X | Linux | 1.2.2 | 3.2 | 482.970 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 392.364 |
| Ryzen 9 7950X | Linux | 1.2.3 | 3.2 | 483.027 |
| Ryzen 9 7950X | Linux | 1.2.4 | 3.2 | 456.252 |
| Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz | Linux | 1.2.6 | 3.2 | 1235.028 |
| Ryzen 9 7950X | Linux | 1.2.6 | 3.2 | 480.577 |