These are results of the experiments for Retro:
| Benchmark |
pyex-hard-prod |
| Timeout |
10 s |
| Benchmarks |
20022 |
It seems your data is too big for client-side DataTables. You may consider server-side processing: https://rstudio.github.io/DT/server.htmlIt seems your data is too big for client-side DataTables. You may consider server-side processing: https://rstudio.github.io/DT/server.html
| RMC timeouts |
3143 |
| CVC4 timeouts |
331 |
| Z3 timeouts |
3236 |
| CVC4 and Z3 timeout |
331 |
| CVC4 timeouts but RMC does not |
181 |
| Z3 timeouts but RMC does not |
2640 |
Cases where RMC finishes and CVC4 doesn’t
It seems your data is too big for client-side DataTables. You may consider server-side processing: https://rstudio.github.io/DT/server.htmlIt seems your data is too big for client-side DataTables. You may consider server-side processing: https://rstudio.github.io/DT/server.html
LS0tCnRpdGxlOiAiUmV0cm8gUmVzdWx0IEFuYWx5c2lzIgpwYXJhbXM6CiAgI2Rpcjoga2FsdXphLWhhcmRfMQogICNzdWZmaXg6IGthbHV6YS1oYXJkCiAgZGlyOiBweWV4LWhhcmQtcHJvZAogIHN1ZmZpeDogcHlleC1oYXJkLXByb2QKICAjZGlyOiBzdHIyXzEKICAjc3VmZml4OiBzdHIyCm91dHB1dDoKICBodG1sX25vdGVib29rOgogICAgY29kZV9mb2xkaW5nOiBoaWRlCiAgcGRmX2RvY3VtZW50OiBkZWZhdWx0CiAgaHRtbF9kb2N1bWVudDoKICAgIGRmX3ByaW50OiBwYWdlZAogICAgdG9jOiB0cnVlCiAgICB0b2NfZmxvYXQ6IHRydWUKLS0tCgpgYGB7ciBlY2hvPUZBTFNFfQojPT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09CiMgUFJFQU1CTEUKIz09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PT09PQoKIyBsb2FkIHRoZSBwbG90dGluZyBsaWJyYXJ5CnN1cHByZXNzTWVzc2FnZXMobGlicmFyeShnZ3Bsb3QyKSkKbGlicmFyeShncmlkKQpsaWJyYXJ5KGdyaWRFeHRyYSkKbGlicmFyeSh0aWt6RGV2aWNlKQpsaWJyYXJ5KERUKQoKdGhlbWVfc2V0KHRoZW1lX2J3KCkpCgpvcHRpb25zKHNjaXBlbj05OTkpICAjIHR1cm4tb2ZmIHNjaWVudGlmaWMgbm90YXRpb24gbGlrZSAxZSs0OAoKIyBzaXplIG9mIHBvaW50IGZvciBzY2F0dGVycGxvdHMKUE9JTlRfU0laRSA9IDAuNQoKIyB0aW1lb3V0ClRJTUVPVVQgPSAxMApUSU1FT1VUX1ZBTCA9IDEuMDUgKiBUSU1FT1VUCgojIEZVTkNUSU9OUwpyZWFkX2ZpbGUgPC0gZnVuY3Rpb24oYXBwcm9hY2gpIHsKICBmaWxlbmFtZSA9IHBhc3RlMCgiLi8iLCBwYXJhbXMkZGlyLCAiLyIsIGFwcHJvYWNoLCAiLSIsIHBhcmFtcyRzdWZmaXgsICIuY3N2IikKICBkZiA8LSByZWFkLmNzdjIoZmlsZW5hbWUsCiAgICAgICAgICAgICAgICAgIGhlYWRlcj1UUlVFLAogICAgICAgICAgICAgICAgICBzZXA9IjsiLAogICAgICAgICAgICAgICAgICBjb21tZW50LmNoYXI9IiMiLAogICAgICAgICAgICAgICAgICBzdHJpcC53aGl0ZT1UUlVFKQoKICAjIGJhc2ljIHByZXByb2Nlc3NpbmcKICBkZiA8LSBkZltjKCJGb3JtdWxhIiwgInNhdCIsICJ0aW1lIildCgogICMgZGVhbCB3aXRoIHRpbWVvdXRzIGFuZCB0aW1lIHR5cGUKICBkZltkZiA9PSAiVE8iXSA8LSBOQQogIGRmJHRpbWUgPC0gYXMuZG91YmxlKGFzLmNoYXJhY3RlcihkZiR0aW1lKSkKICBkZiR0aW1lW2lzLm5hKGRmJHRpbWUpXSA8LSBUSU1FT1VUX1ZBTAogCiAgIyByZW5hbWUgdGhlIHRpbWUgY29sdW1uIAogIHRpbWVjb2xfbmFtZSA9IHBhc3RlMCgidGltZS4iLCBhcHByb2FjaCkKICBuYW1lcyhkZilbbmFtZXMoZGYpID09ICJ0aW1lIl0gPC0gdGltZWNvbF9uYW1lCiAgCiAgcmV0dXJuKGRmKQp9CgptZXJnZV9kZnMgPC0gZnVuY3Rpb24obHMpIHsKICBkZiA8LSBsc1tbMV1dCiAgZGYkc2F0IDwtIGFzLmNoYXJhY3RlcihkZiRzYXQpCiAgZm9yKGkgaW4gMjpsZW5ndGgobHMpKSB7CiAgICBkZl9scyA8LSBsc1tbaV1dCiAgICBkZl9scyRzYXQgPC0gYXMuY2hhcmFjdGVyKGRmX2xzJHNhdCkKCiAgICAjIGV4dHJhY3QgInNhdCIgY29sdW1ucwogICAgZGZfc2F0IDwtIGRmW2MoIkZvcm11bGEiLCAic2F0IildCiAgICBkZl9zYXRfbHMgPC0gZGZfbHNbYygiRm9ybXVsYSIsICJzYXQiKV0KICAgIG5hbWVzKGRmX3NhdF9scylbMl0gPC0gInNhdF9uZXciCiAgICBkZl9zYXQgPC0gbWVyZ2UoZGZfc2F0LCBkZl9zYXRfbHMsIGJ5PSJGb3JtdWxhIikKICAgIGRmX3NhdCRjb25zaXN0ZW50IDwtIChkZl9zYXQkc2F0ID09IGRmX3NhdCRzYXRfbmV3KQogICAgZGZfc2F0JGNvbnNpc3RlbnQgPC0gZGZfc2F0JGNvbnNpc3RlbnQgfCAKICAgICAgICAgICAgICAgICAgICAgICAgIGlzLm5hKGRmX3NhdCRzYXQpIHwgCiAgICAgICAgICAgICAgICAgICAgICAgICBpcy5uYShkZl9zYXQkc2F0X25ldykKICAgICNhc3NpZ24oImRmX3NhdCIsIGRmX3NhdCwgLkdsb2JhbEVudikKCiAgICAjIGNoZWNrIHRoZSByZXN1bHRzIGFyZSBjb25zaXN0ZW50CiAgICB3cml0ZS5jc3YyKGRmX3NhdFshZGZfc2F0JGNvbnNpc3RlbnQsXSwgImluY29uc2lzdGVudC5jc3YiKQogICAgc3RvcGlmbm90KGFsbChkZl9zYXQkY29uc2lzdGVudCwgbmEucm09VFJVRSkpCiAgICAKICAgICMgY29uc3RydWN0IG5ldyAic2F0IiBjb2x1bW4KICAgIGRmJHNhdCA8LSBpZmVsc2UoaXMubmEoZGYkc2F0KSwgZGYkc2F0LCBkZl9scyRzYXQpCgogICAgIyByZW5hbWUgdGhlIHRpbWUgY29sdW1uCiAgICB0aW1lY29sX25hbWUgPC0gZ3JlcCgidGltZS4iLCBuYW1lcyhkZl9scyksIHZhbHVlPVRSVUUpCiAgICBkZl9hdXggPC0gZGZfbHNbLCBjKCJGb3JtdWxhIiwgdGltZWNvbF9uYW1lKV0KICAgIAogICAgIyBtZXJnZSB3aXRoIHRoZSByZXN0CiAgICBkZiA8LSBtZXJnZShkZiwgZGZfYXV4LCBieT0iRm9ybXVsYSIpCiAgfQogIHJldHVybihkZikKfQoKbWVyZ2VfZGZzX29sZCA8LSBmdW5jdGlvbihscykgewogIGRmIDwtIGxzW1sxXV0KICBmb3IoaSBpbiAyOmxlbmd0aChscykpIHsKICAgIGRmX2xzIDwtIGxzW1tpXV0KCiAgICAjIGV4dHJhY3QgInNhdCIgY29sdW1ucyAgICAKICAgIG9yaWdfc2F0IDwtIGFzLmNoYXJhY3RlcihkZltbInNhdCJdXSkKICAgIG5ld19zYXQgPC0gYXMuY2hhcmFjdGVyKGRmX2xzW1sic2F0Il1dKQogICAgCiAgICAjIGNoZWNrIHRoZSByZXN1bHRzIGFyZSBjb25zaXN0ZW50CiAgICBzYW5pdHlfY2hlY2sgPSAobmV3X3NhdCA9PSBvcmlnX3NhdCkKICAgIHN0b3BpZm5vdChhbGwoc2FuaXR5X2NoZWNrLCBuYS5ybT1UUlVFKSkKICAgIAogICAgIyBjb25zdHJ1Y3QgbmV3ICJzYXQiIGNvbHVtbgogICAgZmlsbGVkX3NhdCA8LSBpZmVsc2UoaXMubmEob3JpZ19zYXQpLCBuZXdfc2F0LCBvcmlnX3NhdCkKICAgIGRmJHNhdCA8LSBmaWxsZWRfc2F0CiAgICAKICAgICMgcmVuYW1lIHRoZSB0aW1lIGNvbHVtbgogICAgdGltZWNvbF9uYW1lIDwtIGdyZXAoInRpbWUuIiwgbmFtZXMoZGZfbHMpLCB2YWx1ZT1UUlVFKQogICAgZGZfYXV4IDwtIGRmX2xzWywgYygiRm9ybXVsYSIsIHRpbWVjb2xfbmFtZSldCiAgICAKICAgICMgbWVyZ2Ugd2l0aCB0aGUgcmVzdAogICAgZGYgPC0gbWVyZ2UoZGYsIGRmX2F1eCwgYnk9IkZvcm11bGEiKQogIH0KICByZXR1cm4oZGYpCn0KYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQpkZl9ybWMgPC0gcmVhZF9maWxlKCJybWMiKQojZGZfcm1jX2hldXIgPC0gcmVhZF9maWxlKCJybWNfaGV1ciIpCmRmX2N2YzQgPC0gcmVhZF9maWxlKCJjdmM0IikKZGZfejMgPC0gcmVhZF9maWxlKCJ6MyIpCgojZGYgPC0gbWVyZ2VfZGZzKGxpc3QoZGZfcm1jLCBkZl9ybWNfaGV1ciwgZGZfY3ZjNCwgZGZfejMpKQpkZiA8LSBtZXJnZV9kZnMobGlzdChkZl9ybWMsIGRmX2N2YzQsIGRmX3ozKSkKZGYkdXJsIDwtIHBhc3RlMCgiPGEgaHJlZj1cImh0dHBzOi8vZ2l0aHViLmNvbS9WZXJpRklUL3JldHJvL3RyZWUvbWFzdGVyL2VxdWF0aW9ucy9weWV4LWhhcmQtejMtZW51bS9zbXQvIiwgZGYkRm9ybXVsYSwgIlwiPiIsIGRmJEZvcm11bGEsICI8L2E+IikKYGBgCgpUaGVzZSBhcmUgcmVzdWx0cyBvZiB0aGUgZXhwZXJpbWVudHMgZm9yIFJldHJvOgoKfCAgICAgICAgICAgICAgICAgICAgIHwgICAgICAgICAgICAgICAgICAgICB8CnwtLS0tLS0tLS0tLS0tLS0tLS0tLS18LS0tLS0tLS0tLS0tLS0tLS0tLS06fAp8ICoqQmVuY2htYXJrKiogICAgICAgfCBgciBwYXJhbXMkZGlyYCAgICAgIHwKfCAqKlRpbWVvdXQqKiAgICAgICAgIHwgYHIgVElNRU9VVGAgcyAgICAgICB8CnwgKipCZW5jaG1hcmtzKiogICAgICB8IGByIG5yb3coZGYpYCAgICAgICAgfAoKYGBge3IgZWNobz1GQUxTRX0KZGF0YXRhYmxlKGRmLCBlc2NhcGU9RkFMU0UpCmBgYApgYGB7ciBlY2hvPUZBTFNFfQpybWNfdGltZW91dCAgICAgIDwtIG5yb3coZGZbZGYkdGltZS5ybWMgPT0gVElNRU9VVF9WQUwsXSkKI3JtY19oZXVyX3RpbWVvdXQgPC0gbnJvdyhkZltkZiR0aW1lLnJtY19oZXVyID09IFRJTUVPVVRfVkFMLF0pCmN2YzRfdGltZW91dCAgICAgPC0gbnJvdyhkZltkZiR0aW1lLmN2YzQgPT0gVElNRU9VVF9WQUwsXSkKejNfdGltZW91dCAgICAgICA8LSBucm93KGRmW2RmJHRpbWUuejMgPT0gVElNRU9VVF9WQUwsXSkKejNfY3ZjNF90aW1lb3V0ICA8LSBucm93KGRmW2RmJHRpbWUuejMgPT0gVElNRU9VVF9WQUwgJiBkZiR0aW1lLmN2YzQgPT0gVElNRU9VVF9WQUwsXSkKCmN2YzRfdGltZW91dF9ybWNfbm9fdGltZW91dCA8LSBkZltkZiR0aW1lLmN2YzQgPT0gVElNRU9VVF9WQUwgJiBkZiR0aW1lLnJtYyAhPSBUSU1FT1VUX1ZBTCwgXQpjbnRfY3ZjNF90aW1lb3V0X3JtY19ub190aW1lb3V0IDwtIG5yb3coY3ZjNF90aW1lb3V0X3JtY19ub190aW1lb3V0KQpjbnRfejNfdGltZW91dF9ybWNfbm9fdGltZW91dCAgIDwtIG5yb3coZGZbZGYkdGltZS56MyA9PSBUSU1FT1VUX1ZBTCAmIGRmJHRpbWUucm1jICE9IFRJTUVPVVRfVkFMLCBdKQoKYGBgCnwgICAgICAgICAgICAgICAgICAgICAgICAgfCAgICAgICAgICAgICAgICAgICAgIHwKfC0tLS0tLS0tLS0tLS0tLS0tLS0tLS0tLS18LS0tLS0tLS0tLS0tLS0tLS0tLS06fAp8ICoqUk1DIHRpbWVvdXRzKiogICAgICAgIHwgYHIgcm1jX3RpbWVvdXRgICAgICB8CnwgKipDVkM0IHRpbWVvdXRzKiogICAgICAgfCBgciBjdmM0X3RpbWVvdXRgICAgIHwKfCAqKlozIHRpbWVvdXRzKiogICAgICAgICB8IGByIHozX3RpbWVvdXRgICAgICAgfAp8ICoqQ1ZDNCBhbmQgWjMgdGltZW91dCoqIHwgYHIgejNfY3ZjNF90aW1lb3V0YCB8CnwgKipDVkM0IHRpbWVvdXRzIGJ1dCBSTUMgZG9lcyBub3QqKiB8IGByIGNudF9jdmM0X3RpbWVvdXRfcm1jX25vX3RpbWVvdXRgIHwKfCAqKlozIHRpbWVvdXRzIGJ1dCBSTUMgZG9lcyBub3QqKiAgIHwgYHIgY250X3ozX3RpbWVvdXRfcm1jX25vX3RpbWVvdXRgIHwKCiMjIENhc2VzIHdoZXJlIFJNQyBmaW5pc2hlcyBhbmQgQ1ZDNCBkb2Vzbid0CgpgYGB7ciBlY2hvPUZBTFNFfQpkYXRhdGFibGUoY3ZjNF90aW1lb3V0X3JtY19ub190aW1lb3V0LCBlc2NhcGU9RkFMU0UpCmBgYAoKIyMgU2NhdHRlciBwbG90cwoKYGBge3IgZWNobz1GQUxTRX0KCnBsb3Rfc2NhdHRlciA8LSBmdW5jdGlvbihkZiwgeGxhYiwgeWxhYikgewogIHBzY2F0IDwtIGdncGxvdChkZiwgYWVzX3N0cmluZyh4PXhsYWIsIHk9eWxhYikpICsKICAgIGdlb21fcG9pbnQoc2l6ZT1QT0lOVF9TSVpFKSArCiAgICBnZW9tX2FibGluZShzaXplPTAuMSkgKwogICAgc2NhbGVfeF9sb2cxMCgpICsKICAgIHNjYWxlX3lfbG9nMTAoKSArCiAgICB0aGVtZShheGlzLnRleHQueSA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDkwLCBoanVzdCA9IDAuNSkpICsKICAgIyBjb29yZF9maXhlZCh4bGltID0gYygwLjksIFRJTUVPVVRfVkFMKSwgeWxpbSA9IGMoMC45LCBUSU1FT1VUX1ZBTCkpICsKICAgIGNvb3JkX2ZpeGVkKHhsaW0gPSBjKDAuMDA5LCBUSU1FT1VUX1ZBTCksIHlsaW0gPSBjKDAuMDA5LCBUSU1FT1VUX1ZBTCkpICsKICAgIGxhYnMoCiAgICAgIHRpdGxlPSJUaXRsZSIsCiAgICAgIHN1YnRpdGxlPSJTdWJ0aXRsZSIsCiAgICAgIHg9eGxhYiwKICAgICAgeT15bGFiKQogIHJldHVybihwc2NhdCkKfQoKcGxvdDEgPC0gcGxvdF9zY2F0dGVyKGRmLCAidGltZS5ybWMiLCAidGltZS5jdmM0IikKI3Bsb3QyIDwtIHBsb3Rfc2NhdHRlcihkZiwgInRpbWUucm1jX2hldXIiLCAidGltZS5jdmM0IikKCiNncmlkLmFycmFuZ2UocGxvdDEsIHBsb3QyLCBuY29sID0gMikKZ3JpZC5hcnJhbmdlKHBsb3QxLCBuY29sID0gMSkKCnBsb3QxIDwtIHBsb3Rfc2NhdHRlcihkZiwgInRpbWUucm1jIiwgInRpbWUuejMiKQojcGxvdDIgPC0gcGxvdF9zY2F0dGVyKGRmLCAidGltZS5ybWNfaGV1ciIsICJ0aW1lLnozIikKCiNncmlkLmFycmFuZ2UocGxvdDEsIHBsb3QyLCBuY29sID0gMikKZ3JpZC5hcnJhbmdlKHBsb3QxLCBuY29sID0gMSkKCnBsb3QxIDwtIHBsb3Rfc2NhdHRlcihkZiwgInRpbWUuY3ZjNCIsICJ0aW1lLnozIikKcGxvdChwbG90MSkKYGBgCgpgYGB7ciBlY2hvPUZBTFNFfQoKIyBnZXR0aW5nIGRhdGEgZm9yIGNhY3R1cyBwbG90CmdldF9zb3J0ZWRfY29sIDwtIGZ1bmN0aW9uKGRmLCBjb2wpIHsKICB0aW1lIDwtIGRmW1tjb2xdXQogIHRpbWUgPC0gdGltZVt0aW1lICE9IFRJTUVPVVRfVkFMXQogIHRpbWUgPC0gc29ydCh0aW1lKQoKICBkZl9hdXggPC0gYXMuZGF0YS5mcmFtZSh0aW1lKQogIGRmX2F1eCRpbmRleCA8LSAxOmxlbmd0aChkZl9hdXgkdGltZSkKICAKICByZXR1cm4oZGZfYXV4KQp9CgojIGFkZHMgYSBuZXcgZGF0YSBzZXQgaW50byBhIGNhY3R1cyBwbG90CmFkZF90b19jYWN0dXMgPC0gZnVuY3Rpb24ocGNhY3R1cywgZGYsIGNvbCwgY29sb3VyKSB7CiAgZGZfYXV4IDwtIGdldF9zb3J0ZWRfY29sKGRmLCBjb2wpCiAgcGNhY3R1cyA8LSBwY2FjdHVzICsKICAgIGdlb21fbGluZShkYXRhPWRmX2F1eCwgYWVzKHg9aW5kZXgsIHk9dGltZSwgY29sb3I9Y29sb3VyKSwgbmEucm09VFJVRSkgKwogICAgZ2VvbV9wb2ludChkYXRhPWRmX2F1eCwgYWVzKHg9aW5kZXgsIHk9dGltZSwgY29sb3I9Y29sb3VyKSwgc2l6ZT1QT0lOVF9TSVpFLCBuYS5ybT1UUlVFKQogIAogIHJldHVybihwY2FjdHVzKQp9CgptYWtlX2NhY3R1cyA8LSBmdW5jdGlvbihkZiwgdGl0bGUpIHsKICBwY2FjdHVzIDwtIGdncGxvdCgpCgogIHBjYWN0dXMgPC0gYWRkX3RvX2NhY3R1cyhwY2FjdHVzLCBkZiwgInRpbWUucm1jIiwgIlJNQyIpCiAgI3BjYWN0dXMgPC0gYWRkX3RvX2NhY3R1cyhwY2FjdHVzLCBkZiwgInRpbWUucm1jX2hldXIiLCAiUk1DLWhldXIiKQogIHBjYWN0dXMgPC0gYWRkX3RvX2NhY3R1cyhwY2FjdHVzLCBkZiwgInRpbWUuY3ZjNCIsICJDVkM0IikKICBwY2FjdHVzIDwtIGFkZF90b19jYWN0dXMocGNhY3R1cywgZGYsICJ0aW1lLnozIiwgIlozIikKICAKICBwY2FjdHVzIDwtIHBjYWN0dXMgKyBsYWJzKAogICAgICAgICAgICAgICB0aXRsZT10aXRsZSwKICAgICAgICAgICAgICAgc3VidGl0bGU9IlN1YnRpdGxlIiwKICAgICAgICAgICAgICAgeD0iQmVuY2htYXJrcyIsCiAgICAgICAgICAgICAgIHk9IlRpbWUgW3NdIikKCiAgcmV0dXJuKHBjYWN0dXMpCn0KCnBjYWN0dXMgPC0gbWFrZV9jYWN0dXMoZGYsICJBbGwiKQpwbG90KHBjYWN0dXMpCgpkZl9zYXQgPC0gZGZbZGYkc2F0ID09ICJzYXQiLF0KcHJpbnQoZGZfc2F0KQpwbG90KG1ha2VfY2FjdHVzKGRmX3NhdCwgInNhdCIpKQoKI2RmX3Vuc2F0IDwtIGRmW2RmJHNhdCA9PSAidW5zYXQiLF0KI3ByaW50KGRmX3Vuc2F0KQojcGxvdChtYWtlX2NhY3R1cyhkZl91bnNhdCwgInVuc2F0IikpCgoKYGBgCg==