This is a database to store benchmark results for programs that answer queries from large input data files, e.g. logic programming systems, deductive databases, relational databases, and graph databases/RDF stores.
The benchmarks were inspired by the OpenRuleBench, see also OpenRuleBench Download.
However, the OpenRuleBench scripts do not support executing the same benchmark for the same program multiple times. This is important to get reliable data, especially when one is also interested in slight differences. Since we are developing our own deductive database based on the Push Method, we are interested even in minor runtime improvements. Furthermore, it is very important for us to compare different versions of our software, or the same version with different compiler settings. Moreover, we run benchmarks on different machines with different operating systems. The OpenRuleBench scripts have no support for this (they were developed with a different application in mind, namely to get an overview of the performance of rule-based systems for semantic web applications).
One soon loses the overview if one keeps the measurements in simple text files. Therefore we developed a relational database to store and evaluate runtime measurements (and memory consumption, too).
A problem for fair benchmarks is to find the best settings for each systems for a given benchmark, e.g. which indexes should be used for a relational database. As we understood the OpenRuleBench, the developers of each system were ask to define the optimal configuration of their system for each benchmark. However, we did find some non-optimal configurations.
In order to give some confidence in our benchmark results, we tried several different settings for the same program and benchmark, and store the results in our benchmark database. In the main comparison, we use only the best settings, but in the database it can be seen what we tried and what were the results. Sometimes, it is not clear, what are the best settings, because most benchmarks are run on different input data files, and sometimes the best settings are different for different data files. However, for the main comparison of the systems, we require that for each benchmark there is only one setting manually selected as best. But our database contains all measurements, and one can check the alternatives.
Of course, we would be grateful for settings recommended by the system developers or experts for one of the systems we tested in our benchmarks.
Another design goal of our database was to be able to generate results in different formats (e.g. HTML, LaTeX, csv). In particular, we want to be able to generate a web site with the benchmark results more or less automatically (so that there are no errors in copying the data, and updates are simple).
MACHINE
: List of Machines on which Benchmarks are ExecutedThis table contains some data about the machines on which benchmarks were run. One machine is marked as the stardard machine by the value 'S' in column STATUS. However, the database permits to store and evaluate benchmark measurements for different machines with different architectures and operating systems.
INPUT_TYPE
: List of Input Data (Problem Instance) TypesWe have many input data files for the transitive closure benchmark, but since they contain general graph data, they can be used for different benchmarks. The entries in this table are used to classify data files. For instance, there are also data files for the DBLP benchmark and for the Join1 benchmark, and so on.
The currently defined input types are:
ITYPE | DESCRIPTION |
---|---|
GRAPH | Directed graph with edges stored in predicate par |
JOIN1 | Directed graph with edges in five relations: c2, c3, c4, d1, d2 |
DBLP | DBLP data in an EAV table att |
WINE | Wine ontology facts |
BENCHMARK
: List of BenchmarksThis is the list of all benchmarks, e.g. 'tcff' is a benchmark identifier in column BENCH. It corresponds to the computation of the transitive closure with query 'tc(X,Y)', i.e. with both arguments free. Another benchmark, 'tcbf', is the computation of transitive closure with query 'tc(1,X)', i.e. the first argument bound (to a given value), and the second argument free.
The currently defined benchmarks are:
BENCH | SEQ_NO | ITYPE | HEADLINE |
---|---|---|---|
dblp | 1 | DBLP | DBLP Benchmark |
j1axy | 2 | JOIN1 | The Join1 Benchmark, Query a(X,Y) |
j1axyd | 3 | JOIN1 | The Join1 Benchmark (without duplicate elimination) |
tcff | 4 | GRAPH | Transitive Closure Benchmark without Query Bindings |
tcbf | 5 | GRAPH | Transitive Closure Benchmark with the First Argument Bound |
tcbfs | 6 | GRAPH | Transitive Closure Benchmark, First Arg. Bound, with SLDMagic |
tcfb | 7 | GRAPH | Transitive Closure Benchmark with the Second Argument Bound |
wine | 8 | WINE | Wine Ontology Benchmark |
loadd | 9 | DBLP | Test of Loader for DBLP Data |
loadt | 10 | GRAPH | Test of Loader for Transitive Closure |
set1 | 11 | NULL | Test of Set Insertion and Lookup (One Column) |
set2 | 12 | NULL | Test of Set Insertion and Lookup (Two Columns) |
PROG_TYPES
: Program TypesThis table defines broad classes of programs, e.g. Prolog systems and Relational Databases. We use it for generating subheadlines in the list of all tested programs.
The current entries are:
PTYPE | SEQ_NO | HEADLINE |
---|---|---|
Prolog | 1 | Prolog Systems |
Datalog | 2 | Datalog Systems |
RDB | 3 | Relational Database Systems |
Graph | 4 | Graph Databases and RDF Stores |
PROGRAM
: List of Programs to be Tested/ComparedThe current list of programs is:
PROG | SEQ_NO | FULL_NAME | PTYPE | URL |
---|---|---|---|---|
xsb | 10 | XSB | Prolog | http://xsb.sourceforge.net/ |
yap | 11 | YAProlog | Prolog | https://www.dcc.fc.up.pt/~vsc/Yap/ |
swipl | 12 | SWI-Prolog | Prolog | http://www.swi-prolog.org/ |
mercury | 13 | Melbourne Mercury | Prolog | http://www.mercurylang.org/ |
bam | 21 | Push (bam) | Datalog | http://www.informatik.uni-halle.de/~brass/push/ |
ydb | 22 | Push (ydb) | Datalog | http://www.informatik.uni-halle.de/~brass/push/ |
dlv | 23 | DLV | Datalog | http://www.dlvsystem.com/ |
souffle | 24 | Souffle | Datalog | http://souffle-lang.org/ |
datascript | 25 | Datascript | Datalog | https://github.com/tonsky/datascript |
hsql | 31 | HSQLDB | RDB | http://hsqldb.org/ |
sqlite3 | 32 | SQLite 3 | RDB | https://www.sqlite.org/index.html |
mariadb | 33 | MariaDB | RDB | https://www.mariadb.com |
psql | 34 | PostgreSQL | RDB | https://www.postgresql.org |
jena | 41 | Apache Jena | Graph | https://jena.apache.org/ |
PROG_VER
: Program VersionsThis table contains information about program versions. Note that this table describes a version of the source code. PROG_INSTALL (below) describes a version of the program binary, which also depends on the machine, the compiler, compiler settings, and configuration settings.
PROG | VER | VER_ADDTEXT | VER_DATE |
---|---|---|---|
xsb | 3.7. | (Clan MacGregor) | 2016-07-06 |
xsb | 3.8.0 | (Three-Buck Chuck) | 2017-10-28 |
yap | 6.2.2 | (x86_64-linux) | 2011-10-21 |
yap | 6.3.3 | (x86_64-linux) | 2013-01-20 |
swipl | 7.7.15 | (x86_64-linux) | 2018-06-01 |
swipl | 7.5.15 | (x86_64-linux) | 2017-09-12 |
swipl | 7.6.4 | (x86_64-linux) | 2018-01-12 |
swipl | 7.7.7 | (x86_64-linux) | 2017-12-21 |
ydb | 0.1 | 2016-11-21 | |
bam | 0.1a | 2018-04-24 | |
souffle | 1.0 | NULL | |
datascript | 0.16.6 | 2018-06-06 | |
sqlite3 | 3.8.7.1 | 2014-10-29 | |
sqlite3 | 3.22.0 | 2018-01-22 | |
sqlite3 | 3.24.0 | 2018-06-04 | |
sqlite3 | 3.25.1 | 2018-09-18 | |
sqlite3 | 3.25.2 | 2018-09-25 | |
psql | 9.4.15 | 2017-11-09 | |
mariadb | 10.0.35 | 2018-05-03 | |
mariadb | 10.3.9 | 2018-08-15 | |
jena | 3.7.0 | 2018-04-05 | |
jena | 3.8.0 | 2018-06-25 |
PROG_INSTALL
: Table for Installed Program Versions on MachinesThis table lists program versions that are or were installed on a test machine. They are identified here by a number INSTALL that is unique together with the program name and the machine. These three elements (program, installation ID, and machine) are also encoded in the file names with the test results.
Whereas PROG_VER describes a version of the source code, PROG_INSTALL describes a binary program. For instance, if the same source code is compiled with g++ and with clang, these are two different entries in PROG_INSTALL, both referring to the same entry in PROG_VER. Also different configuration settings may give different binaries, and therefore different entries in PROG_INSTALL.
I try to make sure that the same version of a program gets the same INSTALL number even on different machines, but the database schema does not require this (and it is impossible when there are really different installations of the same program version).
Only program installations that are listed here can appear in the test runs.
This table also define a current version (more precisely, installation) of a program on a machine (by STATUS='C'). Some views show only results for the current version. It is also possible to define a previous version (STATUS='P') as reference, if the result of a program update should be evaluated. Finally, one can mark an alternative version (STATUS='A'), e.g. compiled with a different compiler or compiler options, which can also be used for comparison.
PROG | VER | MACHINE | INSTALL | COMPILER | STATUS |
---|---|---|---|---|---|
xsb | 3.7. | dbs1 | 1 | C | |
xsb | 3.8.0 | dbs1 | 2 | C | |
xsb | 3.8.0 | bach | 2 | C | |
yap | 6.2.2 | dbs1 | 1 | C | |
yap | 6.3.3 | bach | 2 | C | |
swipl | 7.7.15 | dbs1 | 1 | C | |
swipl | 7.5.15 | dbs1 | 2 | P | |
swipl | 7.6.4 | dbs1 | 3 | C | |
swipl | 7.7.7 | dbs1 | 4 | C | |
ydb | 0.1 | bach | 1 | gcc version 4.8.5 (-O2) | C |
ydb | 0.1 | dbs1 | 1 | gcc version 4.9.2 (-O2) | C |
bam | 0.1a | bach | 1 | gcc version 4.8.5 (-O2) | C |
bam | 0.1a | dbs1 | 1 | gcc version 4.9.2 (-O2) | C |
souffle | 1.0 | dbs1 | 1 | C | |
datascript | 0.16.6 | dbs1 | 1 | node 10.5.0 | C |
sqlite3 | 3.8.7.1 | dbs1 | 1 | P | |
sqlite3 | 3.22.0 | dbs1 | 2 | P | |
sqlite3 | 3.24.0 | dbs1 | 3 | P | |
sqlite3 | 3.25.1 | dbs1 | 4 | C | |
sqlite3 | 3.25.2 | dbs1 | 5 | A | |
psql | 9.4.15 | dbs1 | 1 | C | |
mariadb | 10.0.35 | dbs1 | 1 | P | |
mariadb | 10.3.9 | dbs1 | 2 | C | |
jena | 3.7.0 | dbs1 | 1 | openJDK 8 | P |
jena | 3.8.0 | dbs1 | 2 | openJDK 8 | C |
BENCH_IMPL
: Implementation Variants of BenchmarksIn general, there are different option settings (e.g. index selections) that should be tested for the same program to execute a given benchmark as fast as possible. We call such a combination of benchmark, program and option settings a benchmark implementation. STATUS='B' marks the best implementation that is used in some views to compare different programs. This is selected manually, but there are also views that help to find the best implementation. For instance, we tested the InnoDB und the Memory storage engine for MariaDB. It depended on the input data, which was faster. We selected InnoDB in this table for the main comparison between the systems, but the benchmark database contains results for both.
The current list of benchmark implementations is:
PROG | BENCH | IMPL | STATUS | DESCRIPTION |
---|---|---|---|---|
bam | j1axy | 1 | Join1, a(X,Y) [V1: RelAttr, -goto, fullDupCh] | |
bam | j1axy | 2 | Join1, a(X,Y) [V2: no goto, bitmap-dupcheck] | |
bam | j1axy | 3 | B | Join1, a(X,Y) [V3: goto, bitmap-dupC, cursor] |
bam | j1axy | 4 | A | Join1, a(X,Y) [V4: functions, bitmap-dupC] |
bam | j1axy | 5 | Join1, a(X,Y) [V5: fun without args, bitmap] | |
bam | j1axy | 6 | A | Join1, a(X,Y) [V6: Abstract Machine, bitmap] |
bam | tcff | 1 | tc(X,Y) [V1: RelLocalVar] | |
bam | tcff | 2 | A | tc(X,Y) [V2: Cursor Interface, switch] |
bam | tcff | 3 | A | tc(X,Y) [V3: Seminaive Evaluation] |
bam | tcff | 4 | A | tc(X,Y) [V4: Functions] |
bam | tcff | 5 | B | tc(X,Y) [V5: Abstract Machine] |
bam | loadt | 1 | A | transitive closure data file (with handcrafted loader) |
bam | loadt | 2 | B | transitive closure data file (with general loader) |
bam | set1 | 1 | B | Set with 1 column, extendible Hash table |
bam | set2 | 1 | B | Set with 2 columns, extendible Hash table |
xsb | dblp | dyn | A | load_dyn, no index |
xsb | dblp | dync | A | load_dync, no index |
xsb | dblp | dynin | A | load_dyn, index on col 1+2 |
xsb | dblp | dyncin | B | load_dync, index on col 1+2 |
xsb | dblp | dyntr | A | load_dyn, trie index |
xsb | dblp | dynctr | A | load_dync, trie index |
xsb | j1axyd | none | B | Join1 with duplicates, a(X,Y) [V0: no tabling] |
xsb | j1axy | a | Join1, a(X,Y) [V1: only a tabled] | |
xsb | j1axy | all | Join1, a(X,Y) [V2: all derived preds tabled (subsumptive)] | |
xsb | j1axy | allv | B | Join1, a(X,Y) [V4: all derived preds tabled, variant tabled] |
xsb | j1axy | alli | Join1, a(X,Y) [V4: all derived preds tabled, with indexes] | |
xsb | j1axy | allt | A | Join1, a(X,Y) [V5: all derived preds tabled, with trie index] |
xsb | tcff | dyn | tc(X,Y) [V1: load_dyn, trie-index for par] | |
xsb | tcff | B | tc(X,Y) [V2: load_dync, trie-index for par] | |
xsb | tcbf | dyn | tc(1,X) [V1: load_dyn, trie-index for par] | |
xsb | tcbf | B | tc(1,X) [V2: load_dync, trie-index for par] | |
xsb | tcbfs | slddc | B | tc(1,X) [V3: SLDMagic, load_dync, trie-index] |
xsb | tcfb | B | tc(X,1) [unknown implementation] | |
xsb | wine | is | B | wine ontology [V1: load_dync, subsumptive tabling, std. indexes] |
yap | dblp | 1 | B | DBLP Benchmark |
yap | j1axyd | none | B | Join1 with duplicates, a(X,Y) [V0: no tabling] |
yap | j1axy | a | Join1, a(X,Y) [V1: only a tabled] | |
yap | j1axy | all | B | Join1, a(X,Y) [V2: all derived preds tabled] |
yap | tcff | B | tc(X,Y) [V1: tc tabled] | |
yap | tcbf | B | tc(1,X) [V1: tc tabled] | |
yap | tcbfs | sldm | B | tc(1,X) [V2: with SLDMagic, all preds tabled] |
yap | tcfb | B | tc(X,1) [V1: tc tabled] | |
yap | wine | tab | B | wine ontology [V1: table 225 derived preds, reorder body lits |
swipl | dblp | B | DBLP Benchmark | |
swipl | j1axyd | B | Join1 with duplicates, a(X,Y) | |
swipl | j1axy | B | Join1, a(X,Y) | |
swipl | tcff | B | tc(X,Y) [tc tabled] | |
swipl | tcbf | B | tc(1,X) [tc tabled] | |
swipl | tcbfs | sldm | B | tc(1,X) [with SLDMagic, all preds tabled] |
swipl | tcfb | B | tc(X,1) [tc tabled] | |
swipl | wine | B | wine ontology | |
sqlite3 | tcff | B | transitive closure [ff] for SQLite3 | |
sqlite3 | tcbf | B | transitive closure [bf] for SQLite3 | |
sqlite3 | tcfb | B | transitive closure [fb] for SQLite3 | |
psql | tcff | B | transitive closure [ff] for PostgreSQL | |
psql | tcbf | B | transitive closure [bf] for PostgreSQL | |
psql | tcfb | B | transitive closure [fb] for PostgreSQL | |
mariadb | tcff | m | A | transitive closure for MariaDB, storage manager MEMORY |
mariadb | tcff | i | B | transitive closure for MariaDB, storage manager InnoDB |
mariadb | tcbf | B | transitive closure for MariaDB, unknown storage manager | |
mariadb | tcfb | B | transitive closure for MariaDB, unknown storage manager | |
jena | tcff | B | transitive closure [ff] with Property Paths for Apache Jena | |
jena | tcbf | B | transitive closure [bf] with Property Paths for Apache Jena | |
jena | tcfb | B | transitive closure [fb] with Property Paths for Apache Jena | |
datascript | tcff | B | transitive closure for DataScript | |
souffle | tcff | B | tc(X,Y) [V1: compiled, -j1] |
DATA_FILE
: Information about Data FilesThis a list of data files that can be used as input for a benchmark. Each file has a type ITYPE, and in the table BENCHMARK (see above) it is listed what kind of input is required for a benchmark. For instance, we have many data files that define directed graphs with a predicate par (parent, OpenRuleBench used this name for the transitive closure benchmarks).
Different programs need different file formats, therefore each file is available in several formats such as + Prolog facts (with extension '.P') + Tab-Separated-Values (extension '.tsv') + Comma-separated Values (extension '.csv') + SQL INSERT Statements (extension '.sql') + RDF in Turtle Syntax (extension '.ttl') + JSON (extension '.json') + graphviz (extension '.dot')
The columns of the table are:
The current contents of this table is:
FILE_ID | SEQ_NO | ITYPE | LINES | BYTES | FILENAME |
---|---|---|---|---|---|
k465 | 101 | GRAPH | 216225 | 3141075 | tc_complete_465 |
k500 | 102 | GRAPH | 250000 | 3640000 | tc_complete_500 |
k585 | 103 | GRAPH | 342225 | 5004675 | tc_complete_585 |
k670 | 104 | GRAPH | 448900 | 6586100 | tc_complete_670 |
k735 | 105 | GRAPH | 540225 | 7941675 | tc_complete_735 |
k795 | 106 | GRAPH | 632025 | 9305475 | tc_complete_795 |
k845 | 107 | GRAPH | 714025 | 10524475 | tc_complete_845 |
k890 | 108 | GRAPH | 792100 | 11685700 | tc_complete_890 |
k930 | 109 | GRAPH | 864900 | 12768900 | tc_complete_930 |
k965 | 110 | GRAPH | 931225 | 13756075 | tc_complete_965 |
k1k | 111 | GRAPH | 1000000 | 14780000 | tc_complete_1000 |
k2k | 112 | GRAPH | 4000000 | 63560000 | tc_complete_2000 |
t500 | 121 | GRAPH | 124750 | 1816360 | tc_total_500 |
t585 | 122 | GRAPH | 170820 | 2498060 | tc_total_585 |
t735 | 123 | GRAPH | 269745 | 3965435 | tc_total_735 |
t845 | 124 | GRAPH | 356590 | 5256010 | tc_total_845 |
t930 | 125 | GRAPH | 431985 | 6377585 | tc_total_930 |
t1k | 126 | GRAPH | 499500 | 7382610 | tc_total_1000 |
t1065 | 127 | GRAPH | 566580 | 8450820 | tc_total_1065 |
t1120 | 128 | GRAPH | 626640 | 9410790 | tc_total_1120 |
t1170 | 129 | GRAPH | 683865 | 10328115 | tc_total_1170 |
t1215 | 130 | GRAPH | 737505 | 11190045 | tc_total_1215 |
t1260 | 131 | GRAPH | 793170 | 12086400 | tc_total_1260 |
t2k | 132 | GRAPH | 1999000 | 31764110 | tc_total_2000 |
c1k | 141 | GRAPH | 1000 | 14780 | tc_cycle_1000 |
c2k | 142 | GRAPH | 2000 | 31780 | tc_cycle_2000 |
c3k | 143 | GRAPH | 3000 | 48780 | tc_cycle_3000 |
c4k | 144 | GRAPH | 4000 | 65780 | tc_cycle_4000 |
c10k | 145 | GRAPH | 10000 | 167780 | tc_cycle_10000 |
c14k1 | 146 | GRAPH | 14100 | 245680 | tc_cycle_14100 |
c17k3 | 147 | GRAPH | 17300 | 306480 | tc_cycle_17300 |
c20k | 148 | GRAPH | 20000 | 357780 | tc_cycle_20000 |
c22k4 | 149 | GRAPH | 22400 | 403380 | tc_cycle_22400 |
c24k5 | 150 | GRAPH | 24500 | 443280 | tc_cycle_24500 |
c26k5 | 151 | GRAPH | 26500 | 481280 | tc_cycle_26500 |
c28k3 | 152 | GRAPH | 28300 | 515480 | tc_cycle_28300 |
c30k | 153 | GRAPH | 30000 | 547780 | tc_cycle_30000 |
c31k6 | 154 | GRAPH | 31600 | 578180 | tc_cycle_31600 |
s1k_1 | 161 | GRAPH | 2000 | 29560 | tc_shorted_cycle_1000_1 |
s1k_2 | 162 | GRAPH | 3000 | 44340 | tc_shorted_cycle_1000_2 |
s1k_3 | 163 | GRAPH | 4000 | 59120 | tc_shorted_cycle_1000_3 |
s1k_4 | 164 | GRAPH | 5000 | 73900 | tc_shorted_cycle_1000_4 |
s1k_5 | 165 | GRAPH | 6000 | 88680 | tc_shorted_cycle_1000_5 |
s1k2_1 | 166 | GRAPH | 2400 | 36360 | tc_shorted_cycle_1200_1 |
s1k2_2 | 167 | GRAPH | 3600 | 54540 | tc_shorted_cycle_1200_2 |
s1k2_4 | 168 | GRAPH | 6000 | 90900 | tc_shorted_cycle_1200_4 |
s1k2_5 | 169 | GRAPH | 3357 | 50265 | tc_shorted_cycle_1200_5 |
s2k_1 | 170 | GRAPH | 4000 | 63560 | tc_shorted_cycle_2000_1 |
s2k_2 | 171 | GRAPH | 6000 | 95340 | tc_shorted_cycle_2000_2 |
s2k_3 | 172 | GRAPH | 8000 | 127120 | tc_shorted_cycle_2000_3 |
s2k_4 | 173 | GRAPH | 10000 | 158900 | tc_shorted_cycle_2000_4 |
s2k_5 | 174 | GRAPH | 12000 | 190680 | tc_shorted_cycle_2000_5 |
s2k4_1 | 175 | GRAPH | 4800 | 77160 | tc_shorted_cycle_2400_1 |
s2k4_2 | 176 | GRAPH | 7200 | 115740 | tc_shorted_cycle_2400_2 |
s2k4_4 | 177 | GRAPH | 12000 | 192900 | tc_shorted_cycle_2400_4 |
s2k4_5 | 178 | GRAPH | 14400 | 231480 | tc_shorted_cycle_2400_5 |
s4k8_1 | 179 | GRAPH | 9600 | 158760 | tc_shorted_cycle_4800_1 |
s4k8_2 | 180 | GRAPH | 14400 | 238140 | tc_shorted_cycle_4800_2 |
s4k8_4 | 181 | GRAPH | 24000 | 396900 | tc_shorted_cycle_4800_4 |
s4k8_5 | 182 | GRAPH | 28800 | 476280 | tc_shorted_cycle_4800_5 |
s6k3_1 | 183 | GRAPH | 12600 | 209760 | tc_shorted_cycle_6300_1 |
s6k3_2 | 184 | GRAPH | 18900 | 314640 | tc_shorted_cycle_6300_2 |
s6k3_4 | 185 | GRAPH | 31500 | 524400 | tc_shorted_cycle_6300_4 |
s6k3_5 | 186 | GRAPH | 37800 | 629280 | tc_shorted_cycle_6300_5 |
s7k2_1 | 187 | GRAPH | 14400 | 240360 | tc_shorted_cycle_7200_1 |
s7k2_2 | 188 | GRAPH | 21600 | 360540 | tc_shorted_cycle_7200_2 |
s7k2_4 | 189 | GRAPH | 36000 | 600900 | tc_shorted_cycle_7200_4 |
s7k2_5 | 190 | GRAPH | 43200 | 721080 | tc_shorted_cycle_7200_5 |
s9k_1 | 191 | GRAPH | 18000 | 301560 | tc_shorted_cycle_9000_1 |
s9k_2 | 192 | GRAPH | 27000 | 452340 | tc_shorted_cycle_9000_2 |
s9k_4 | 193 | GRAPH | 45000 | 753900 | tc_shorted_cycle_9000_4 |
s9k_5 | 194 | GRAPH | 54000 | 904680 | tc_shorted_cycle_9000_5 |
s12k_1 | 195 | GRAPH | 24000 | 411560 | tc_shorted_cycle_12000_1 |
s12k_2 | 196 | GRAPH | 36000 | 617340 | tc_shorted_cycle_12000_2 |
s12k_4 | 197 | GRAPH | 60000 | 1028900 | tc_shorted_cycle_12000_4 |
s12k_5 | 198 | GRAPH | 72000 | 1234680 | tc_shorted_cycle_12000_5 |
s13k5_1 | 199 | GRAPH | 27000 | 468560 | tc_shorted_cycle_13500_1 |
s13k5_2 | 200 | GRAPH | 40500 | 702840 | tc_shorted_cycle_13500_2 |
s13k5_4 | 201 | GRAPH | 67500 | 1171400 | tc_shorted_cycle_13500_4 |
s13k5_5 | 202 | GRAPH | 81000 | 1405680 | tc_shorted_cycle_13500_5 |
s15k9_1 | 203 | GRAPH | 31800 | 559760 | tc_shorted_cycle_15900_1 |
s15k9_2 | 204 | GRAPH | 47700 | 839640 | tc_shorted_cycle_15900_2 |
s15k9_4 | 205 | GRAPH | 79500 | 1399400 | tc_shorted_cycle_15900_4 |
s15k9_5 | 206 | GRAPH | 95400 | 1679280 | tc_shorted_cycle_15900_5 |
s18k6_1 | 207 | GRAPH | 37200 | 662360 | tc_shorted_cycle_18600_1 |
s18k6_2 | 208 | GRAPH | 55800 | 993540 | tc_shorted_cycle_18600_2 |
s18k6_4 | 209 | GRAPH | 93000 | 1655900 | tc_shorted_cycle_18600_4 |
s18k6_5 | 210 | GRAPH | 111600 | 1987080 | tc_shorted_cycle_18600_5 |
p1k | 221 | GRAPH | 999 | 14767 | tc_path_1000 |
p2k | 222 | GRAPH | 1999 | 31766 | tc_path_2000 |
p3k | 223 | GRAPH | 2999 | 48766 | tc_path_3000 |
p4k | 224 | GRAPH | 3999 | 65766 | tc_path_4000 |
p14k1 | 225 | GRAPH | 14099 | 245665 | tc_path_14100 |
p20k | 226 | GRAPH | 19999 | 357765 | tc_path_20000 |
p24k5 | 227 | GRAPH | 24499 | 443265 | tc_path_24500 |
p28k3 | 228 | GRAPH | 28299 | 515465 | tc_path_28300 |
p31k6 | 229 | GRAPH | 31599 | 578165 | tc_path_31600 |
p34k6 | 230 | GRAPH | 34599 | 635165 | tc_path_34600 |
p37k4 | 231 | GRAPH | 37399 | 688365 | tc_path_37400 |
p40k | 232 | GRAPH | 39999 | 737765 | tc_path_40000 |
p42k4 | 233 | GRAPH | 42399 | 783365 | tc_path_42400 |
p44k7 | 234 | GRAPH | 44699 | 827065 | tc_path_44700 |
m4_2ki | 241 | GRAPH | 6144 | 103341 | tc_multipath_4_2048 |
m16_512 | 242 | GRAPH | 7680 | 128966 | tc_multipath_16_512 |
m64_128 | 243 | GRAPH | 8064 | 135110 | tc_multipath_64_128 |
m256_32 | 244 | GRAPH | 8160 | 136579 | tc_multipath_256_32 |
m1ki_8 | 245 | GRAPH | 8184 | 136938 | tc_multipath_1024_8 |
m4ki_2 | 246 | GRAPH | 8190 | 137022 | tc_multipath_4096_2 |
b17 | 251 | GRAPH | 131070 | 2488088 | tc_bintree_17 |
b18 | 252 | GRAPH | 262142 | 5171672 | tc_bintree_18 |
b19 | 253 | GRAPH | 524286 | 10676696 | tc_bintree_19 |
b20 | 254 | GRAPH | 1048574 | 21735320 | tc_bintree_20 |
b21 | 255 | GRAPH | 2097150 | 44901144 | tc_bintree_21 |
b22 | 256 | GRAPH | 4194302 | 93135640 | tc_bintree_22 |
b23 | 257 | GRAPH | 8388606 | 189604632 | tc_bintree_23 |
b24 | 258 | GRAPH | 16777214 | 389319832 | tc_bintree_24 |
b25 | 259 | GRAPH | 33554430 | 805527448 | tc_bintree_25 |
b26 | 260 | GRAPH | 67108862 | 1644388248 | tc_bintree_26 |
y500_4k | 271 | GRAPH | 4499 | 74378 | tc_y_500_4000 |
y500_8k | 272 | GRAPH | 8499 | 142378 | tc_y_500_8000 |
y1k_4k | 273 | GRAPH | 4999 | 83876 | tc_y_1000_4000 |
y1k_8k | 274 | GRAPH | 8999 | 151876 | tc_y_1000_8000 |
y1k_15k | 275 | GRAPH | 15999 | 282877 | tc_y_1000_15000 |
y1k_25k | 276 | GRAPH | 25999 | 472877 | tc_y_1000_25000 |
y1k_35k | 277 | GRAPH | 35999 | 662877 | tc_y_1000_35000 |
y5k_15k | 278 | GRAPH | 19999 | 358877 | tc_y_5000_15000 |
y5k_25k | 279 | GRAPH | 29999 | 548877 | tc_y_5000_25000 |
y5k_35k | 280 | GRAPH | 39999 | 738877 | tc_y_5000_35000 |
y10k_15k | 281 | GRAPH | 24999 | 463875 | tc_y_10000_15000 |
y10k_25k | 282 | GRAPH | 34999 | 653875 | tc_y_10000_25000 |
y10k_35k | 283 | GRAPH | 44999 | 843875 | tc_y_10000_35000 |
u1k_50k | 301 | GRAPH | 50000 | 739011 | tc_uniform_1000_50000 |
u1k_125k | 302 | GRAPH | 125000 | 1847511 | tc_uniform_1000_125000 |
u1k_250k | 303 | GRAPH | 250000 | 3695016 | tc_uniform_1000_250000 |
u2k_200k | 304 | GRAPH | 200000 | 3178014 | tc_uniform_2000_200000 |
u2k_250k | 305 | GRAPH | 250000 | 3972534 | tc_uniform_2000_250000 |
u2k_500k | 306 | GRAPH | 500000 | 7944982 | tc_uniform_2000_500000 |
u2k_1m | 307 | GRAPH | 1000000 | 15890006 | tc_uniform_2000_1000000 |
a1k_50k | 311 | GRAPH | 50000 | 738966 | tc_acyc_1000_50000 |
a1k_125k | 312 | GRAPH | 125000 | 1847531 | tc_acyc_1000_125000 |
a1k_250k | 313 | GRAPH | 250000 | 3695001 | tc_acyc_1000_250000 |
a2k_200k | 314 | GRAPH | 200000 | 3177896 | tc_acyc_2000_200000 |
a2k_500k | 315 | GRAPH | 500000 | 7944738 | tc_acyc_2000_500000 |
a2k_1m | 316 | GRAPH | 1000000 | 15890152 | tc_acyc_2000_1000000 |
f1 | 401 | GRAPH | 50000 | 689318 | tc_d1000_parsize50000_xsb_nocyc |
f2 | 402 | GRAPH | 50000 | 689318 | tc_d1000_parsize50000_xsb_cyc |
f3 | 403 | GRAPH | 250000 | 3446506 | tc_d1000_parsize250000_xsb_nocyc |
f4 | 404 | GRAPH | 250000 | 3446506 | tc_d1000_parsize250000_xsb_cyc |
f5 | 405 | GRAPH | 500000 | 6893031 | tc_d1000_parsize500000_xsb_nocyc |
f6 | 406 | GRAPH | 500000 | 6893031 | tc_d1000_parsize500000_xsb_cyc |
f7 | 407 | GRAPH | 500000 | 7446477 | tc_d2000_parsize500000_xsb_nocyc |
f8 | 408 | GRAPH | 500000 | 7446477 | tc_d2000_parsize500000_xsb_cyc |
f9 | 409 | GRAPH | 1000000 | 14892982 | tc_d2000_parsize1000000_xsb_nocyc |
f10 | 410 | GRAPH | 1000000 | 14892982 | tc_d2000_parsize1000000_xsb_cyc |
10k | 501 | JOIN1 | 50000 | 639300 | d1000_relsize10000_xsb_cyc |
50k | 502 | JOIN1 | 250000 | 3196504 | d1000_relsize50000_xsb_cyc |
250k | 503 | JOIN1 | 1250000 | 15982480 | d1000_relsize250000_xsb_cyc |
std | 601 | DBLP | 2437867 | 128444409 | dblp |
big | 602 | DBLP | 50221773 | 3082908204 | dblp_big |
wdat | 701 | WINE | 654 | 26386 | wine_data |
GRAPH_TYPES
: For Classifying Graphs (A Subclass of Input Files)A subclass of the input data files are graphs. This table defines types of graphs that can be used e.g. for subheadlines in the list of data files.
The current list of graph types is:
GTYPE | SEQ_NO | NAME |
---|---|---|
K | 1 | Complete Graph (with loops) |
T | 2 | Total Order Graph (i.e. maximal acyclic graph) |
C | 3 | Cycle Graph |
S | 4 | Cycle Graph with Shortcuts |
P | 5 | Path |
M | 6 | Multi-Path |
B | 7 | Binary Tree |
Y | 8 | Y-Graph (many nodes pointing to first node of path) |
U | 9 | Random Graph with Uniform Node Degrees |
A | 10 | Random Acyclic Graph with Uniform Node Degrees |
GRAPH_DATA
: More Data on Data Files That are Directed GraphsA subset of the data files describe directed graphs (the data files with ITYPE='GRAPH'). This table contains additional attributes for those data file (properties of the graph). We need some of the attributes for out work on runtime estimation. We computed these numbers from the data files also as an additional safety measure to find possible errors in the graph-generating program. For instance, the program used in the OpenRuleBench did produce duplicate edges for acyclic graphs.
The current contents of this table is:
FILE_ID | GTYPE | PARAM_1 | PARAM_2 | NUM_NODES | NUM_EDGES | DUP_EDGES | LOOPS | MIN_IN_DEGREE | MAX_IN_DEGREE | MIN_OUT_DEGREE | MAX_OUT_DEGREE | CYCLES | TC_SIZE | NUM_ITER | COST |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
k500 | K | 500 | 500 | 250000 | 0 | Y | 500 | 500 | 500 | 500 | Y | 250000 | 1 | 125250000 | |
k1k | K | 1000 | 1000 | 1000000 | 0 | Y | 1000 | 1000 | 1000 | 1000 | Y | 1000000 | 1 | 1001000000 | |
k2k | K | 2000 | 2000 | 4000000 | 0 | Y | 2000 | 2000 | 2000 | 2000 | Y | 4000000 | 1 | 8004000000 | |
t500 | T | 500 | 500 | 124750 | 0 | N | 0 | 499 | 0 | 499 | N | 124750 | 1 | 62126000 | |
t1k | T | 1000 | 1000 | 499500 | 0 | N | 0 | 999 | 0 | 999 | N | 499500 | 1 | 166666500 | |
t2k | T | 2000 | 2000 | 1999000 | 0 | N | 0 | 1999 | 0 | 1999 | N | 1999000 | 1 | 1333333000 | |
c1k | C | 1000 | 1000 | 1000 | 0 | N | 1 | 1 | 1 | 1 | Y | 1000000 | 1000 | 1001000 | |
c2k | C | 2000 | 2000 | 2000 | 0 | N | 1 | 1 | 1 | 1 | Y | 4000000 | 2000 | 4002000 | |
c3k | C | 3000 | 3000 | 3000 | 0 | N | 1 | 1 | 1 | 1 | Y | 9000000 | 3000 | 9003000 | |
c4k | C | 4000 | 4000 | 4000 | 0 | N | 1 | 1 | 1 | 1 | Y | 16000000 | 4000 | 16004000 | |
s1k_1 | S | 1000 | 1 | 1000 | 2000 | 0 | N | 2 | 2 | 2 | 2 | Y | 1000000 | 500 | 2002000 |
s1k_2 | S | 1000 | 2 | 1000 | 3000 | 0 | N | 3 | 3 | 3 | 3 | Y | 1000000 | 201 | 3003000 |
s1k_3 | S | 1000 | 3 | 1000 | 4000 | 0 | N | 4 | 4 | 4 | 4 | Y | 1000000 | 250 | 4004000 |
s1k_4 | S | 1000 | 4 | 1000 | 5000 | 0 | N | 5 | 5 | 5 | 5 | Y | 1000000 | 200 | 5005000 |
s1k_5 | S | 1000 | 5 | 1000 | 6000 | 0 | N | 6 | 6 | 6 | 6 | Y | 1000000 | 42 | 6006000 |
s2k_1 | S | 2000 | 1 | 2000 | 4000 | 0 | N | 2 | 2 | 2 | 2 | Y | 4000000 | 1000 | 8004000 |
s2k_2 | S | 2000 | 2 | 2000 | 6000 | 0 | N | 3 | 3 | 3 | 3 | Y | 4000000 | 288 | 12006000 |
s2k_3 | S | 2000 | 3 | 2000 | 8000 | 0 | N | 4 | 4 | 4 | 4 | Y | 4000000 | 500 | 16008000 |
s2k_4 | S | 2000 | 4 | 2000 | 10000 | 0 | N | 5 | 5 | 5 | 5 | Y | 4000000 | 400 | 20010000 |
s2k_5 | S | 2000 | 5 | 2000 | 12000 | 0 | N | 6 | 6 | 6 | 6 | Y | 4000000 | 127 | 24012000 |
p1k | P | 1000 | 1000 | 999 | 0 | N | 0 | 1 | 0 | 1 | N | 499500 | 999 | 499500 | |
p2k | P | 2000 | 2000 | 1999 | 0 | N | 0 | 1 | 0 | 1 | N | 1999000 | 1999 | 1999000 | |
p3k | P | 3000 | 3000 | 2999 | 0 | N | 0 | 1 | 0 | 1 | N | 4498500 | 2999 | 4498500 | |
p4k | P | 4000 | 4000 | 3999 | 0 | N | 0 | 1 | 0 | 1 | N | 7998000 | 3999 | 7998000 | |
m4_2ki | M | 4 | 2048 | 8192 | 6144 | 0 | N | 0 | 1 | 0 | 1 | N | 12288 | 3 | 12288 |
m16_512 | M | 16 | 512 | 8192 | 7680 | 0 | N | 0 | 1 | 0 | 1 | N | 61440 | 15 | 61440 |
m64_128 | M | 64 | 128 | 8192 | 8064 | 0 | N | 0 | 1 | 0 | 1 | N | 258048 | 63 | 258048 |
m256_32 | M | 256 | 32 | 8192 | 8160 | 0 | N | 0 | 1 | 0 | 1 | N | 1044480 | 255 | 1044480 |
m1ki_8 | M | 1024 | 8 | 8192 | 8184 | 0 | N | 0 | 1 | 0 | 1 | N | 4190208 | 1023 | 4190208 |
m4ki_2 | M | 4096 | 2 | 8192 | 8190 | 0 | N | 0 | 1 | 0 | 1 | N | 16773120 | 4095 | 16773120 |
b17 | B | 17 | 131071 | 131070 | 0 | N | 0 | 1 | 0 | 2 | N | 1966082 | 16 | 1966082 | |
b18 | B | 18 | 262143 | 262142 | 0 | N | 0 | 1 | 0 | 2 | N | 4194306 | 17 | 4194306 | |
b19 | B | 19 | 524287 | 524286 | 0 | N | 0 | 1 | 0 | 2 | N | 8912898 | 18 | 8912898 | |
y500_4k | Y | 500 | 4000 | 4500 | 4499 | 0 | N | 0 | 500 | 0 | 1 | N | 9998000 | 4000 | 9998000 |
y500_8k | Y | 500 | 8000 | 8500 | 8499 | 0 | N | 0 | 500 | 0 | 1 | N | 35996000 | 8000 | 35996000 |
y1k_4k | Y | 1000 | 4000 | 5000 | 4999 | 0 | N | 0 | 1000 | 0 | 1 | N | 11998000 | 4000 | 11998000 |
y1k_8k | Y | 1000 | 8000 | 9000 | 8999 | 0 | N | 0 | 1000 | 0 | 1 | N | 39996000 | 8000 | 39996000 |
u1k_50k | U | 1000 | 50000 | 1000 | 50000 | 0 | N | 46 | 51 | 46 | 51 | Y | 1000000 | 3 | 50050000 |
u1k_125k | U | 1000 | 125000 | 1000 | 125000 | 0 | N | 122 | 127 | 122 | 127 | Y | 1000000 | 2 | 125125000 |
u1k_250k | U | 1000 | 250000 | 1000 | 250000 | 0 | N | 248 | 251 | 248 | 251 | Y | 1000000 | 2 | 250250000 |
u2k_200k | U | 2000 | 200000 | 2000 | 200000 | 0 | N | 98 | 101 | 98 | 101 | Y | 4000000 | 3 | 400200000 |
u2k_500k | U | 2000 | 500000 | 2000 | 500000 | 0 | N | 248 | 251 | 248 | 251 | Y | 4000000 | 2 | 1000500000 |
u2k_1m | U | 2000 | 1000000 | 2000 | 1000000 | 0 | N | 499 | 502 | 499 | 502 | Y | 4000000 | 2 | 2001000000 |
a1k_50k | A | 1000 | 50000 | 1000 | 50000 | 0 | N | 0 | 102 | 0 | 102 | N | 471621 | 8 | 15269270 |
a1k_125k | A | 1000 | 125000 | 1000 | 125000 | 0 | N | 0 | 250 | 0 | 250 | N | 492680 | 5 | 40853746 |
a1k_250k | A | 1000 | 250000 | 1000 | 250000 | 0 | N | 0 | 500 | 0 | 501 | N | 497763 | 4 | 82941061 |
a2k_200k | A | 2000 | 200000 | 2000 | 200000 | 0 | N | 0 | 201 | 0 | 201 | N | 1944729 | 8 | 128420988 |
a2k_500k | A | 2000 | 500000 | 2000 | 500000 | 0 | N | 0 | 500 | 0 | 500 | N | 1985321 | 6 | 330666855 |
a2k_1m | A | 2000 | 1000000 | 2000 | 1000000 | 0 | N | 0 | 1000 | 0 | 1000 | N | 1995698 | 4 | 666033613 |
BENCH_RUN
: Results of Benchmark RunsThis is finally the main table, in which the runtime of benchmark runs is stored.
There are three sources for timing information: Measurements done by the program itself, external measurements done with /usr/bin/time
, and measurements through the /proc/[procid]/stats
virtual file system.
In order to distinguish between the time needed for loading the data and for executing the query, we printed these times from the program that runs the benchmark (where possible). For instance, Prolog systems usually have a timing function. In addition, we printed the total runtime as computed by the program itself. This should normally be simply the sum of load time and execution time. It differs from the externally measured time by not including the time needed for compiling the program. If the program can compute CPU time and real time (wallclock time), we selected CPU time. If that is not available, we take wallclock time. These measurements are anyway not used for comparisons between the programs. They are used to analyze where the runtime is used up. For instance, XSB was earlier quite slow in loading the data (this was improved with load_dync), but fast in query execution.
The CPU time in user mode (USER_TIME), the CPU time in system/kernel mode (SYS_TIME) and the REAL_TIME are measured externally with /usr/bin/time. They should be comparable for all systems.
We also measured 'maximum resident set size' with /usr/bin/time. This is the 'high water mark' for the amount of memory a process uses and that is present in real RAM (e.g., not swapped out). It includes shared libraries (as far as actually used).
For server-based systems such as MariaDB and PostgreSQL the measurements are more difficult, because we cannot use /usr/bin/time for the server process. The startup time for the server process is not included.
All times are stored in seconds.
ESTIMATE_PAR
: Experimental Table for Runtime Estimation ParametersThese are parameters for a formula that tries to generate an approximation of the measured runtime from characteristics of the input data. Currently, it is only used for the tcff benchmark. Note that more than one set of parameters can be stored for the same program. It is possible to compare the proposed parameters in this way.
DUMMY_TAB
: Dummy Table with Exactly One RowThis is used in some views (to write a FROM clause).
The table has only one column:
The table contents is:
ATTR |
---|
0 |
Sorry, currently we can only list the views. More documentation will be added later.
There is one view for each tested program that lists the benchmark results for that program.
These views are intended to help finding/checking the best implementation of a benchmark for a program, e.g. which indexes should be chosen. The views consider only the runtimes of the current program version (installation) on the standard machine.
We use these views for getting the data for plots of performance data vs. cost measures, in particular the run time vs. the number of applicable rule instances. The results are available in csv and tsv formats. These views look only at the current program version (installation) on the standard machine, with the best implementation of the benchmark for the system (best indexes, options).
The following views are for comparing system runtimes for the TCFF benchmark. These views look only at the current program version (installation) on the standard machine, with the best implementation of the benchmark for the system (best indexes, options).