• Intro
• Benchmark Database
• tcff benchmark
  Soufflé

  SQLite3 PostgreSQL MariaDB

  xsb YAProlog SWI-Prolog

  Apache Jena Neo4j
• sgff benchmark
  Soufflé

  SQLite3 PostgreSQL MariaDB

  xsb YAProlog

  Neo4j

This is a database to store benchmark results for programs that answer queries from large input data files, e.g. logic programming systems, deductive databases, relational databases, and graph databases/RDF stores.

The benchmarks were inspired by the OpenRuleBench, see also OpenRuleBench Download.

However, the OpenRuleBench scripts do not support executing the same benchmark for the same program multiple times. This is important to get reliable data, especially when one is also interested in slight differences. Since we are developing our own deductive database based on the Push Method, we are interested even in minor runtime improvements. Furthermore, it is very important for us to compare different versions of our software, or the same version with different compiler settings. Moreover, we run benchmarks on different machines with different operating systems. The OpenRuleBench scripts have no support for this (they were developed with a different application in mind, namely to get an overview of the performance of rule-based systems for semantic web applications).

One soon loses the overview if one keeps the measurements in simple text files. Therefore we developed a relational database to store and evaluate runtime measurements (and memory consumption, too).

A problem for fair benchmarks is to find the best settings for each systems for a given benchmark, e.g. which indexes should be used for a relational database. As we understood the OpenRuleBench, the developers of each system were ask to define the optimal configuration of their system for each benchmark. However, we did find some non-optimal configurations.

In order to give some confidence in our benchmark results, we tried several different settings for the same program and benchmark, and store the results in our benchmark database. In the main comparison, we use only the best settings, but in the database it can be seen what we tried and what were the results. Sometimes, it is not clear, what are the best settings, because most benchmarks are run on different input data files, and sometimes the best settings are different for different data files. However, for the main comparison of the systems, we require that for each benchmark there is only one setting manually selected as best. But our database contains all measurements, and one can check the alternatives.

Of course, we would be grateful for settings recommended by the system developers or experts for one of the systems we tested in our benchmarks.

Another design goal of our database was to be able to generate results in different formats (e.g. HTML, LaTeX, csv). In particular, we want to be able to generate a web site with the benchmark results more or less automatically (so that there are no errors in copying the data, and updates are simple).

Tables of the Relational Database

MACHINE: List of Machines on which Benchmarks are Executed

This table contains some data about the machines on which benchmarks were run. One machine is marked as the stardard machine by the value 'S' in column STATUS. However, the database permits to store and evaluate benchmark measurements for different machines with different architectures and operating systems.

• MACHINE
  • Short name (ID) of the machine, e.g. 'dbs1'.
  • VARCHAR(20) NOT NULL, PRIMARY KEY
• SEQ_NO
  • Sequential number for sorting a list of machines, starting at 1.
  • NUMERIC(2) NOT NULL, UNIQUE, CHECK(SEQ_NO > 0),
• STATUS
  • The standard machine is marked by 'S' in this column, all other machines contain a space in this column. Many views for comparing benchmark results look only at measurements for the standard machine. Only one row of the table should have STATUS='S'. In future, there many be more STATUS values, e.g. for comparing the standard machine to an alternative machine.
  • CHAR(1) NOT NULL, CHECK(STATUS IN ('S', ' '))
• MODEL
  • The model name of the machine. E.g. the main machine that we use for the benchmarks is a 'HP Blade Server, HPE BL460c Gen9'.
  • VARCHAR(60) NULL
• YEAR
  • The year when the machine was bought. E.g. out main test machine was bought in October 2016, so this column contains 2016.
  • NUMERIC(4) NULL, CHECK(YEAR >= 2000)
• MONTH
  • The month when the machine was bought. For our machine, bought in October 2016, this contains 10.
  • NUMERIC(2) NULL CHECK(MONTH >= 1 AND MONTH <= 12)
• PRICE
  • The price of the machine in euros. E.g. the price of our main test machine was 3319 euros.
  • NUMERIC(7) NULL CHECK(PRICE >= 0)
• NUM_CPUS
  • The number of CPUs in the machine. E.g. the main machine we use for the benchmarks contains two CPUs.
  • NUMERIC(4) NOT NULL, CHECK(NUM_CPUS > 0)
• CPU
  • The name of the CPU, e.g. 'Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz'.
  • VARCHAR(60) NOT NULL
• CPU_CORES
  • The number of cores per CPU in the machine. E.g. the Intel Xeon E5-2630 has 10 cores.
  • NUMERIC(4) NULL, CHECK(CPU_CORES > 0)
• CPU_THREADS
  • Because of hyperthreading, the number of hardware-supported threads might be larger than the number of cores. This column contains the total number of hardware threads per CPU. E.g. the Intel Xeon E5-2630 supports 20 concurrent threads.
  • NUMERIC(4) NULL, CHECK(CPU_THREADS > 0)
• GHZ
  • The clock frequency of the CPU in GHz, e.g. 2.2 in case of the Intel Xeon E5-2630.
  • NUMERIC(5,2) NULL
• ARCH
  • The architecture of the CPU, e.g. 'x86_64'.
  • VARCHAR(20) NULL
• RAM
  • The main memory size (RAM) in Gigabytes. Our main test machine has 64 GB.
  • NUMERIC(8,3) NOT NULL, CHECK(RAM > 0)
• NUM_DISKS
  • The number of disks of the machine.
  • NUMERIC(4) NULL, CHECK(NUM_DISKS > 0)
• DISK
  • The type of disk of the machine.
  • VARCHAR(40) NULL
• SSD
  • If the disk that used for reading data of the benchmarks is solid state disk, then this column is 'Y'. If it is a standard magnetic disk, the value of this column is 'N'.
  • CHAR(1) NULL, CHECK(SSD IN ('Y','N'))
• OS
  • The operating system running on the machine, e.g. 'Debian GNU/Linux 8.10 (jessie)'.
  • VARCHAR(40) NULL
• OS_CORE
  • In case of a Linux operting system, the version of the core, e.g. 'Linux 3.16.0-5-amd64 x86_64'.
  • VARCHAR(40) NULL
  • Note that due to updates, this might not be current. The above value was current when we started the benchmarks in May 2018. It gives some impression of the machine and its operating system. We are planning to create a table MACHINE_HISTORY that contains data of a machine that might change.

INPUT_TYPE: List of Input Data (Problem Instance) Types

We have many input data files for the transitive closure benchmark, but since they contain general graph data, they can be used for different benchmarks. The entries in this table are used to classify data files. For instance, there are also data files for the DBLP benchmark and for the Join1 benchmark, and so on.

• ITYPE
  • A short ID for a class of data files, e.g. 'GRAPH'.
  • VARCHAR(10) NOT NULL, PRIMARY KEY
• DESCRIPTION
  • A description of this class of data files, e.g. 'Directed graph with edges stored in predicate par'.
  • VARCHAR(80) NOT NULL

The currently defined input types are:

BENCHMARK: List of Benchmarks

This is the list of all benchmarks, e.g. 'tcff' is a benchmark identifier in column BENCH. It corresponds to the computation of the transitive closure with query 'tc(X,Y)', i.e. with both arguments free. Another benchmark, 'tcbf', is the computation of transitive closure with query 'tc(1,X)', i.e. the first argument bound (to a given value), and the second argument free.

• BENCH
  • This is a short ID of the benchmark, e.g. 'tcff'.
  • VARCHAR(20) NOT NULL, PRIMARY KEY
• SEQ_NO
  • The number in this column is used for ordering benchmarks, when a web page with all benchmark results is generated.
  • NUMERIC(2) NOT NULL, CHECK(SEQ_NO >= 0), UNIQUE
• ITYPE
  • This is the class of data files that can be used as input for the benchmark.
  • VARCHAR(10) NULL, FOREIGN KEY (ITYPE) REFERENCES INPUT_TYPE
• HEADLINE
  • This is a headline to be printed in a web page with results for this benchmark.
  • VARCHAR(80) NOT NULL

The currently defined benchmarks are:

PROG_TYPES: Program Types

This table defines broad classes of programs, e.g. Prolog systems and Relational Databases. We use it for generating subheadlines in the list of all tested programs.

• PTYPE
  • A short ID of the class of programs, e.g. 'Prolog'.
  • VARCHAR(10) NOT NULL, PRIMARY KEY
• SEQ_NO
  • A number used to define the output sequence.
  • NUMERIC(2) NOT NULL, UNIQUE, CHECK(SEQ_NO >= 0)
• HEADLINE
  • The subheadline for this program class in the list of programs.
  • VARCHAR(80) NOT NULL,

The current entries are:

PROGRAM: List of Programs to be Tested/Compared

• PROG
  • A short name (ID) of the program, e.g. 'psql'.
  • VARCHAR(20) NOT NULL PRIMARY KEY
• SEQ_NO
  • A number used for sorting the list of programs.
  • NUMERIC(2) NOT NULL CHECK(SEQ_NO >= 0) UNIQUE
• FULL_NAME
  • The full name of the program, to be used in output. E.g. 'PostgreSQL'.
  • VARCHAR(40) NOT NULL
• PTYPE
  • Type of the program for structured lists of all tested programs. E.g. 'RDB'. This is a foreign key referencing PROG_TYPE.
  • VARCHAR(10) NOT NULL REFERENCES PROG_TYPE
• URL
  • A web address where one can download the program or get more information about it.
  • VARCHAR(80) NULL
• REMARK
  • An optional remark (not yet used).
  • VARCHAR(800) NULL

The current list of programs is:

PROG_VER: Program Versions

This table contains information about program versions. Note that this table describes a version of the source code. PROG_INSTALL (below) describes a version of the program binary, which also depends on the machine, the compiler, compiler settings, and configuration settings.

• PROG
  • The ID of the program (see table PROGRAM), e.g. 'xsb'.
  • VARCHAR(20) NOT NULL REFERENCES PROGRAM
• VER
  • A version number (any string), e.g. '3.7.'. Together with the program ID, this forms the primary key of this table.
  • VARCHAR(20) NOT NULL,
  • PRIMARY KEY (PROG, VER)
• VER_ADDTEXT
  • Some program versions have code names or other additions. On output, this should be concatenated with the value of the attribute VER to get the full name of the program version. However, it is not necessary for identifying the program version. E.g., for XSB 3.7, this attribute has the value ' (Clan MacGregor)'. Note the leading space (VER and VER_ADDTEXT are simply concatenated).
  • VARCHAR(80) NOT NULL,
• VER_DATE
  • The release date of the program version. It must be a legal date value of the used database system. For many systems, the ISO format 'YYYY-MM-DD' works. E.g., XSB 3.7 was released on July 6, 2016: '2016-07-06'.
  • DATE NULL
• REMARK
  • This optional column can be used to store additional information about the program version (e.g., major changes or new features).
  • VARCHAR(800) NULL

PROG_INSTALL: Table for Installed Program Versions on Machines

This table lists program versions that are or were installed on a test machine. They are identified here by a number INSTALL that is unique together with the program name and the machine. These three elements (program, installation ID, and machine) are also encoded in the file names with the test results.

Whereas PROG_VER describes a version of the source code, PROG_INSTALL describes a binary program. For instance, if the same source code is compiled with g++ and with clang, these are two different entries in PROG_INSTALL, both referring to the same entry in PROG_VER. Also different configuration settings may give different binaries, and therefore different entries in PROG_INSTALL.

I try to make sure that the same version of a program gets the same INSTALL number even on different machines, but the database schema does not require this (and it is impossible when there are really different installations of the same program version).

Only program installations that are listed here can appear in the test runs.

This table also define a current version (more precisely, installation) of a program on a machine (by STATUS='C'). Some views show only results for the current version. It is also possible to define a previous version (STATUS='P') as reference, if the result of a program update should be evaluated. Finally, one can mark an alternative version (STATUS='A'), e.g. compiled with a different compiler or compiler options, which can also be used for comparison.

• PROG
  • The ID of the program, see table PROGRAM. E.g. 'psql'.
  • VARCHAR(20) NOT NULL
• VER
  • VARCHAR(20) NOT NULL
  • FOREIGN KEY (PROG, VER) REFERENCES PROG_VER,
• MACHINE
  • The ID of the machine on which the program is installed (see table MACHINE). E.g. 'dbs1'.
  • VARCHAR(20) NOT NULL REFERENCES MACHINE
• INSTALL
  • A number that uniquely identifies the program installation together with the program ID (column PROG) and the machine (MACHINE). Since the version VER can be an arbitrary string, it is useful to have a single small number instead. In addition, this is necessary, because there might be different installations of the same program version (with different configuration settings or compilers).
  • NUMERIC(4) NOT NULL
  • PRIMARY KEY (PROG, INSTALL, MACHINE),
• COMPILER
  • Sometimes we want to check whether different compilers (e.g. g++ or clang) or different compiler options (e.g. -O2 vs. -O3) produce faster code. This column can be used to specify the compiler. However, one can leave it blank if one is not interested in this information, or a standard compiler defined in the installation script was used.
  • VARCHAR(80) NOT NULL
• STATUS
  • This column is used to mark a current version of the program with the value 'C'. Many views for selecting benchmark results look only at the current version on the standard machine. There are other options defined below. They could be used in views, e.g. for comparing the performance of the current version with the previous version. Only one installation of a program on a machine should be marked as current version.
  • CHAR(1) NOT NULL CHECK(STATUS IN ('C','P','A',' '))
  • The meaning of the status values are:
    • 'C': Current version
    • 'P': Previous version
    • 'A': Alternate version
    • ' ': Any other version
• REMARK
  • VARCHAR(800) NULL

BENCH_IMPL: Implementation Variants of Benchmarks

In general, there are different option settings (e.g. index selections) that should be tested for the same program to execute a given benchmark as fast as possible. We call such a combination of benchmark, program and option settings a benchmark implementation. STATUS='B' marks the best implementation that is used in some views to compare different programs. This is selected manually, but there are also views that help to find the best implementation. For instance, we tested the InnoDB und the Memory storage engine for MariaDB. It depended on the input data, which was faster. We selected InnoDB in this table for the main comparison between the systems, but the benchmark database contains results for both.

• PROG
  • ID of the program that executes the benchmark, see table PROGRAM. E.g. 'psql'.
  • VARCHAR(20) NOT NULL REFERENCES PROGRAM
• BENCH
  • ID of the benchmark, see table BENCHMARK. E.g. 'tcff'.
  • VARCHAR(20) NOT NULL REFERENCES BENCHMARK
• IMPL
  • This value is used to distinguish between several implementations of the same benchmark with the same program. For instance, the implemenations on a relational database might differ in the indexes that have been created. If there is only one implementation, it is simplest to choose a single space. The program for loading benchmark runtimes from tsv-files will map a missing implementation to a space.
  • VARCHAR(20) NOT NULL
  • PRIMARY KEY (PROG, BENCH, IMPL),
• STATUS
  • CHAR(1) NOT NULL CHECK(STATUS IN ('B','A',' '))
  • Meaning of the STATUS values:
    • STATUS='B' marks the implementation variant (option settings) that is considered best and is used in comparisons between systems. One can use the views BEST_IMPL_REAL and BEST_IMPL_CPU to check whether these implementation variants are really best. Sometimes the best implementation depends on the specific data file, but it would somehow be unfair to adapt the options to the data file.
    • STATUS='A' marks an interesting alternative variants that appear in some result tables. There should be only one best version, but there can be several alterative versions.
    • STATUS=' ' means that the variant was tested in order to prove that the best version is really good, but it will not be listed in summary tables.
• DESCRIPTION
  • In this attribute, one would typically explain which indexes have been chosen, or other information on the specific benchmark implementation.
  • VARCHAR(80) NULL

The current list of benchmark implementations is:

DATA_FILE: Information about Data Files

This a list of data files that can be used as input for a benchmark. Each file has a type ITYPE, and in the table BENCHMARK (see above) it is listed what kind of input is required for a benchmark. For instance, we have many data files that define directed graphs with a predicate par (parent, OpenRuleBench used this name for the transitive closure benchmarks).

Different programs need different file formats, therefore each file is available in several formats such as + Prolog facts (with extension '.P') + Tab-Separated-Values (extension '.tsv') + Comma-separated Values (extension '.csv') + SQL INSERT Statements (extension '.sql') + RDF in Turtle Syntax (extension '.ttl') + JSON (extension '.json') + graphviz (extension '.dot')

The columns of the table are:

• FILE_ID
  • A short name (ID) that uniquely identifies the file. E.g. 'k1000' for the complete graph with 1000 nodes.
  • VARCHAR(20) NOT NULL PRIMARY KEY
• SEQ_NO
  • A number that defines a sort order for the files (or benchmark results for files).
  • NUMERIC(3) NOT NULL UNIQUE CHECK(SEQ_NO >= 0)
• ITYPE
  • The type of data in the file, see table INPUT_TYPE above. E.g. 'GRAPH'.
  • VARCHAR(10) NOT NULL REFERENCES INPUT_TYPE
• LINES
  • The number of lines in the file. Formally, the Prolog facts version is chosen to count the lines. But most other file formats will have the same number of lines. This is also the number of facts in the file (rows in the corresponding tables).
  • NUMERIC(12) NOT NULL CHECK(LINES >= 0)
• BYTES
  • This is the file size in bytes of the Prolog facts version. It should give an impression of the approximate file size in the other versions, too.
  • NUMERIC(12) NOT NULL CHECK(BYTES >= 0)
• FILENAME
  • This is the file name of the data file without the extension.
  • VARCHAR(40) NOT NULL

The current contents of this table is:

GRAPH_TYPES: For Classifying Graphs (A Subclass of Input Files)

A subclass of the input data files are graphs. This table defines types of graphs that can be used e.g. for subheadlines in the list of data files.

• GTYPE
  • A one-letter ID of the graph type. This is also the first letter of the FILE_ID of the graph (maybe with different case).
  • CHAR(1) NOT NULL PRIMARY KEY
• SEQ_NO
  • This defines an output order for the graph types.
  • NUMERIC(2) NOT NULL CHECK(SEQ_NO >= 0) UNIQUE
• NAME
  • Name of the graph type (possibly with short explanation).
  • VARCHAR(80) NOT NULL

The current list of graph types is:

GRAPH_DATA: More Data on Data Files That are Directed Graphs

A subset of the data files describe directed graphs (the data files with ITYPE='GRAPH'). This table contains additional attributes for those data file (properties of the graph). We need some of the attributes for out work on runtime estimation. We computed these numbers from the data files also as an additional safety measure to find possible errors in the graph-generating program. For instance, the program used in the OpenRuleBench did produce duplicate edges for acyclic graphs.

• FILE_ID
  • This identifies the data file. It is at the same time foreign key to DATA_FILE and primary key (this is a usual translation of subclasses).
  • VARCHAR(20) NOT NULL PRIMARY KEY REFERENCES DATA_FILE
• GTYPE
  • This is the type of the graph. E.g. all complete graphs have GTYPE='K'. See table GRAPH_TYPE.
  • CHAR(1) NOT NULL REFERENCES GRAPH_TYPE
• PARAM_1
  • For graphs that were generated with our graph generation program, this is the value of the first (and often only) parameter. E.g. for a complete graph with 1000 nodes, it will be 1000.
  • NUMERIC(12) NULL CHECK(PARAM_1 >= 0)
• PARAM_2
  • This is the value of the second parameter for the graph generation program (if applicable). E.g. for multi-paths, the first parameter is the length of the single paths, and the second parameter is the number of paths.
  • NUMERIC(12) NULL CHECK(PARAM_2 >= 0)
• NUM_NODES
  • The number of nodes in the graph.
  • NUMERIC(12) NOT NULL CHECK(NUM_NODES >= 0)
• NUM_EDGES
  • The number of edges in the graph.
  • NUMERIC(12) NOT NULL CHECK(NUM_EDGES >= 0)
• DUP_EDGES
  • If this is not 0, the graph contains duplicate edges, i.e. more than one edge between the same two nodes. Formally, the value of this attribute is the number of pairs (x,y) of nodes, such that there are at least two edges between x and y.
  • NUMERIC(12) NOT NULL CHECK(DUP_EDGES >= 0)
• LOOPS
  • This is 'Y' if the graph contains loops, i.e. edges of the form (x,x). It is 'N' otherwise.
  • CHAR(1) NOT NULL CHECK(LOOPS in ('Y','N'))
• MIN_IN_DEGREE
  • This is the minimal in-degree of the nodes, i.e. the minimum of the count of incoming edges over all nodes.
  • NUMERIC(12) NOT NULL CHECK(MIN_IN_DEGREE >= 0)
• MAX_IN_DEGREE
  • This is the maximal in-degree of the nodes, i.e. the minimum of the count of incoming edges over all nodes.
  • NUMERIC(12) NOT NULL CHECK(MAX_IN_DEGREE >= 0)
• MIN_OUT_DEGREE
  • This is the minimal out-degree of the nodes, i.e. the minimum of the count of outgoing edges over all nodes.
  • NUMERIC(12) NOT NULL CHECK(MIN_OUT_DEGREE >= 0)
• MAX_OUT_DEGREE
  • This is the maximal out-degree of the nodes, i.e. the maximum of the count of outgoing edges over all nodes.
  • NUMERIC(12) NOT NULL CHECK(MAX_OUT_DEGREE >= 0)
• CYCLES
  • This is 'Y' if the graph is cyclic, i.e. there is a node x such that there is a non-empty path starting at x and ending at x. It is 'N' for acyclic graphs.
  • CHAR(1) NOT NULL CHECK(CYCLES IN ('Y','N'))
• TC_SIZE
  • This is the size of the transitive closure of the graph, i.e. the number of node pairs (x,y) such that y is reachable from x via one or more edges.
  • NUMERIC(12) NOT NULL CHECK(TC_SIZE >= 0)
• NUM_ITER
  • This is the number of iterations that naive/seminaive evaluation would need to compute the transitive closure. I.e. it is the length of the longest path for which there is no shorter path between the same two nodes. In other words, it is the maximum of the distance (length of the shortest path) of all nodes x and y such that y is reachable from x. For connected graphs, this is known as the diameter of the graph.
  • NUMERIC(12) NOT NULL CHECK(NUM_ITER >= 0)
• COST
  • This is the number of applicable rule instances in the standard tail-recursive Prolog/Datalog program for computing the transitive closure (where 'applicable' means that the rule body is true in the minimal Herbrand model). One can compute this as the number of edges (for the rule tc(X,Y) <- par(X,Y)) plus the size of the join of the edge relation with the transitive closure relation (for the rule tc(X,Z) <- par(X,Y), tc(Y,Z)).
  • NUMERIC(12) NOT NULL CHECK(COST >= 0)

The current contents of this table is:

BENCH_RUN: Results of Benchmark Runs

This is finally the main table, in which the runtime of benchmark runs is stored.

There are three sources for timing information: Measurements done by the program itself, external measurements done with /usr/bin/time, and measurements through the /proc/[procid]/stats virtual file system.

In order to distinguish between the time needed for loading the data and for executing the query, we printed these times from the program that runs the benchmark (where possible). For instance, Prolog systems usually have a timing function. In addition, we printed the total runtime as computed by the program itself. This should normally be simply the sum of load time and execution time. It differs from the externally measured time by not including the time needed for compiling the program. If the program can compute CPU time and real time (wallclock time), we selected CPU time. If that is not available, we take wallclock time. These measurements are anyway not used for comparisons between the programs. They are used to analyze where the runtime is used up. For instance, XSB was earlier quite slow in loading the data (this was improved with load_dync), but fast in query execution.

The CPU time in user mode (USER_TIME), the CPU time in system/kernel mode (SYS_TIME) and the REAL_TIME are measured externally with /usr/bin/time. They should be comparable for all systems.

We also measured 'maximum resident set size' with /usr/bin/time. This is the 'high water mark' for the amount of memory a process uses and that is present in real RAM (e.g., not swapped out). It includes shared libraries (as far as actually used).

For server-based systems such as MariaDB and PostgreSQL the measurements are more difficult, because we cannot use /usr/bin/time for the server process. The startup time for the server process is not included.

All times are stored in seconds.

• RUN_DATE
  • The date on which the benchmark was run.
  • DATE NOT NULL
• RUN_NO
  • To distinguish runs of the same program on the same machine, a sequential number RUN_NO was added to the key.
  • NUMERIC(4) NOT NULL CHECK(RUN_NO > 0)
• PROG
  • The ID of the program that was used to execute the benchmark, e.g. 'xsb' (see table PROGRAM above).
  • VARCHAR(20) NOT NULL
• INSTALL
  • The installation number of the program on the machine (see table PROG_INSTALL above).
  • NUMERIC(4) NOT NULL
• MACHINE
  • The ID of the machine on which the benchmark was executed (see table MACHINE above).
  • VARCHAR(20) NOT NULL
  • PRIMARY KEY(RUN_DATE,RUN_NO,PROG,INSTALL,MACHINE)
  • FOREIGN KEY(PROG, INSTALL, MACHINE) REFERENCES PROG_INSTALL
• BENCH
  • The ID of the benchmark, e.g. 'tcff' (see table BENCHMARK above).
  • VARCHAR(20) NOT NULL
• IMPL
  • This identifies a concrete implementation of a benchmark on a program/system, e.g. one can distinguish different option settings or indexes in this way (see table BENCH_IMPL above).
  • VARCHAR(20) NOT NULL
  • FOREIGN KEY(PROG, BENCH, IMPL) REFERENCES BENCH_IMPL
• FILE_ID
  • The ID of the input file for which the benchmark was executed (see table DATA_FILE above).
  • VARCHAR(20) NOT NULL REFERENCES DATA_FILE
• WITH_OUTPUT
  • Since the result of some benchmarks is very large, we normally do not printed it in the benchmark runs. If possible, we compute the size of the query result, e.g. with 'SELECT COUNT(*)'. If not even that is possible (e.g. for Prolog systems), we simply backtrack over all solutions. However, sometimes we check that the system indeed computes the correct output. In this way, we might also detect if the optimizer does too clever optimizations because the output is not printed. The column WITH_OUTPUT is 'Y' if output is printed, and 'N' if not. Most views for comparing benchmark results only select rows where WITH_OUTPUT='N'.
  • CHAR(1) NOT NULL CHECK(WITH_OUTPUT IN ('Y','N'))
• LOAD_TIME
  • The time (in seconds) that the program itself reported as time for loading the data (CPU time if possible, otherwise real time). This column is NULL if we were not able to get this information easily.
  • NUMERIC(10,3) NULL CHECK(LOAD_TIME >= 0)
• EXEC_TIME
  • The time (in seconds) that the program itself reported as time for executing the query. As for the LOAD_TIME this can be CPU time, real time, or null.
  • NUMERIC(10,3) NULL CHECK(EXEC_TIME >= 0)
• TOTAL_TIME
  • The total runtime that the program reported for executing the benchmark. Normally, this should be the sum of LOAD_TIME and EXEC_TIME. Again, it can be null if the program has no built-in timeing function.
  • NUMERIC(10,3) NULL CHECK(TOTAL_TIME >= 0)
• USER_TIME
  • The CPU time that the program needed in user mode for executing the benchmark. This time is measured externally, normally with /usr/bin/time. For server-based systems, it might not be available (NULL).
  • NUMERIC(10,3) NULL CHECK(USER_TIME >= 0),
• SYS_TIME
  • The CPU time that the program needed in system/kernel mode. This will mainly be the time for the operating system calls for reading the input data. For server-based systems, it might not be available (NULL).
  • NUMERIC(10,3) NULL CHECK(SYS_TIME >= 0)
• REAL_TIME
  • This is the only performance measurement that is always available (never NULL). It is the real time that the benchmark program has run. For server based systems, it is only the time that the client has run, i.e. the startup and shutdown time for the server is not included.
  • NUMERIC(10,3) NOT NULL CHECK(REAL_TIME >= 0)
• MEM
  • This is the amount of memory (RAM) that the program used. More precisely, it is the 'maximum resident set size' reported by /usr/bin/time (in kilobytes). Shared libraries do count for the maximum resident set size, as far as they are actually used. Memory that was longer not accessed by the process does not count if the operating system reclaims the page. This in particular includes memory that is actually swapped out to disk. See e.g.
    • [https://utcc.utoronto.ca/~cks/space/blog/unix/UnderstandingRSS]
    • [https://unix.stackexchange.com/questions/35129/need-explanation-on-resident-set-size-virtual-size]
    • [https://unix.stackexchange.com/questions/30940/getrusage-system-call-what-is-maximum-resident-set-size/30941]
  • NUMERIC(10,3) NULL CHECK(MEM >= 0)
• STATUS
  • This should normally be just 'OK'. If the benchmark program exited with non-zero exit code, it is 'FAILED'. This might include cases where a limit was reached, e.g. the program used too much memory. The valued 'OUTL' is used for marking outliers. Measurements with values 'FAILED' and 'OUTL' are ignored in the benchmark comparisons. There are also various warnings generated by programs when real time and CPU time differ unusually much.
  • VARCHAR(80) NULL CHECK(STATUS IN ('OK', 'FAILED', 'OUTL', 'Warn', 'Warn1', 'Warn2', 'Warn3', 'Warn4')),
• REMARK
  • One could add a remark or tag to a measurement in this column.
  • VARCHAR(800) NULL

ESTIMATE_PAR: Experimental Table for Runtime Estimation Parameters

These are parameters for a formula that tries to generate an approximation of the measured runtime from characteristics of the input data. Currently, it is only used for the tcff benchmark. Note that more than one set of parameters can be stored for the same program. It is possible to compare the proposed parameters in this way.

• PROG
  • Program ID, see table PROGRAM above.
  • VARCHAR(20) NOT NULL REFERENCES PROGRAM
• BENCH
  • Benchmark ID, see table BENCHMARK above. Currently, this is actually only used for the tcff benchmark.
  • VARCHAR(20) NOT NULL REFERENCES BENCHMARK
• STARTUP
  • Startup/Initialization time in milliseconds.
  • NUMERIC(10) NOT NULL CHECK(STARTUP >= 0)
• LOAD_IDX
  • Time in milliseconds for loading data per one million 'edges * log(edges)'. The logarithmic factor simulate the creation of an index.
  • NUMERIC(10) NOT NULL CHECK(LOAD_IDX >= 0)
• RULE_APP
  • Time in milliseconds per one million rule applications (deductions). For the tcff benchmark, the number of necessary rule applications are stored in the column COST of the table GRAPH_DATA above.
  • NUMERIC(10) NOT NULL CHECK(RULE_APP >= 0)
• RESULT
  • Time in milliseconds be one million result tuples (answers). This can be used to give a different weight to successful rule applications in contrast to computing a duplicate of an already known result.
  • NUMERIC(10) NOT NULL CHECK(RESULT >= 0)
  • PRIMARY KEY(PROG,BENCH,STARTUP,LOAD_IDX,RULE_APP,RESULT)
• HITS
  • This is the number of input graphs (from 43) for which the formula generates an acceptable approximation of the real runtime. Of course, we can compute this number with a view (NUM_BAD_ESTIMATES) below. The number stored in this table is what our optimization program reported.
  • NUMERIC(10) NOT NULL CHECK(HITS >= 0)

DUMMY_TAB: Dummy Table with Exactly One Row

This is used in some views (to write a FROM clause).

The table has only one column:

• ATTR NUMERIC(1)

The table contents is:

Views for Analyzing the Collected Data

Sorry, currently we can only list the views. More documentation will be added later.

Statistical Evaluation of Run Data:

• RUN_STATISTICS: Full statistical evaluation of run data. We run each benchmark several times (usually ten times), and this view computes averages, minimum, maximum, interval size and standard deviation for several performance parameters.
• RUN_STAT: This is the same as RUN_STATISTICS, but without STDDEV, which is not supported in SQLite. Since most other views are based on the averages, we actually use RUN_STAT in the other views.
• RESULT: This is a selection of the benchmark results in RUN_STAT for current program version (installation) on the standard machine without output.
• BEST_RESULT: This is again a subset of RESULT that only considers the benchmark implementation for a program that was manually marked 'best' in PROG_IMPL.

Outlier Detection, Checking of Results for Plausibility

• OUTLIER_REAL: Detect Outliers with regard to Real Time.
• OUTLIER_CPU: Detect Outliers with regard to CPU Time.
• OUTLIER_MEM: Detect Outliers with regard to Memory.
• REAL_TOO_LARGE: Check where Realtime is much more than CPU time (at least double).
• PCT_CPU: Compare Real Time with CPU Time.
• PCT_CPU_INPUT: Compare Real Time with CPU Time for each input file.

Views with Benchmark Results for a Specific System

There is one view for each tested program that lists the benchmark results for that program.

• BAM
• XSB
• YAP
• SQLITE
• PSQL
• MARIA
• SOUFFLE
• JENA
• DATASCRIPT

Views to Find/Check Best Option Settings for a System and a Benchmark

These views are intended to help finding/checking the best implementation of a benchmark for a program, e.g. which indexes should be chosen. The views consider only the runtimes of the current program version (installation) on the standard machine.

• BEST_IMPL_REAL: Best option settings in real time.
• BEST_IMPL_CPU: Best Option Settings (in CPU Time) for a Program for a Benchmark.
• MARIA_COMPARE: Compare Storage Managers for MariaDB.

Views for Plots of Performance Data vs. Cost Measures (only for TCFF)

We use these views for getting the data for plots of performance data vs. cost measures, in particular the run time vs. the number of applicable rule instances. The results are available in csv and tsv formats. These views look only at the current program version (installation) on the standard machine, with the best implementation of the benchmark for the system (best indexes, options).

• PLOT_REAL: Plot of Real Time vs. Cost for TCFF Benchmark:
• PLOT_CPU: Plot of CPU Time vs. Cost for TCFF Benchmark
• PLOT_MEM: Plot of Memory vs. Number of Tuples for TCFF Benchmark.

Benchmark Results for TCFF - Comparison of Different Systems

The following views are for comparing system runtimes for the TCFF benchmark. These views look only at the current program version (installation) on the standard machine, with the best implementation of the benchmark for the system (best indexes, options).

• TCFF_CPU: Benchmark Results for TCFF (Comparison of Various Systems on CPU Time).
• TCFF_REAL: Benchmark Results for TCFF (Comparison of Various Systems on REAL Time). This version contains the runtime factors compared with BAM.
• TCFF_REAL_FACTORS: This is a projection of TCFF_REAL that shows only the factors compared to BAM. This makes the output rows a little smaller.
• TCFF_REAL_LATEX: LaTeX output for TCFF_REAL.
• TCFF_REAL_XSB: Benchmark Results for TCFF (Comparison of Various Systems on REAL Time). This version contains the runtime factors compared with XSB and rounds the factors to two decimal places after the decimal point. Otherwise it is identical to TCFF_REAL (which compares with BAM).
• TCFF_REAL_XSB_FACTORS: Projection of TCFF_REAL_XSB on factors only.
• TCFF_REAL_XSB_LATEX: LaTeX output from TCFF_REAL_XSB.
• TCFF_REAL_XSB_HTML: HTML output form TCFF_REAL_XSB.
• TCFF_MEM: Benchmark Results for TCFF (Comparison of Various Systems on Memory).
• TCFF_CPU_LATEX: LatEX output for comparison of old selection of systems on CPU time.

Graph Data in Various Formats

• GRAPH_DATA_LATEX: The table GRAPH_DATA in LaTeX format.
• GRAPH_DATA_HTML: The table GRAPH_DATA in HTML format.

Support for Reasearch on Runtime Estimation

• SPEED_REAL: System Speed in Real Time
• SPEED_REAL_DIFF: System Speed in Real Time With Startup Time
• SPEED_CPU: System Speed in CPU Time
• MEM_EFFICIENCY: Memory efficiency (memory usage divided by number of facts).
• ESTIMATE_FORMULA: Compute Run Time Estimates (Real Time).
• ESTIMATE_ERR: Check System Speed Estimates (Real Time).
• NUM_BAD_ESTIMATES: Count the Number of Wrong Estimates (more than 20%)
• ESTIMATE_DATA: Generate C++ Objects with Graph Data and Runtime (for Optimization).
• LOAD_EFFICIENCY: Load time per million edges in s.
• PAR_RANGE: Search Range for Parameter Values.

Output Table Data in Markdown Format (for Documentation)

• BENCH_IMPL_MD: The table BENCH_IMPL in MD format.
• DATA_FILE_MD: The table DATA_FILE in MD format.
• GRAPH_DATA_MD: The table GRAPH_DATA in MD format.