bench_run

Arguments

module

a vector of string. Contains the names of a Scilab modules to benchmark.

path_to_module

a vector of string. Contains the paths to directories of modules to test. If "/path/to/directory" is given as input parameter, tests are retrieved in the subdirectory /path/to/directory/tests/benchmarks .Used for homemade benchmarks.

test_name

a vector of string. Contains the names of the tests to perform.

The name of a test is its filename without .tst. If several modules or directory are given as first input parameter, scans for tests in each of these modules or directory.

Partial test names are allowed to run a subset of benchmarks dedicated to the same function/feature. For instance, specifying "ascii" will select all tests (in given module(s)) whose names contain "ascii" (See examples).

options

a vector of string

• "list": list of the benchmark tests (test_name) available in a module

• "help": displays some examples of use in the Scilab console

• "nb_run=value": runs each benchmark value times ; by default bench_run runs 10000 times the code between BENCH START and BENCH END tags (see below). Overrides any BENCH NB RUN specified in the benchmark test files.

exportToFile

a single string

File path to the result of the bench_run in xml format. By default, or if "", "[]" or [] is given, the output directory is TMPDIR/benchmarks/.

If exportToFile is a directory, creates a timestamped output file is the directory, otherwise creates the file exportToFile. If the file could not be created a warning is issued and the file is created under TMPDIR/benchmarks/ instead.

modutests_names

a N-by-2 matrix of strings

the first column lists the modules tested by bench_run, the second column lists the names of the benchmarks

elapsed_time

a vector of doubles

the execution time for each benchmark

nb_iterations

a vector of doubles of size N

the number of iterations of respective test

Description

Performs benchmark tests, measures execution time and produces a report about benchmark tests.

Searches for .tst files in benchmark test library or input parameter path under tests/benchmark subdirectory, executes them 10000 times and displays a report about execution time.

Special tags may be inserted in the .tst file, which help to control the processing of the corresponding test. These tags are expected to be found in Scilab comments.

These are the available tags :

• <-- BENCH NB RUN : 10 -->

  By default, this test will be repeated 10 times, unless the "nb_run=###"bench_run(..) option is used. The value given for the flag can be set to any integer value.

• //    <-- BENCH START -->
  [code to be executed]
  //    <-- BENCH END -->

  Code between these tags will be repeated. Any code written before/after will be executed only once before/after the repetition, without being timed. If these tags are missing, the entire code will be repeated.

Examples

Some simple examples of invocation of bench_run

// Launch all tests
// This may take some time...
// bench_run();
// bench_run([]);
// bench_run([],[]);

// Test one or several module
bench_run('core');
bench_run('core',[]);
bench_run(['core','string']);

// Launch one or several test in a specified module
bench_run('core',['trycatch','opcode']);

// With options
bench_run([],[],'list');
bench_run([],[],'help');
bench_run("string", [], 'nb_run=100');
// results in an output file in the current directory
bench_run("string", [], 'nb_run=100', 'my_output_file.xml');
// results in an output directory, TMPDIR/benchmarks is the default
bench_run("string", [], 'nb_run=100', TMPDIR);

An example of a benchmark file. This file corresponds to the file SCI/modules/linear_algebra/tests/benchmarks/bench_chol.tst.

// =============================================================================
// Scilab ( https://www.scilab.org/ ) - This file is part of Scilab
// Copyright (C) 2007-2008 - INRIA
//
//  This file is distributed under the same license as the Scilab package.
// =============================================================================

//==============================================================================
// Benchmark for chol function
//==============================================================================

// <-- BENCH NB RUN : 10 -->

a = 0;
b = 0;
a = rand(900, 900, 'n');
a = a'*a;

// <-- BENCH START -->
b = chol(a);
// <-- BENCH END -->

The result of the test

--> bench_run('linear_algebra','bench_chol')

           For Loop (as reference) ...........................      33.20 ms [ 1000000 x]

 001/001 - [linear_algebra] bench_chol ......................     1233.93 ms [      10 x]

Running a subset of dedicated benchmarks by using a partial/generic testname:

--> bench_run string ascii

          For Loop (as reference) ...........................      102.98 ms [ 1000000 x]

 001/005 - [string] bench_ascii_1 ...........................      447.40 ms [   10000 x]
 002/005 - [string] bench_ascii_2 ...........................    31727.98 ms [ 1000000 x]
 003/005 - [string] bench_ascii_3 ...........................     4173.69 ms [   10000 x]
 004/005 - [string] bench_ascii_4 ...........................     5145.06 ms [   10000 x]
 005/005 - [string] bench_ascii_UTF8 ........................       23.26 ms [      10 x]

See also

• test_run — runs unit tests and non-regression tests of a module or in a directory
• coverage — Instruments some macros to store runtime information on code coverage and execution time
• slint — Checks the Scilab code of given macros against a set of criteria
• debug — enters the Scilab debugger mode

History