Skip to content

Testing

Testing GEF

This page describes how GEF testing is done. Any new command/functionality must receive adequate testing to be merged. Also PR failing CI (test + linting) won't be merged either.

Prerequisites

All the prerequisite packages are in requirements.txt file at the root of the project. So running

python -m pip install -r tests/requirements.txt --user -U

is enough to get started.

Running tests

Using make tests

For testing GEF on the architecture on the host running the tests (most cases), simply run

cd /root/of/gef
make test

Otherwise, make can be adjusted to : - run one test/one group of tests specifically - eg. make test_cmd_heap will trigger all the test functions for the heap command - run for a different architecture - eg. on x86, you can test x86 too with TARGET=i686 make TARGET=i686 test

Unless a specific test is mentioned, the command make test will run all the non-benchmark related tests. At the end, a summary of explanation will be shown, clearly indicating the tests that have failed: for instance:

=================================== short test summary info ==================================
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_large - AssertionError: 'siz...
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_small - AssertionError: 'siz...
FAILED tests/commands/heap.py::HeapCommand::test_cmd_heap_bins_unsorted - AssertionError: '...
======================== 3 failed, 4 passed, 113 deselected in 385.77s (0:06:25)==============

You can then use pytest directly to help you fix each error specifically.

Using pytest

GEF entirely relies on pytest for its testing. Refer to the project documentation for details.

Adding a new command requires for extensive testing in a new dedicated test module that should be located in /root/of/gef/tests/commands/my_new_command.py

A skeleton of a test module would look something like:

"""
`my-command` command test module
"""


from tests.utils import GefUnitTestGeneric, gdb_run_cmd, gdb_start_silent_cmd


class MyCommandCommand(GefUnitTestGeneric):
    """`my-command` command test module"""

    def test_cmd_my_command(self):
        # `my-command` is expected to fail if the session is not active
        self.assertFailIfInactiveSession(gdb_run_cmd("my-command"))

        # `my-command` should never throw an exception in GDB when running
        res = gdb_start_silent_cmd("my-command")
        self.assertNoException(res)

        # it also must print out a "Hello World" message
        self.assertIn("Hello World", res)

When running your test, you can summon pytest with the --pdb flag to enter the python testing environment to help you get more information about the reason of failure.

Linting GEF

You can use the Makefile at the root of the project to get the proper linting settings. For most cases, the following command is enough:

cd /root/of/gef
make lint

Benchmarking GEF

Benchmarking relies on pytest-benchmark and is experimental for now.

You can run all benchmark test cases as such:

cd /root/of/gef
pytest -k benchmark

which will return (after some time) an execution summary

tests/perf/benchmark.py ..                                                               [100%]


---------------------------------------- benchmark: 3 tests -----------------------------------
Name (time in ms)          Min                 Max                Mean            StdDev              Median                IQR            Outliers     OPS            Rounds  Iterations
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
time_baseline         612.2325 (1.0)      630.3416 (1.01)     623.7984 (1.01)     7.2848 (1.64)     626.1485 (1.01)      9.9971 (1.81)          1;0  1.6031 (0.99)          5           1
time_cmd_context      613.8124 (1.00)     625.8964 (1.0)      620.1908 (1.0)      4.4532 (1.0)      619.8831 (1.0)       5.5109 (1.0)           2;0  1.6124 (1.0)           5           1
time_elf_parsing      616.5053 (1.01)     638.6965 (1.02)     628.1588 (1.01)     8.2465 (1.85)     629.0099 (1.01)     10.7885 (1.96)          2;0  1.5920 (0.99)          5           1
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Legend:
  Outliers: 1 Standard Deviation from Mean; 1.5 IQR (InterQuartile Range) from 1st Quartile and 3rd Quartile.
  OPS: Operations Per Second, computed as 1 / Mean
============================================== 3 passed, 117 deselected in 14.78s =============================================