Configurable test dependencies

[jayvdb]

https://github.com/moremoban/pypi-mobans/blob/55ccacf/templates/setup.py.jj2#L279 hard-codes nose as the only test dependency.

This also implies that nose is the preferred test runner for the project.

Ideally the test runner is chosen , and all of its dependencies are defined automatically.

e.g.

test_runner: (unittest|nose|pytest)

Then there are other dependencies which are needed for executing the test runner, and other dependencies which are custom for the test suite, and some which are workarounds, and there are optional dependencies, which are needed for tests.

If we use https://github.com/coala/coala/blob/b3bb626/test-requirements.txt , reordered slightly, as an example

pip>=9.0.0,<10  # ci extras
wheel~=0.29
codecov~=2.0.5
moban~=0.0.9

argcomplete~=1.8  # optional dependency in the code

coverage~=4.3.4  # test runner dep
coverage-env-plugin~=0.1
coverage-config-reload-plugin~=0.2
pytest~=3.1.1
pytest-cov~=2.2
pytest-env~=0.6.0
pytest-error-for-skips~=1.0
pytest-instafail~=0.3.0
pytest-mock~=1.1
pytest-reorder~=0.1.0
pytest-reqs~=0.0.6
pytest-timeout~=1.0
pytest-xdist~=1.14

freezegun~=0.3.9  # unit test deps
requests-mock~=1.2

Very closely related is https://github.com/coala/coala/blob/249e715/setup.cfg#L8 which contains

[tool:pytest]
norecursedirs =
    .git
    dist
    build
    venv
    .env
testpaths =
    coalib
    docs
    tests
python_files = *Test.py
python_classes = *Test
timeout = 35
addopts =
    --instafail
    --reorder 'requirements.txt' 'test-requirements.txt' '*'
    --color=yes
    --doctest-glob=*.rst
    --doctest-modules
    --doctest-ignore-import-error
    -r a
    --ignore=tests/collecting/collectors_test_dir/bears/incorrect_bear.py
    --error-for-skips
    --cov
doctest_optionflags =
    ELLIPSIS
    IGNORE_EXCEPTION_DETAIL
reqsfilenamepatterns =
    requirements.txt
    test-requirements.txt
env =
    PYTHONHASHSEED=0
# PYTHONHASHSEED=0 is required to use same hashes in pytests-xdist's workers

[coverage:coverage_env_plugin]
markers = True

[coverage:run]
branch = True
cover_pylib = False
source =
    coalib
plugins =
    coverage_env_plugin
    coverage_config_reload_plugin

[coverage:report]
fail_under = 100
show_missing = True
skip_covered = False
sort = Miss
include =
    coalib/*
exclude_lines =
    pragma: no ?cover
    pragma ${PLATFORM_SYSTEM}: no cover
    pragma ${OS_NAME}: no cover
    pragma Python [0-9.,]*${PYTHON_VERSION}[0-9.,]*: no cover

Of that list, only "pytest" is can be unambiguously defined as a 'test runner' dependency, but coala tests doesnt work with the latest pytest (due to a bug I will have to fix..). So even the most basic test dependency which could be generated from test_runner: pytest would not help coala, unless we also have a way to specify the version pytest.

But if we ignore coala, then we maybe can ignore versions...

Then we could add

• a yaml bool flag to turn on coverage, and
• a flag for coveralls vs codecov, and
• a bool flag 'test_extra_dependencies' to enable copying the extra_requires into test_requires

then at least pytest-cov and codecov and argcomplete can also be added automatically.
That could be enough for most people ... and maybe works for me if I fix a few bugs in various upstreams.