Test Runner fails to detect test definitions in subdirectory test files with non-matching namespace structures

[ikappaki]

Hi,

There seems to be an issue when running Basilisp tests located in a subdirectory, where the test namespace does not include the root directory but the subdirectory is added to the sys.path.

For example, a test file at test2/a/test2_test.lpy with the namespace a.test2-test should run when basilisp test is invoked with the correct --include-path. However, the tests are not detected despite the file is loaded.

case 1

For example in the first example below where the namespace a.test2-test does not start with the top test directory test2/

test2/a/test2_test.lpy

(ns a.test2-test
  (:require [basilisp.test :refer [deftest is]]))

(deftest a-test2-test
  (is (= 7 8)))

the tests are not detected even though I specified that the top subdirectory is in the sys.path (so no need to start the namespace with it)

> basilisp test --include-path test2 test2
==== test session starts ===================================================================
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
rootdir: D:\bas\issue-bas-test-startdir
configfile: pyproject.toml
plugins: basilisp-0.3.3
collected 0 items

case 2

In the second example below where I do place the top directory name in the namespace test1.a.test1-test the tests are loaded as expected (failure here is intentional to prove the test was loaded and executed)
test1/a/test1_test.lpy

(ns test1.a.test1-test
  (:require
   [basilisp.test :refer [deftest is]]))

(deftest the-test1-test
  (is (= 5 6)))

> basilisp test test1
==== test session starts ===================================================================
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
rootdir: D:\bas\issue-bas-test-startdir
configfile: pyproject.toml
plugins: basilisp-0.3.3
collected 1 item

test1\a\test1_test.lpy F                                                                                                                            [100%]

==== FAILURES ========================================================================
_____ the-test1-test _____________________________________________________________________
FAIL in (the-test1-test) (test1_test.lpy:6)
    Test failure: (= 5 6)

    expected: 5
      actual: 6

expectations

I expected that if I have tests in a subdirectory of the project, with test namespaces starting from that subdirectory (case 1), I should be able to run them by specifying basilisp test --include-path <subdirectory> <subdirectory>. Am I perhaps missing something?

I’ve created a repository with the test cases mentioned: https://github.com/ikappaki/issue-bas-test-startdir, in case you'd like to have a look.

Thanks

Another strange behavior occurs if I add a println statement at the top level of test2_test.lpy as shown below It results in a name 'basilisp_core' is not defined error, which I don’t fully understand where it is coming from:

(ns a.test2-test
  (:require [basilisp.test :refer [deftest is]]))

(println :test2)
;; NameError: name 'basilisp_core' is not defined

(deftest a-test2-test
  (is (= 7 8)))

> basilisp test --include-path test2 test2
==== test session starts ===================================================================
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
rootdir: D:\bas\issue-bas-test-startdir
configfile: pyproject.toml
plugins: basilisp-0.3.3
collected 0 items / 1 error

========================================================================= ERRORS =========================================================================
________________________________________________________ ERROR collecting test2/a/test2_test.lpy _________________________________________________________
.venv\Lib\site-packages\basilisp\importer.py:408: in exec_module
    self._exec_cached_module(fullname, spec.loader_state, path_stats, ns)
.venv\Lib\site-packages\basilisp\importer.py:313: in _exec_cached_module
    cached_code = _get_basilisp_bytecode(
.venv\Lib\site-packages\basilisp\importer.py:78: in _get_basilisp_bytecode
    raise ImportError(message, **exc_details)
E   ImportError: Non-matching timestamp (1732907277) in test2.a.test2_test bytecode cache; expected 1732912476

During handling of the above exception, another exception occurred:
test2\a\test2_test.lpy:7: in <module>
    (println :test2)
E   NameError: name 'basilisp_core' is not defined

[chrisrink10]

If I'm understanding the ticket correctly, I think you're just doing something that isn't supported (and maybe isn't really best practice anyway).

You're basically executing the 2 cases the same way, but expecting them to have different discovery rules. PyTest adds the test directory to the path by default (see the --import-mode=prepend CLI option), so you shouldn't have to actually manipulate the path.

The behavior you're seeing is somewhat specific to Basilisp (and Clojure) since each namespace typically declares its name at the top via the ns macro (and that namespace has to map closely to the filesystem layout of the project). Basilisp (via PyTest) attempts to guess the namespace from the path (relative to sys.path). When you run basilisp test --include-path test2 test2, the base is still test2 and Basilisp attempts to load test2.a.test2-test which doesn't exist so collection "fails" (from your perspective). This is somewhat different than how PyTest works, where it doesn't necessarily need to know the fully qualified name of a file to load the file using the default heuristics.

I think you would probably be better served using PyTest's test selection (e.g. -k option) to select subsets of the test suite, rather than manipulating the path to execute specific directories.

[ikappaki]

If I'm understanding the ticket correctly, I think you're just doing something that isn't supported (and maybe isn't really best practice anyway).

You're basically executing the 2 cases the same way, but expecting them to have different discovery rules. PyTest adds the test directory to the path by default (see the --import-mode=prepend CLI option), so you shouldn't have to actually manipulate the path.

Apologies, t’s unclear what the --import-mode=prepend option above does in this context. In practice, inspecting sys.path using a (throw (python/Exception sys/path)) at the top level of a test file doesn’t show the test2 directory in sys.path unless it is explicitly included with the --include-path option.

For example, consider an updated test2/a/test2_test.lpy:

(ns a.test2-test
  (:require [basilisp.test :refer [deftest is]]))

(throw (python/Exception sys/path))
;; => test2/ is in the sys.path?

(deftest a-test2-test
  (is (= 7 8)))

If you run the test specifying only the test2 directory, sys.path does not include the path to test2

> basilisp test test2
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
...

test2\a\test2_test.lpy:4: in <module>
    (throw (python/Exception sys/path))
E   Exception: ['', 'D:\\issue-bas-test-rootdir\\.venv\\Scriptsilisp.exe', 'C:\\local\\Python311\\python311.zip', 'C:\\local\\Python311\\DLLs', 'C:\\local\\Python311\\Lib', 'C:\\local\\Python311', 'D:\\issue-bas-test-rootdir\\.venv', 'D:\\issue-bas-test-rootdir\\.venv\\Lib\\site-packages', 'D:\\issue-bas-test-rootdir\\src']

However, when the --include-path option is explicitly used, the path to the test2 directory is added to sys.path as expected (D:\\issue-bas-test-rootdir\\test2):

> basilisp test --include-path test2 test2
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
rootdir: D:\bas\issue-bas-test-rootdir

test2\a\test2_test.lpy:4: in <module>
    (throw (python/Exception sys/path))
E   Exception: ['', 'D:\\issue-bas-test-rootdir\\test2', 'D:\\issue-bas-test-rootdir\\.venv\\Scriptsilisp.exe', 'C:\\local\\Python311\\python311.zip', 'C:\\local\\Python311\\DLLs', 'C:\\local\\Python311\\Lib', 'C:\\local\\Python311', 'D:\\issue-bas-test-rootdir\\.venv', 'D:\\issue-bas-test-rootdir\\.venv\\Lib\\site-packages', 'D:\\issue-bas-test-rootdir\\src']

The behavior you're seeing is somewhat specific to Basilisp (and Clojure) since each namespace typically declares its name at the top via the ns macro (and that namespace has to map closely to the filesystem layout of the project). Basilisp (via PyTest) attempts to guess the namespace from the path (relative to sys.path). When you run basilisp test --include-path test2 test2, the base is still test2 and Basilisp attempts to load test2.a.test2-test which doesn't exist so collection "fails" (from your perspective). This is somewhat different than how PyTest works, where it doesn't necessarily need to know the fully qualified name of a file to load the file using the default heuristics.

I think you would probably be better served using PyTest's test selection (e.g. -k option) to select subsets of the test suite, rather than manipulating the path to execute specific directories.

Sorry, I should have asked this question earlier instead of trying to figure out where things were going wrong.

In Clojure projects, tests are typically located in the ./test directory, and test files follow a structure like ./test/myproject/core_test.clj. The core-test namespace is declared as (ns myproject.core-test), without including the test part in the namespace.

If I were to port this test to Basilisp, I would replace the .clj with .lpy as ./test/myproject/core_test.lpy (with the aim to eventually back port them as .cljc), while everything else remaining the same. How can I invoke the tests in core_test.lpy without changing the (ns myproject.core-test) declaration to (ns test.myproject.core-test)? I’d like to avoid making this change if possible when porting libraries to Basilisp.

This is the case 1 I mentioned earlier in trying to explain the issue with my limited understanding.

Thanks!

[chrisrink10]

Can you try --include-path test2/a?

[ikappaki]

Can you try --include-path test2/a?

Still, no tests are being picked up 🥲

 basilisp test --include-path test2/a test2
================================================================== test session starts ===================================================================
platform win32 -- Python 3.11.4, pytest-8.3.3, pluggy-1.5.0
rootdir: D:\issue-bas-test-rootdir
configfile: pyproject.toml
plugins: basilisp-0.3.3
collected 0 items

[ikappaki]

(I think I might have figured this out, I'll open a PR once I've written down the test cases, You can assign this to me unless is being already looked up, it should also solve the sister issue with the underscores in the namespaces)

[chrisrink10]

Fixed in #1176