[libcxx][docs] Make test name pattern documentation more obvious

libcxx/docs/TestingLibcxx.rst

@@ -325,18 +325,98 @@ This macro is in a different header as ``assert_macros.h`` since it pulls in
 additional headers.
 
  .. note: This macro can only be used in test using C++20 or newer. The macro
-          was added at a time where most of lib++'s C++17 support was complete.
+          was added at a time where most of libc++'s C++17 support was complete.
           Since it is not expected to add this to existing tests no effort was
           taken to make it work in earlier language versions.
 
 
-Additional reading
-------------------
-
-The function ``CxxStandardLibraryTest`` in the file
-``libcxx/utils/libcxx/test/format.py`` has documentation about writing test. It
-explains the difference between the test named  ``foo.pass.cpp`` and named
-``foo.verify.cpp`` are.
+Test names
+----------
+
+The names of test files have meaning for the libc++-specific configuration of
+Lit. Based on the pattern that matches the name of a test file, Lit will test
+the code contained therein in different ways. Refer to the `Lit Meaning of libc++
+Test Filenames`_ when determining the names for new test files.
+
+.. _Lit Meaning of libc++ Test Filenames:
+.. list-table:: Lit Meaning of libc++ Test Filenames
+   :widths: 25 75
+   :header-rows: 1
+
+   * - Name Pattern
+     - Meaning
+   * - ``FOO.pass.cpp``
+     - Checks whether the C++ code in the file compiles, links and runs successfully.
+   * - ``FOO.pass.mm``
+     - Same as ``FOO.pass.cpp``, but for Objective-C++.
+
+   * - ``FOO.compile.pass.cpp``
+     - Checks whether the C++ code in the file compiles successfully. In general, prefer ``compile`` tests over ``verify`` tests, 
+       subject to the specific recommendations, below, for when to write ``verify`` tests.
+   * - ``FOO.compile.pass.mm``
+     - Same as ``FOO.compile.pass.cpp``, but for Objective-C++.
+   * - ``FOO.compile.fail.cpp``
+     - Checks that the code in the file does *not* compile successfully.
+
+   * - ``FOO.verify.cpp``
+     - Compiles with clang-verify. This type of test is automatically marked as UNSUPPORTED if the compiler does not support clang-verify.
+       For additional information about how to write ``verify`` tests, see the `Internals Manual <https://clang.llvm.org/docs/InternalsManual.html#verifying-diagnostics>`_.
+       Prefer `verify` tests over ``compile`` tests to test that compilation fails for a particular reason. For example, use a ``verify`` test
+       to ensure that
+
+       * an expected ``static_assert`` is triggered;
+       * the use of deprecated functions generates the proper warning;
+       * removed functions are no longer usable; or
+       * return values from functions marked ``[[nodiscard]]`` are stored.
+
+   * - ``FOO.link.pass.cpp``
+     - Checks that the C++ code in the file compiles and links successfully -- no run attempted.
+   * - ``FOO.link.pass.mm``
+     - Same as ``FOO.link.pass.cpp``, but for Objective-C++.
+   * - ``FOO.link.fail.cpp``
+     - Checks whether the C++ code in the file fails to link after successful compilation.
+   * - ``FOO.link.fail.mm``
+     - Same as ``FOO.link.fail.cpp``, but for Objective-C++.
+
+   * - ``FOO.sh.<anything>``
+     - A *builtin Lit Shell* test.
+   * - ``FOO.gen.<anything>``
+     - A variant of a *Lit Shell* test that generates one or more Lit tests on the fly. Executing this test must generate one or more files as expected
+       by LLVM split-file. Each generated file will drive an invocation of a separate Lit test. The format of the generated file will determine the type
+       of Lit test to be executed. This can be used to generate multiple Lit tests from a single source file, which is useful for testing repetitive properties
+       in the library. Be careful not to abuse this since this is not a replacement for usual code reuse techniques.
+
+
+libc++-Specific Lit Features
+----------------------------
+
+Custom Directives
+~~~~~~~~~~~~~~~~~
+
+Lit has many directives built in (e.g., ``DEFINE``, ``UNSUPPORTED``). In addition to those directives, libc++ adds two additional libc++-specific directives that makes
+writing tests easier. See `libc++-specific Lit Directives`_ for more information about the ``FILE_DEPENDENCIES`` and ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific directives.
+
+.. _libc++-specific Lit Directives:
+.. list-table:: libc++-specific Lit Directives
+   :widths: 20 35 45
+   :header-rows: 1
+
+   * - Directive
+     - Parameters
+     - Usage
+   * - ``FILE_DEPENDENCIES``
+     - ``// FILE_DEPENDENCIES: file, directory, /path/to/file, ...``
+     - The paths given to the ``FILE_DEPENDENCIES`` directive can specify directories or specific files upon which a given test depend. For example, a test that requires some test
+       input stored in a data file would use this libc++-specific Lit directive. When a test file contains the ``FILE_DEPENDENCIES`` directive, Lit will collect the named files and copy
+       them to the directory represented by the ``%T`` substitution before the test executes. The copy is performed from the directory represented by the ``%S`` substitution
+       (i.e. the source directory of the test being executed) which makes it possible to use relative paths to specify the location of dependency files. After Lit copies
+       all the dependent files to the directory specified by the ``%T`` substitution, that directory should contain *all* the necessary inputs to run. In other words,
+       it should be possible to copy the contents of the directory specified by the ``%T`` substitution to a remote host where the execution of the test will actually occur.
+   * - ``ADDITIONAL_COMPILE_FLAGS``
+     - ``// ADDITIONAL_COMPILE_FLAGS: flag1 flag2 ...``
+     - The additional compiler flags specified by a space-separated list to the ``ADDITIONAL_COMPILE_FLAGS`` libc++-specific Lit directive will be added to the end of the ``%{compile_flags}``
+       substitution for the test that contains it. This libc++-specific Lit directive makes it possible to add special compilation flags without having to resort to writing a ``.sh.cpp`` test (see
+       `Lit Meaning of libc++ Test Filenames`_), more powerful but perhaps overkill.
 
 Benchmarks
 ==========

libcxx/utils/libcxx/test/format.py

@@ -151,38 +151,11 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
     """
     Lit test format for the C++ Standard Library conformance test suite.
 
-    This test format is based on top of the ShTest format -- it basically
-    creates a shell script performing the right operations (compile/link/run)
-    based on the extension of the test file it encounters. It supports files
-    with the following extensions:
-
-    FOO.pass.cpp            - Compiles, links and runs successfully
-    FOO.pass.mm             - Same as .pass.cpp, but for Objective-C++
-
-    FOO.compile.pass.cpp    - Compiles successfully, link and run not attempted
-    FOO.compile.pass.mm     - Same as .compile.pass.cpp, but for Objective-C++
-    FOO.compile.fail.cpp    - Does not compile successfully
-
-    FOO.link.pass.cpp       - Compiles and links successfully, run not attempted
-    FOO.link.pass.mm        - Same as .link.pass.cpp, but for Objective-C++
-    FOO.link.fail.cpp       - Compiles successfully, but fails to link
-
-    FOO.sh.<anything>       - A builtin Lit Shell test
-
-    FOO.gen.<anything>      - A .sh test that generates one or more Lit tests on the
-                              fly. Executing this test must generate one or more files
-                              as expected by LLVM split-file, and each generated file
-                              leads to a separate Lit test that runs that file as
-                              defined by the test format. This can be used to generate
-                              multiple Lit tests from a single source file, which is
-                              useful for testing repetitive properties in the library.
-                              Be careful not to abuse this since this is not a replacement
-                              for usual code reuse techniques.
-
-    FOO.verify.cpp          - Compiles with clang-verify. This type of test is
-                              automatically marked as UNSUPPORTED if the compiler
-                              does not support Clang-verify.
-
+    Lit tests are contained in files that follow a certain pattern, which determines the semantics of the test.
+    Under the hood, we basically generate a builtin Lit shell test that follows the ShTest format, and perform
+    the appropriate operations (compile/link/run). See
+    https://libcxx.llvm.org/TestingLibcxx.html#test-names
+    for a complete description of those semantics.
 
     Substitution requirements
     ===============================
@@ -200,30 +173,6 @@ class CxxStandardLibraryTest(lit.formats.FileBasedTest):
     in the same command line. In other words, the test format doesn't perform
     separate compilation and linking steps in this case.
 
-
-    Additional supported directives
-    ===============================
-    In addition to everything that's supported in Lit ShTests, this test format
-    also understands the following directives inside test files:
-
-        // FILE_DEPENDENCIES: file, directory, /path/to/file
-
-            This directive expresses that the test requires the provided files
-            or directories in order to run. An example is a test that requires
-            some test input stored in a data file. When a test file contains
-            such a directive, this test format will collect them and copy them
-            to the directory represented by %T. The intent is that %T contains
-            all the inputs necessary to run the test, such that e.g. execution
-            on a remote host can be done by simply copying %T to the host.
-
-        // ADDITIONAL_COMPILE_FLAGS: flag1 flag2 flag3
-
-            This directive will cause the provided flags to be added to the
-            %{compile_flags} substitution for the test that contains it. This
-            allows adding special compilation flags without having to use a
-            .sh.cpp test, which would be more powerful but perhaps overkill.
-
-
     Additional provided substitutions and features
     ==============================================
     The test format will define the following substitutions for use inside tests: