TestViewNamesAndDescriptions.test_markdown test fails

[mcepl]

Checklist

• Raised initially as discussion #... (given that the test suite fail, I hope that the discussion is not necessary)
• This cannot be dealt with as a third party library. (We prefer new functionality to be in the form of third party libraries where possible.)
• I have reduced the issue to the simplest possible case. (again, I don't think it applies for the failing test)

When packaging this project as a package for openSUSE/Factory, this test fails:

[   15s] =================================== FAILURES ===================================
[   15s] __________________ TestViewNamesAndDescriptions.test_markdown __________________
[   15s] tests/test_description.py:170: in test_markdown
[   15s]     assert apply_markdown(DESCRIPTION) == MARKDOWN_BASE % MARKDOWN_gte_33
[   15s] E   AssertionError: assert ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<p><code>json\n'\n '[{\n'\n '    "alpha": 1,\n'\n '    "beta: "this is a string"\n'\n '}]</code></p>') == ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<div class="highlight"><pre><span></span><span class="p">[{</span><br />    '\n '<span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span '\n 'class="mi">1</span><span class="p">,</span><br />    <span '\n 'class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span '\n 'class="err">is</span> <span class="err">a</span> <span '\n 'class="err">string&quot;</span><br /><span class="p">}]</span><br '\n '/></pre></div>\n'\n '<p><br /></p>')
[   15s] E       <h2 id="an-example-docstring">an example docstring</h2>
[   15s] E       <ul>
[   15s] E       <li>list</li>
[   15s] E       <li>list</li>
[   15s] E       </ul>
[   15s] E       <h3 id="another-header">another header</h3>
[   15s] E       <pre><code>code block
[   15s] E       </code></pre>
[   15s] E       <p>indented</p>
[   15s] E       <h2 id="hash-style-header">hash style header</h2>
[   15s] E     - <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span class="err">is</span> <span class="err">a</span> <span class="err">string&quot;</span><br /><span class="p">}]</span><br /></pre></div>
[   15s] E     - <p><br /></p>
[   15s] E     + <p><code>json
[   15s] E     + [{
[   15s] E     +     "alpha": 1,
[   15s] E     +     "beta: "this is a string"
[   15s] E     + }]</code></p>

Using Python 3.6.14, djangorestframework-3.12.4, complete build log with all packages used and steps taken.

[sandrotosi]

Can anyone look into this issue? it is now affecting debian: https://ci.debian.net/data/autopkgtest/testing/amd64/d/djangorestframework/17493317/log.gz

thanks!

[tomchristie]

My assumption would be that this is the result of some minor patch version of the markdown library handling the markdown rendering slightly differently. Not super easy to see why since our test cases aren't actually failing.

PR #8293 cleans up a bit of the markdown that was rendering into the text you're seeing, and that I suspect was causing the issue, so I'll close this off on the assumption that it's most likely where the issue was caused.

As an aside, I don't really understand linux distributions that bundle Django or Django REST framework. They're not really intended to be used as system-wide-installations.

(And no, I don't want particularly want to have a discussion about that.)

[mcepl]

Yeah, the patch from #8293 doesn’t help at all:

[   16s] __________________ TestViewNamesAndDescriptions.test_markdown __________________
[   16s] tests/test_description.py:170: in test_markdown
[   16s]     assert apply_markdown(DESCRIPTION) == MARKDOWN_BASE % MARKDOWN_gte_33
[   16s] E   AssertionError: assert ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<p><code>json\n'\n '[{\n'\n '    "alpha": 1,\n'\n '    "beta: "this is a string"\n'\n '}]</code></p>') == ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<div class="highlight"><pre><span></span><span class="p">[{</span><br />    '\n '<span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span '\n 'class="mi">1</span><span class="p">,</span><br />    <span '\n 'class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span '\n 'class="err">is</span> <span class="err">a</span> <span '\n 'class="err">string&quot;</span><br /><span class="p">}]</span><br '\n '/></pre></div>\n'\n '<p><br /></p>')
[   16s] E       <h2 id="an-example-docstring">an example docstring</h2>
[   16s] E       <ul>
[   16s] E       <li>list</li>
[   16s] E       <li>list</li>
[   16s] E       </ul>
[   16s] E       <h3 id="another-header">another header</h3>
[   16s] E       <pre><code>code block
[   16s] E       </code></pre>
[   16s] E       <p>indented</p>
[   16s] E       <h2 id="hash-style-header">hash style header</h2>
[   16s] E     - <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span class="err">is</span> <span class="err">a</span> <span class="err">string&quot;</span><br /><span class="p">}]</span><br /></pre></div>
[   16s] E     - <p><br /></p>
[   16s] E     + <p><code>json
[   16s] E     + [{
[   16s] E     +     "alpha": 1,
[   16s] E     +     "beta: "this is a string"
[   16s] E     + }]</code></p>

[sandrotosi]

My assumption would be that this is the result of some minor patch version of the markdown library handling the markdown rendering slightly differently. Not super easy to see why since our test cases aren't actually failing.

this started happening in Debian after we upgraded pygments from 2.7.1 to 2.10.0. The reason your tests dont fail is likely because of this pygments requirement

@tomchristie please reopen as the issue is still present

thanks!

[slyon]

I can confirm this. We're seeing the same issue in Ubuntu, the proposed fix does not actually solve the problem. Please re-open and try to verify you test suite with pygments 2.10, as suggested. (The markdown library did not change)

[sandrotosi]

The failure we're seeing in Debian is slightly different, so i'm going to report it here if that helps (please note we're still on 3.12.4):

=================================================================================================================================== FAILURES ===================================================================================================================================
__________________________________________________________________________________________________________________ TestViewNamesAndDescriptions.test_markdown __________________________________________________________________________________________________________________
tests/test_description.py:170: in test_markdown
    assert apply_markdown(DESCRIPTION) == MARKDOWN_BASE % MARKDOWN_gte_33
E   AssertionError: assert ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<div class="highlight"><pre><span></span><span class="p">[{</span><br />    '\n '<span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span '\n 'class="mi">1</span><span class="p">,</span><br />    <span '\n 'class="s2">&quot;beta: &quot;</span><span class="kc">t</span><span '\n 'class="err">his</span> <span class="err">is</span> <span '\n 'class="err">a</span> <span class="err">s</span><span '\n 'class="kc">tr</span><span class="err">i</span><span class="kc">n</span><span '\n 'class="err">g&quot;</span><br /><span class="err">}]</span><br '\n '/></pre></div>\n'\n '<p><br /></p>') == ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<div class="highlight"><pre><span></span><span class="p">[{</span><br />    '\n '<span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span '\n 'class="mi">1</span><span class="p">,</span><br />    <span '\n 'class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span '\n 'class="err">is</span> <span class="err">a</span> <span '\n 'class="err">string&quot;</span><br /><span class="p">}]</span><br '\n '/></pre></div>\n'\n '<p><br /></p>')
E       <h2 id="an-example-docstring">an example docstring</h2>
E       <ul>
E       <li>list</li>
E       <li>list</li>
E       </ul>
E       <h3 id="another-header">another header</h3>
E       <pre><code>code block
E       </code></pre>
E       <p>indented</p>
E       <h2 id="hash-style-header">hash style header</h2>
E     - <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="nt">&quot;beta: &quot;</span><span class="err">this</span> <span class="err">is</span> <span class="err">a</span> <span class="err">string&quot;</span><br /><span class="p">}]</span><br /></pre></div>
E     ?                                                                                                                                                                                                                       ^^                                        ^^^^^^                                                                                     ^^^^                                 ^
E     + <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="s2">&quot;beta: &quot;</span><span class="kc">t</span><span class="err">his</span> <span class="err">is</span> <span class="err">a</span> <span class="err">s</span><span class="kc">tr</span><span class="err">i</span><span class="kc">n</span><span class="err">g&quot;</span><br /><span class="err">}]</span><br /></pre></div>
E     ?                                                                                                                                                                                                                       ^^                                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                 ^^^
E       <p><br /></p>

I'm going to skip this failing test for now, until this issue is addressed here

[stanislavlevin]

Exactly this issue is due to the changes in Pygments 2.11.0:

- <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="nt">&quot;beta&quot;</span><span class="p">:</span> <span class="s2">&quot;this is a string&quot;</span><br /><span class="p">}]</span><br /></pre></div>
+ <div class="highlight"><pre><span></span><span class="p">[{</span><span class="w"></span><br /><span class="w">    </span><span class="nt">&quot;alpha&quot;</span><span class="p">:</span><span class="w"> </span><span class="mi">1</span><span class="p">,</span><span class="w"></span><br /><span class="w">    </span><span class="nt">&quot;beta&quot;</span><span class="p">:</span><span class="w"> </span><span class="s2">&quot;this is a string&quot;</span><span class="w"></span><br /><span class="p">}]</span><span class="w"></span><br /></pre></div>

Bisect points to pygments/pygments@b4f0583

See pygments/pygments#1905 for details.

It seems this test is very flaky in the current implementation.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mcepl]

Still as fresh as ever with 3.13.1 and Markdown 3.3.7:

[   26s] =================================== FAILURES ===================================
[   26s] __________________ TestViewNamesAndDescriptions.test_markdown __________________
[   26s] 
[   26s] self = <tests.test_description.TestViewNamesAndDescriptions testMethod=test_markdown>
[   26s] 
[   26s]     @pytest.mark.skipif(not apply_markdown, reason="Markdown is not installed")
[   26s]     def test_markdown(self):
[   26s]         """
[   26s]         Ensure markdown to HTML works as expected.
[   26s]         """
[   26s]         # Markdown 3.3 is only supported on Python 3.6 and higher
[   26s]         if sys.version_info >= (3, 6):
[   26s] >           assert apply_markdown(DESCRIPTION) == MARKDOWN_BASE % MARKDOWN_gte_33
[   26s] E           assert ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<p><code>json\n'\n '[{\n'\n '    "alpha": 1,\n'\n '    "beta": "this is a string"\n'\n '}]</code></p>') == ('<h2 id="an-example-docstring">an example docstring</h2>\n'\n '<ul>\n'\n '<li>list</li>\n'\n '<li>list</li>\n'\n '</ul>\n'\n '<h3 id="another-header">another header</h3>\n'\n '<pre><code>code block\n'\n '</code></pre>\n'\n '<p>indented</p>\n'\n '<h2 id="hash-style-header">hash style header</h2>\n'\n '<div class="highlight"><pre><span></span><span class="p">[{</span><br />    '\n '<span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span '\n 'class="mi">1</span><span class="p">,</span><br />    <span '\n 'class="nt">&quot;beta&quot;</span><span class="p">:</span> <span '\n 'class="s2">&quot;this is a string&quot;</span><br /><span '\n 'class="p">}]</span><br /></pre></div>\n'\n '<p><br /></p>')
[   26s] E               <h2 id="an-example-docstring">an example docstring</h2>
[   26s] E               <ul>
[   26s] E               <li>list</li>
[   26s] E               <li>list</li>
[   26s] E               </ul>
[   26s] E               <h3 id="another-header">another header</h3>
[   26s] E               <pre><code>code block
[   26s] E               </code></pre>
[   26s] E               <p>indented</p>
[   26s] E               <h2 id="hash-style-header">hash style header</h2>
[   26s] E             - <div class="highlight"><pre><span></span><span class="p">[{</span><br />    <span class="nt">&quot;alpha&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span><br />    <span class="nt">&quot;beta&quot;</span><span class="p">:</span> <span class="s2">&quot;this is a string&quot;</span><br /><span class="p">}]</span><br /></pre></div>
[   26s] E             - <p><br /></p>
[   26s] E             + <p><code>json
[   26s] E             + [{
[   26s] E             +     "alpha": 1,
[   26s] E             +     "beta": "this is a string"
[   26s] E             + }]</code></p>
[   26s] 
[   26s] tests/test_description.py:168: AssertionError
[   26s] =============================== warnings summary ===============================
[   26s] tests/test_authtoken.py::AuthTokenTests::test_authtoken_can_be_imported_when_not_included_in_installed_apps
[   26s]   /usr/lib/python3.8/site-packages/django/db/models/base.py:364: RuntimeWarning: Model 'authtoken.token' was already registered. Reloading models is not advised as it can lead to inconsistencies, most notably with related models.
[   26s]     new_class._meta.apps.register_model(new_class._meta.app_label, new_class)
[   26s] 
[   26s] tests/test_authtoken.py::AuthTokenTests::test_authtoken_can_be_imported_when_not_included_in_installed_apps
[   26s]   /usr/lib/python3.8/site-packages/django/db/models/base.py:364: RuntimeWarning: Model 'authtoken.tokenproxy' was already registered. Reloading models is not advised as it can lead to inconsistencies, most notably with related models.
[   26s]     new_class._meta.apps.register_model(new_class._meta.app_label, new_class)
[   26s] 
[   26s] -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
[   26s] =========================== short test summary info ============================
[   26s] SKIPPED [1] tests/test_model_serializer.py:462: has models.JSONField
[   26s] ============ 1 failed, 1410 passed, 1 skipped, 2 warnings in 8.06s =============