"make tidy" fails on windows

[slaren]

The file licenseck.py (ironically) fails its own license check, apparently because git uses CR LF line endings on windows for .py files with core.autocrlf enabled.

Also note that tidy is a target of check, so in turn this is making "make check" fail.

[slaren]

Adding *.py to the list of extensions in .gitattributes fixes the problem. Another option would be adding a "# xfail-license" tag to licenseck.py like the other .py files have, or making tidy.py remove the CR characters before passing the contents to do_license_check.

[pnkfelix]

part of #8058

[klutzy]

Small fix:

--- a/src/etc/licenseck.py
+++ b/src/etc/licenseck.py
@@ -80,6 +80,7 @@ exceptions = [

 def check_license(name, contents):
     valid_license = False
+    contents = contents.replace("\r\n", "\n")
     for a_valid_license in licenses:
         if contents.startswith(a_valid_license):
             valid_license = True

[huonw]

@klutzy I think

diff --git a/src/etc/tidy.py b/src/etc/tidy.py
old mode 100644
new mode 100755
index 4815bc6..8f5abd4
--- a/src/etc/tidy.py
+++ b/src/etc/tidy.py
@@ -41,6 +41,7 @@ current_contents = ""

 try:
     for line in fileinput.input(file_names,
+                                mode='rU',
                                 openhook=fileinput.hook_encoded("utf-8")):

         if fileinput.filename().find("tidy.py") == -1:

is the more reliable solution. (It uses Python's built-in support to implicitly treat \r, \n and \r\n as newline separators.)

[klutzy]

@huonw I thought yours is better, but actually it does not work.
If openhook=fileinput.hook_encoded(...) is given, hook_encoded uses codecs.open and it ignores any newline conversion.

[adrientetar]

Could that be fixed on the .gitattributes?

* text=auto
*.py eol=lf
https://www.kernel.org/pub/software/scm/git/docs/gitattributes.html

http://stackoverflow.com/a/10017566/2037879

[klutzy]

@adridu59 In the case we have to force lf for nearly all source code, since make tidy reads files and compares its header and predefined license texts.
Anyway, I'm totally ok with forcing lf. In fact we already force it for some files since crlf leaded curious build failure (#8731 / #8738). I guess crlf can break more thing than we know so far.

[adrientetar]

Looking at some refs:
http://stackoverflow.com/questions/2517190/how-do-i-force-git-to-use-lf-instead-of-crlf-under-windows
http://git-scm.com/docs/gitattributes
http://stackoverflow.com/questions/9933004/best-way-to-disable-git-end-of-line-normalization-crlf-to-lf-across-all-clones/10017566#10017566
It looks like * text=auto basically just does the automatic normalization, which means that if your git if configured as lf all files will be marked lf and vice-versa.
Now I think that replacing it with * eol=lf could probably override that by forcing LF on all files. Any thoughts?

EDIT: apparently adding *.* eol=lf should do the trick.
Github has a doc for that.

cc @thestinger