Preserve line separator format (Unix vs. Windows)

[petsuter]

I tried running python-modernize on Windows on source files that use Unix line separators. This changed them over to Windows line separators. That's not very helpful. A diff now shows every single line as changed. The previous line separator format should be preserved.

[takluyver]

Or maybe we should modernise them all to Unix style ;-)

We'll need to look at 2to3 and see how much code we'd need to duplicate in order to fix this ourselves. If it doesn't require duplicating too much, it should be OK.

[forivall]

For now, you can set your .gitattributes file appropriately so that it ignores line ending type changes and always commits them as unix line separators.

[techtonik]

Most likely that 2to3 need to be opened in binary mode for writing. https://github.com/python/cpython/blob/master/Lib/lib2to3/refactor.py#L527 (write_file)

[daira]

The implementation is different depending on whether python-modernize is running on Python 2 or 3: https://github.com/python/cpython/blob/master/Lib/lib2to3/refactor.py#L113 . On Python 2 at least, setting os.linesep to "\n" will make it use Unix newlines when writing, without any duplication of code.

The Right Thing is to preserve the line-ending style of each file, but that looks significantly more difficult to do, unless we detect the line-ending style ourselves and set os.linesep accordingly.

[techtonik]

@daira why not just read input file as binary? You will get the strings, but without any newline transformations.

[techtonik]

So, it looks like subclassing of lib2to3.RefactoringTool is required.

[takluyver]

2to3 already opens the file as binary, but it normalises the line endings to Unix style before processing the code, and then converts to platform native when writing it again. That presumably means that Windows style line endings can't be passed through 2to3's machinery, or 2to3 wouldn't bother normalising them.

We'd have to subclass RefactoringTool, duplicate _read_python_source, refactor_file, processed_file and write_file to detect and pass through the original newlines. I think the extra code outweighs the benefits, since it's easy to convert back to your preferred line endings after running modernize.

[techtonik]

It is easy to do once, but not every single time. So, how about hack to read the linefeed stats when "-w" is supplied and rewrite of the file after processing?

[takluyver]

It is easy to do once, but not every single time.

That's what scripting is for.

[daira]

As I pointed out, setting os.linesep (as an option, say --linesep=unix or --linesep=windows) may be sufficient to make the existing implementation do what we want.

[Edit: the monkey-patch below is probably better, if it works.]

[petsuter]

It's easy to do, but too much code? Everyone should script it, but doing it right once from the beginning is not worth it? :(

[daira]

I think this should work (untested):

from lib2to3 import refactor

def _identity(obj):
    return obj

refactor._from_system_newlines = _identity
refactor._to_system_newlines = _identity

if sys.version_info >= (3, 0):
    # Force newline mode to '', i.e. 
    # * on input, "universal newline mode is enabled, but line endings 
    #   are returned to the caller untranslated";
    # * on output, "no translation takes place".

    def _open_with_encoding(file, mode='r', buffering=-1, encoding=None, 
                            errors=None, newline=None, closefd=True):
        return open(file, mode=mode, buffering=buffering, encoding=encoding,
                    errors=errors, newline='', closefd=closefd)

    refactor._open_with_encoding = _open_with_encoding