git/convert.c, branch v2.18.2

Merge branch 'ls/checkout-encoding'

2018-05-08T06:59:22Z

The new "checkout-encoding" attribute can ask Git to convert the contents to the specified encoding when checking out to the working tree (and the other way around when checking in). * ls/checkout-encoding: convert: add round trip check based on 'core.checkRoundtripEncoding' convert: add tracing for 'working-tree-encoding' attribute convert: check for detectable errors in UTF encodings convert: add 'working-tree-encoding' attribute utf8: add function to detect a missing UTF-16/32 BOM utf8: add function to detect prohibited UTF-16/32 BOM utf8: teach same_encoding() alternative UTF encoding names strbuf: add a case insensitive starts_with() strbuf: add xstrdup_toupper() strbuf: remove unnecessary NUL assignment in xstrdup_tolower()

convert: add round trip check based on 'core.checkRoundtripEncoding'

2018-04-16T02:40:56Z

UTF supports lossless conversion round tripping and conversions between UTF and other encodings are mostly round trip safe as Unicode aims to be a superset of all other character encodings. However, certain encodings (e.g. SHIFT-JIS) are known to have round trip issues [1]. Add 'core.checkRoundtripEncoding', which contains a comma separated list of encodings, to define for what encodings Git should check the conversion round trip if they are used in the 'working-tree-encoding' attribute. Set SHIFT-JIS as default value for 'core.checkRoundtripEncoding'. [1] https://support.microsoft.com/en-us/help/170559/prb-conversion-problem-between-shift-jis-and-unicode Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

convert: add tracing for 'working-tree-encoding' attribute

2018-04-16T02:40:56Z

Add the GIT_TRACE_WORKING_TREE_ENCODING environment variable to enable tracing for content that is reencoded with the 'working-tree-encoding' attribute. This is useful to debug encoding issues. Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

convert: check for detectable errors in UTF encodings

2018-04-16T02:40:56Z

Check that new content is valid with respect to the user defined 'working-tree-encoding' attribute. Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

convert: add 'working-tree-encoding' attribute

2018-04-16T02:40:56Z

Git recognizes files encoded with ASCII or one of its supersets (e.g. UTF-8 or ISO-8859-1) as text files. All other encodings are usually interpreted as binary and consequently built-in Git text processing tools (e.g. 'git diff') as well as most Git web front ends do not visualize the content. Add an attribute to tell Git what encoding the user has defined for a given file. If the content is added to the index, then Git reencodes the content to a canonical UTF-8 representation. On checkout Git will reverse this operation. Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

convert: convert to struct object_id

2018-03-14T16:23:50Z

Convert convert.c to struct object_id. Add a use of the_hash_algo to replace hard-coded constants and change a strbuf_add to a strbuf_addstr to avoid another hard-coded constant. Note that a strict conversion using the hexsz constant would cause problems in the future if the internal and user-visible hash algorithms differed, as anticipated by the hash function transition plan. Signed-off-by: brian m. carlson Signed-off-by: Junio C Hamano

Merge branch 'po/object-id'

2018-02-15T22:55:43Z

Conversion from uchar[20] to struct object_id continues. * po/object-id: sha1_file: rename hash_sha1_file_literally sha1_file: convert write_loose_object to object_id sha1_file: convert force_object_loose to object_id sha1_file: convert write_sha1_file to object_id notes: convert write_notes_tree to object_id notes: convert combine_notes_* to object_id commit: convert commit_tree* to object_id match-trees: convert splice_tree to object_id cache: clear whole hash buffer with oidclr sha1_file: convert hash_sha1_file to object_id dir: convert struct sha1_stat to use object_id sha1_file: convert pretend_sha1_file to object_id

sha1_file: convert hash_sha1_file to object_id

2018-01-30T18:42:36Z

Convert the declaration and definition of hash_sha1_file to use struct object_id and adjust all function calls. Rename this function to hash_object_file. Signed-off-by: Patryk Obara Signed-off-by: Junio C Hamano

convert_to_git(): safe_crlf/checksafe becomes int conv_flags

2018-01-16T20:35:56Z

When calling convert_to_git(), the checksafe parameter defined what should happen if the EOL conversion (CRLF --> LF --> CRLF) does not roundtrip cleanly. In addition, it also defined if line endings should be renormalized (CRLF --> LF) or kept as they are. checksafe was an safe_crlf enum with these values: SAFE_CRLF_FALSE: do nothing in case of EOL roundtrip errors SAFE_CRLF_FAIL: die in case of EOL roundtrip errors SAFE_CRLF_WARN: print a warning in case of EOL roundtrip errors SAFE_CRLF_RENORMALIZE: change CRLF to LF SAFE_CRLF_KEEP_CRLF: keep all line endings as they are In some cases the integer value 0 was passed as checksafe parameter instead of the correct enum value SAFE_CRLF_FALSE. That was no problem because SAFE_CRLF_FALSE is defined as 0. FALSE/FAIL/WARN are different from RENORMALIZE and KEEP_CRLF. Therefore, an enum is not ideal. Let's use a integer bit pattern instead and rename the parameter to conv_flags to make it more generically usable. This allows us to extend the bit pattern in a subsequent commit. Reported-By: Randall S. Becker Helped-By: Lars Schneider Signed-off-by: Torsten Bögershausen Signed-off-by: Lars Schneider Signed-off-by: Junio C Hamano

Merge branch 'tb/check-crlf-for-safe-crlf'

2017-12-27T19:16:21Z

The "safe crlf" check incorrectly triggered for contents that does not use CRLF as line endings, which has been corrected. * tb/check-crlf-for-safe-crlf: t0027: Adapt the new MIX tests to Windows convert: tighten the safe autocrlf handling