git/userdiff.h, branch v2.41.2

Merge branch 'rs/userdiff-multibyte-regex'

2023-04-20T21:33:35Z

The userdiff regexp patterns for various filetypes that are built into the system have been updated to avoid triggering regexp errors from UTF-8 aware regex engines. * rs/userdiff-multibyte-regex: userdiff: support regexec(3) with multi-byte support

userdiff: support regexec(3) with multi-byte support

2023-04-07T14:38:09Z

Since 1819ad327b (grep: fix multibyte regex handling under macOS, 2022-08-26) we use the system library for all regular expression matching on macOS, not just for git grep. It supports multi-byte strings and rejects invalid multi-byte characters. This broke all built-in userdiff word regexes in UTF-8 locales because they all include such invalid bytes in expressions that are intended to match multi-byte characters without explicit support for that from the regex engine. "|[^[:space:]]|[\xc0-\xff][\x80-\xbf]+" is added to all built-in word regexes to match a single non-space or multi-byte character. The \xNN characters are invalid if interpreted as UTF-8 because they have their high bit set, which indicates they are part of a multi-byte character, but they are surrounded by single-byte characters. Replace that expression with "|[^[:space:]]" if the regex engine supports multi-byte matching, as there is no need to have an explicit range for multi-byte characters then. Check for that capability at runtime, because it depends on the locale and thus on environment variables. Construct the full replacement expression at build time and just switch it in if necessary to avoid string manipulation and allocations at runtime. Additionally the word regex for tex contains the expression "[a-zA-Z0-9\x80-\xff]+" with a similarly invalid range. The best replacement with only valid characters that I can come up with is "([a-zA-Z0-9]|[^\x01-\x7f])+". Unlike the original it matches NUL characters, though. Assuming that tex files usually don't contain NUL this should be acceptable. Reported-by: D. Ben Knoble Reported-by: Eric Sunshine Helped-by: Junio C Hamano Signed-off-by: René Scharfe Signed-off-by: Junio C Hamano

diff: teach diff to read algorithm from diff driver

2023-02-21T17:29:10Z

It can be useful to specify diff algorithms per file type. For example, one may want to use the minimal diff algorithm for .json files, another for .c files, etc. The diff machinery already checks attributes for a diff driver. Teach the diff driver parser a new type "algorithm" to look for in the config, which will be used if a driver has been specified through the attributes. Enforce precedence of the diff algorithm by favoring the command line option, then looking at the driver attributes & config combination, then finally the diff.algorithm config. To enforce precedence order, use a new `ignore_driver_algorithm` member during options parsing to indicate the diff algorithm was set via command line args. Signed-off-by: John Cai Signed-off-by: Junio C Hamano

userdiff: add and use for_each_userdiff_driver()

2021-04-08T19:19:10Z

Refactor the userdiff_find_by_namelen() function so that a new for_each_userdiff_driver() API function does most of the work. This will be useful for the same reason we've got other for_each_*() API functions as part of various APIs, and will be used in a follow-up commit. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

notes-cache.c: remove the_repository references

2018-11-12T05:50:06Z

Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

userdiff.c: remove implicit dependency on the_index

2018-09-21T16:50:58Z

[jc: squashed in missing forward decl in userdiff.h found by Ramsay] Helped-by: Ramsay Jones Signed-off-by: Nguyễn Thái Ngọc Duy Signed-off-by: Junio C Hamano

diff: clarify textconv interface

2016-02-22T18:40:35Z

The memory allocation scheme for the textconv interface is a bit tricky, and not well documented. It was originally designed as an internal part of diff.c (matching fill_mmfile), but gradually was made public. Refactoring it is difficult, but we can at least improve the situation by documenting the intended flow and enforcing it with an in-code assertion. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

refactor get_textconv to not require diff_filespec

2011-05-23T22:46:02Z

This function actually does two things: 1. Load the userdiff driver for the filespec. 2. Decide whether the driver has a textconv component, and initialize the textconv cache if applicable. Only part (1) requires the filespec object, and some callers may not have a filespec at all. So let's split them it into two functions, and put part (2) with the userdiff code, which is a better fit. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

diff: cache textconv output

2010-04-02T07:05:31Z

Running a textconv filter can take a long time. It's particularly bad for a large file which needs to be spooled to disk, but even for small files, the fork+exec overhead can add up for something like "git log -p". This patch uses the notes-cache mechanism to keep a fast cache of textconv output. Caches are stored in refs/notes/textconv/$x, where $x is the userdiff driver defined in gitattributes. Caching is enabled only if diff.$x.cachetextconv is true. In my test repo, on a commit with 45 jpg and avi files changed and a textconv to show their exif tags: [before] $ time git show >/dev/null real 0m13.724s user 0m12.057s sys 0m1.624s [after, first run] $ git config diff.mfo.cachetextconv true $ time git show >/dev/null real 0m14.252s user 0m12.197s sys 0m1.800s [after, subsequent runs] $ time git show >/dev/null real 0m0.352s user 0m0.148s sys 0m0.200s So for a slight (3.8%) cost on the first run, we achieve an almost 40x speed up on subsequent runs. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

color-words: make regex configurable via attributes

2009-01-17T18:44:21Z

Make the --color-words splitting regular expression configurable via the diff driver's 'wordregex' attribute. The user can then set the driver on a file in .gitattributes. If a regex is given on the command line, it overrides the driver's setting. We also provide built-in regexes for the languages that already had funcname patterns, and add an appropriate diff driver entry for C/++. (The patterns are designed to run UTF-8 sequences into a single chunk to make sure they remain readable.) Signed-off-by: Thomas Rast Signed-off-by: Junio C Hamano