git/shared.mak, branch v2.45.4

Makefile: simplify output of the libpath_template

2024-01-31T22:43:00Z

If a platform lacks the support to specify the dynamic library path, there is no suitable value to give to the CC_LD_DYNPATH variable. Allow them to be set to an empty string to signal that they do not need to add the usual -Wl,-rpath, or -R or whatever option followed by a directory name. This way, $(call libpath_template,$(SOMELIBDIR)) would expand to just a single mention of that directory, i.e. -L$(SOMELIBDIR) when CC_LD_DYNPATH is set to an empty string (or a "-L", which would have repeated the same "-L$(SOMELIBDIR)" twice without any ill effect). Signed-off-by: Junio C Hamano

Makefile: reduce repetitive library paths

2024-01-31T18:01:56Z

When we take a library package we depend on (e.g., LIBPCRE) from a directory other than the default location of the system, we add the same directory twice on the linker command like, like so: EXTLIBS += -L$(LIBPCREDIR)/$(lib) $(CC_LD_DYNPATH)$(LIBPCREDIR)/$(lib) Introduce a template "libpath_template" that takes the path to the directory, which can be used like so: EXTLIBS += $(call libpath_template,$(LIBPCREDIR)/$(lib)) and expand it into the "-L$(DIR) $(CC_LD_DYNPATH)$(DIR)" form. Hopefully we can reduce the chance of typoes this way. Signed-off-by: Junio C Hamano

Merge branch 'ab/gnumake-4.4-fix'

2022-12-01T09:38:07Z

Adjust our Makefiles for GNUmake 4.4 * ab/gnumake-4.4-fix: Makefiles: change search through $(MAKEFLAGS) for GNU make 4.4

Makefiles: change search through $(MAKEFLAGS) for GNU make 4.4

2022-11-30T22:24:12Z

Since GNU make 4.4 the semantics of the $(MAKEFLAGS) variable has changed in a backward-incompatible way, as its "NEWS" file notes: Previously only simple (one-letter) options were added to the MAKEFLAGS variable that was visible while parsing makefiles. Now, all options are available in MAKEFLAGS. If you want to check MAKEFLAGS for a one-letter option, expanding "$(firstword -$(MAKEFLAGS))" is a reliable way to return the set of one-letter options which can be examined via findstring, etc. This upstream change meant that e.g.: make man Would become very noisy, because in shared.mak we rely on extracting "s" from the $(MAKEFLAGS), which now contains long options like "--jobserver-auth=fifo:", which we'll conflate with the "-s" option. So, let's change this idiom we've been carrying since [1], [2] and [3] as the "NEWS" suggests. Note that the "-" in "-$(MAKEFLAGS)" is critical here, as the variable will always contain leading whitespace if there are no short options, but long options are present. Without it e.g. "make --debug=all" would yield "--debug=all" as the first word, but with it we'll get "-" as intended. Then "-s" for "-s", "-Bs" for "-s -B" etc. 1. 0c3b4aac8ec (git-gui: Support of "make -s" in: do not output anything of the build itself, 2007-03-07) 2. b777434383b (Support of "make -s": do not output anything of the build itself, 2007-03-07) 3. bb2300976ba (Documentation/Makefile: make most operations "quiet", 2009-03-27) Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

Makefile: don't create a ".build/.build/" for cocci, fix output

2022-11-11T22:21:45Z

Fix a couple of issues in the recently merged 0f3c55d4c2b (Merge branch 'ab/coccicheck-incremental' into next, 2022-11-08): In copying over the "contrib/coccinelle/" rules to ".build/contrib/coccinelle/" we inadvertently ended up with a ".build/.build/contrib/coccinelle/" as well. We'd generate the per-file patches in the former, and keep the rule and overall result in the latter. E.g. running: make contrib/coccinelle/free.cocci.patch COCCI_SOURCES="attr.c grep.c" Would, per "tree -a .build" yield the following result: .build ├── .build │ └── contrib │ └── coccinelle │ └── free.cocci.patch │ ├── attr.c │ ├── attr.c.log │ ├── grep.c │ └── grep.c.log └── contrib └── coccinelle ├── FOUND_H_SOURCES ├── free.cocci └── free.cocci.patch Now we'll instead generate all of our files in ".build/contrib/coccinelle/". Fixing this required renaming the directory where we keep our per-file patches, as we'd otherwise conflict with the result. Now the per-file patch directory is named e.g. "free.cocci.d". And the end result will now be: .build └── contrib └── coccinelle ├── FOUND_H_SOURCES ├── free.cocci ├── free.cocci.d │ ├── attr.c.patch │ ├── attr.c.patch.log │ ├── grep.c.patch │ └── grep.c.patch.log └── free.cocci.patch The per-file patches now have a ".patch" file suffix, which fixes another issue reported against 0f3c55d4c2b: The summary output was confusing. Before for the "make" command above we'd emit: [...] MKDIR -p .build/contrib/coccinelle CP contrib/coccinelle/free.cocci .build/contrib/coccinelle/free.cocci GEN .build/contrib/coccinelle/FOUND_H_SOURCES MKDIR -p .build/.build/contrib/coccinelle/free.cocci.patch SPATCH .build/.build/contrib/coccinelle/free.cocci.patch/grep.c SPATCH .build/.build/contrib/coccinelle/free.cocci.patch/attr.c SPATCH CAT $^ >.build/contrib/coccinelle/free.cocci.patch CP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch But now we'll instead emit (identical output at the start omitted): [...] MKDIR -p .build/contrib/coccinelle/free.cocci.d SPATCH grep.c >.build/contrib/coccinelle/free.cocci.d/grep.c.patch SPATCH attr.c >.build/contrib/coccinelle/free.cocci.d/attr.c.patch SPATCH CAT .build/contrib/coccinelle/free.cocci.d/**.patch >.build/contrib/coccinelle/free.cocci.patch CP .build/contrib/coccinelle/free.cocci.patch contrib/coccinelle/free.cocci.patch I.e. we have an "SPATCH" line that makes it clear that we're running against the "{attr,grep}.c" file. The "SPATCH CAT" is then altered to correspond to it, showing that we're concatenating the "free.cocci.d/**.patch" files into one generated "free.cocci.patch" at the end. Reported-by: SZEDER Gábor Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Taylor Blau

Makefile: copy contrib/coccinelle/*.cocci to build/

2022-11-03T01:22:16Z

Change the "coccinelle" rule so that we first copy the *.cocci source in e.g. "contrib/coccinelle/strbuf.cocci" to ".build/contrib/coccinelle/strbuf.cocci" before operating on it. For now this serves as a rather pointless indirection, but prepares us for the subsequent commit where we'll be able to inject generated *.cocci files. Having the entire dependency tree live inside .build/* simplifies both the globbing we'd need to do, and any "clean" rules. It will also help for future targets which will want to act on the generated patches or the logs, e.g. targets to alert if we can't parse certain files (or, less so than usual) with "spatch", and e.g. a replacement for "ci/run-static-analysis.sh". Such a replacement won't care about placing the patches in the in-tree, only whether they're "OK" (and about the diff). Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Taylor Blau

cocci: make "coccicheck" rule incremental

2022-11-03T01:22:16Z

Optimize the very slow "coccicheck" target to take advantage of incremental rebuilding, and fix outstanding dependency problems with the existing rule. The rule is now faster both on the initial run as we can make better use of GNU make's parallelism than the old ad-hoc combination of make's parallelism combined with $(SPATCH_BATCH_SIZE) and/or the "--jobs" argument to "spatch(1)". It also makes us *much* faster when incrementally building, it's now viable to "make coccicheck" as topic branches are merged down. The rule didn't use FORCE (or its equivalents) before, so a: make coccicheck make coccicheck Would report nothing to do on the second iteration. But all of our patch output depended on all $(COCCI_SOURCES) files, therefore e.g.: make -W grep.c coccicheck Would do a full re-run, i.e. a a change in a single file would force us to do a full re-run. The reason for this (not the initial rationale, but my analysis) is: * Since we create a single "*.cocci.patch+" we don't know where to pick up where we left off, or how to incrementally merge e.g. a "grep.c" change with an existing *.cocci.patch. * We've been carrying forward the dependency on the *.c files since 63f0a758a06 (add coccicheck make target, 2016-09-15) the rule was initially added as a sort of poor man's dependency discovery. As we don't include other *.c files depending on other *.c files has always been broken, as could be trivially demonstrated e.g. with: make coccicheck make -W strbuf.h coccicheck However, depending on the corresponding *.c files has been doing something, namely that *if* an API change modified both *.c and *.h files we'd catch the change to the *.h we care about via the *.c being changed. For API changes that happened only via *.h files we'd do the wrong thing before this change, but e.g. for function additions (not "static inline" ones) catch the *.h change by proxy. Now we'll instead: * Create a / pair in the .build directory, E.g. for swap.cocci and grep.c we'll create .build/contrib/coccinelle/swap.cocci.patch/grep.c. That file is the diff we'll apply for that - combination, if there's no changes to me made (the common case) it'll be an empty file. * Our generated *.patch file (e.g. contrib/coccinelle/swap.cocci.patch) is now a simple "cat $^" of all of all of the / files for a given . In the case discussed above of "grep.c" being changed we'll do the full "cat" every time, so they resulting *.cocci.patch will always be correct and up-to-date, even if it's "incrementally updated". See 1cc0425a27c (Makefile: have "make pot" not "reset --hard", 2022-05-26) for another recent rule that used that technique. As before we'll: * End up generating a contrib/coccinelle/swap.cocci.patch, if we "fail" by creating a non-empty patch we'll still exit with a zero exit code. Arguably we should move to a more Makefile-native way of doing this, i.e. fail early, and if we want all of the "failed" changes we can use "make -k", but as the current "ci/run-static-analysis.sh" expects us to behave this way let's keep the existing behavior of exhaustively discovering all cocci changes, and only failing if spatch itself errors out. Further implementation details & notes: * Before this change running "make coccicheck" would by default end up pegging just one CPU at the very end for a while, usually as we'd finish whichever *.cocci rule was the most expensive. This could be mitigated by combining "make -jN" with SPATCH_BATCH_SIZE, see 960154b9c17 (coccicheck: optionally batch spatch invocations, 2019-05-06). There will be cases where getting rid of "SPATCH_BATCH_SIZE" makes things worse, but a from-scratch "make coccicheck" with the default of SPATCH_BATCH_SIZE=1 (and tweaking it doesn't make a difference) is faster (~3m36s v.s. ~3m56s) with this approach, as we can feed the CPU more work in a less staggered way. * Getting rid of "SPATCH_BATCH_SIZE" particularly helps in cases where the default of 1 yields parallelism under "make coccicheck", but then running e.g.: make -W contrib/coccinelle/swap.cocci coccicheck I.e. before that would use only one CPU core, until the user remembered to adjust "SPATCH_BATCH_SIZE" differently than the setting that makes sense when doing a non-incremental run of "make coccicheck". * Before the "make coccicheck" rule would have to clean "contrib/coccinelle/*.cocci.patch*", since we'd create "*+" and "*.log" files there. Now those are created in .build/contrib/coccinelle/, which is covered by the "cocciclean" rule already. Outstanding issues & future work: * We could get rid of "--all-includes" in favor of manually specifying a list of includes to give to "spatch(1)". As noted upthread of [1] a naïve removal of "--all-includes" will result in broken *.cocci patches, but if we know the exhaustive list of includes via COMPUTE_HEADER_DEPENDENCIES we don't need to re-scan for them, we could grab the headers to include from the .depend.d/.o.d and supply them with the "--include" option to spatch(1).q 1. https://lore.kernel.org/git/87ft18tcog.fsf@evledraar.gmail.com/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Taylor Blau

Makefile + shared.mak: rename and indent $(QUIET_SPATCH_T)

2022-11-03T01:22:15Z

In f7ff6597a75 (cocci: add a "coccicheck-test" target and test *.cocci rules, 2022-07-05) we abbreviated "_TEST" to "_T" to have it align with the rest of the "="'s above it. Subsequent commits will add more QUIET_SPATCH_* variables, so let's stop abbreviating this, and indent it in preparation for adding more of these variables. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Taylor Blau

cocci: add a "coccicheck-test" target and test *.cocci rules

2022-07-06T19:24:43Z

Add a "coccicheck-test" target to test our *.cocci rules, and as a demonstration add tests for the rules added in 39ea59a2570 (remove unnecessary NULL check before free(3), 2016-10-08) and 1b83d1251ed (coccinelle: add a rule to make "expression" code use FREE_AND_NULL(), 2017-06-15). I considered making use of the "spatch --test" option, and the choice of a "tests" over a "t" directory is to make these tests compatible with such a future change. Unfortunately "spatch --test" doesn't return meaningful exit codes, AFAICT you need to "grep" its output to see if the *.res is what you expect. There's "--test-okfailed", but I didn't find a way to sensibly integrate those (it relies on some in-between status files, but doesn't help with the status codes). Instead let's use a "--sp-file" pattern similar to the main "coccicheck" rule, with the difference that we use and compare the two *.res files with cmp(1). The --very-quiet and --no-show-diff options ensure that we don't need to pipe stdout and stderr somewhere. Unlike the "%.cocci.patch" rule we're not using the diff. The "cmp || git diff" is optimistically giving us better output on failure, but even if we only have POSIX cmp and no system git installed we'll still fail with the "cmp", just with an error message that isn't as friendly. The "2>/dev/null" is in case we don't have a "git" installed. Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano

Makefile: add "po-init" rule to initialize po/XX.po

2022-05-26T17:32:57Z

The core translation is the minimum set of work that must be done for a new language translation. There are over 5000 messages in the template message file "po/git.pot" that need to be translated. It is not a piece of cake for such a huge workload. So we used to define a small set of messages called "core translation" that a new l10n contributor must complete before sending pull request to the l10n coordinator. By pulling in some parts of the git-po-helper[^1] logic, we add a new rule to create this core translation message "po/git-core.pot": make po/git-core.pot To help new l10n contributors to initialized their "po/XX.pot" from "po/git-core.pot", we also add new rules "po-init": make po-init PO_FILE=po/XX.po [^1]: https://github.com/git-l10n/git-po-helper/ Signed-off-by: Ævar Arnfjörð Bjarmason Signed-off-by: Junio C Hamano