git/cache.h, branch v2.4.9

Merge branch 'jk/index-pack-reduce-recheck' into maint

2015-07-27T19:21:38Z

Disable "have we lost a race with competing repack?" check while receiving a huge object transfer that runs index-pack. * jk/index-pack-reduce-recheck: index-pack: avoid excessive re-reading of pack directory

index-pack: avoid excessive re-reading of pack directory

2015-06-09T19:26:35Z

Since 45e8a74 (has_sha1_file: re-check pack directory before giving up, 2013-08-30), we spend extra effort for has_sha1_file to give the right answer when somebody else is repacking. Usually this effort does not matter, because after finding that the object does not exist, the next step is usually to die(). However, some code paths make a large number of has_sha1_file checks which are _not_ expected to return 1. The collision test in index-pack.c is such a case. On a local system, this can cause a performance slowdown of around 5%. But on a system with high-latency system calls (like NFS), it can be much worse. This patch introduces a "quick" flag to has_sha1_file which callers can use when they would prefer high performance at the cost of false negatives during repacks. There may be other code paths that can use this, but the index-pack one is the most obviously critical, so we'll start with switching that one. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

Merge branch 'pt/xdg-config-path' into maint

2015-06-05T19:00:04Z

Code clean-up for xdg configuration path support. * pt/xdg-config-path: path.c: remove home_config_paths() git-config: replace use of home_config_paths() git-commit: replace use of home_config_paths() credential-store.c: replace home_config_paths() with xdg_config_home() dir.c: replace home_config_paths() with xdg_config_home() attr.c: replace home_config_paths() with xdg_config_home() path.c: implement xdg_config_home() t0302: "unreadable" test needs POSIXPERM t0302: test credential-store support for XDG_CONFIG_HOME git-credential-store: support XDG_CONFIG_HOME git-credential-store: support multiple credential files

Merge branch 'jc/hash-object' into maint

2015-05-26T20:49:25Z

"hash-object --literally" introduced in v2.2 was not prepared to take a really long object type name. * jc/hash-object: write_sha1_file(): do not use a separate sha1[] array t1007: add hash-object --literally tests hash-object --literally: fix buffer overrun with extra-long object type git-hash-object.txt: document --literally option

Merge branch 'jk/prune-mtime' into maint

2015-05-13T21:05:50Z

Access to objects in repositories that borrow from another one on a slow NFS server unnecessarily got more expensive due to recent code becoming more cautious in a naive way not to lose objects to pruning. * jk/prune-mtime: sha1_file: only freshen packs once per run sha1_file: freshen pack objects before loose reachable: only mark local objects as recent

path.c: remove home_config_paths()

2015-05-06T18:33:53Z

home_config_paths() combines distinct functionality already implemented by expand_user_path() and xdg_config_home(), and it also hard-codes the path ~/.gitconfig, which makes it unsuitable to use for other home config file paths. Since its use will just add unnecessary complexity to the code, remove it. Signed-off-by: Paul Tan Signed-off-by: Junio C Hamano

path.c: implement xdg_config_home()

2015-05-06T18:21:04Z

The XDG base dir spec[1] specifies that configuration files be stored in a subdirectory in $XDG_CONFIG_HOME. To construct such a configuration file path, home_config_paths() can be used. However, home_config_paths() combines distinct functionality: 1. Retrieve the home git config file path ~/.gitconfig 2. Construct the XDG config path of the file specified by `file`. This function was introduced in commit 21cf3227 ("read (but not write) from $XDG_CONFIG_HOME/git/config file"). While the intention of the function was to allow the home directory configuration file path and the xdg directory configuration file path to be retrieved with one function call, the hard-coding of the path ~/.gitconfig prevents it from being used for other configuration files. Furthermore, retrieving a file path relative to the user's home directory can be done with expand_user_path(). Hence, it can be seen that home_config_paths() introduces unnecessary complexity, especially if a user just wants to retrieve the xdg config file path. As such, implement a simpler function xdg_config_home() for constructing the XDG base dir spec configuration file path. This function, together with expand_user_path(), can replace all uses of home_config_paths(). [1] http://standards.freedesktop.org/basedir-spec/basedir-spec-0.7.html Helped-by: Eric Sunshine Signed-off-by: Paul Tan Signed-off-by: Junio C Hamano

hash-object --literally: fix buffer overrun with extra-long object type

2015-05-05T17:14:18Z

"hash-object" learned in 5ba9a93 (hash-object: add --literally option, 2014-09-11) to allow crafting a corrupt/broken object of unknown type. When the user-provided type is particularly long, however, it can overflow the relatively small stack-based character array handed to write_sha1_file_prepare() by hash_sha1_file() and write_sha1_file(), leading to stack corruption (and crash). Introduce a custom helper to allow arbitrarily long typenames just for "hash-object --literally". [jc: Eric's original used a strbuf in the more common codepaths, and I rewrote it to avoid penalizing the non-literally code. Bugs are mine] Signed-off-by: Eric Sunshine Signed-off-by: Junio C Hamano

sha1_file: only freshen packs once per run

2015-04-20T20:09:40Z

Since 33d4221 (write_sha1_file: freshen existing objects, 2014-10-15), we update the mtime of existing objects that we would have written out (had they not existed). For the common case in which many objects are packed, we may update the mtime on a single packfile repeatedly. This can result in a noticeable performance problem if calling utime() is expensive (e.g., because your storage is on NFS). We can fix this by keeping a per-pack flag that lets us freshen only once per program invocation. An alternative would be to keep the packed_git.mtime flag up to date as we freshen, and freshen only once every N seconds. In practice, it's not worth the complexity. We are racing against prune expiration times here, which inherently must be set to accomodate reasonable program running times (because they really care about the time between an object being written and it becoming referenced, and the latter is typically the last step a program takes). Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

reachable: only mark local objects as recent

2015-04-20T20:09:27Z

When pruning and repacking a repository that has an alternate object store configured, we may traverse a large number of objects in the alternate. This serves no purpose, and may be expensive to do. A longer explanation is below. Commits d3038d2 and abcb865 taught prune and pack-objects (respectively) to treat "recent" objects as tips for reachability, so that we keep whole chunks of history. They built on the object traversal in 660c889 (sha1_file: add for_each iterators for loose and packed objects, 2014-10-15), which covers both local and alternate objects. In both cases, covering alternate objects is unnecessary, as both commands can only drop objects from the local repository. In the case of prune, we traverse only the local object directory. And in the case of repacking, while we may or may not include local objects in our pack, we will never reach into the alternate with "repack -d". The "-l" option is only a question of whether we are migrating objects from the alternate into our repository, or leaving them untouched. It is possible that we may drop an object that is depended upon by another object in the alternate. For example, imagine two repositories, A and B, with A pointing to B as an alternate. Now imagine a commit that is in B which references a tree that is only in A. Traversing from recent objects in B might prevent A from dropping that tree. But this case isn't worth covering. Repo B should take responsibility for its own objects. It would never have had the commit in the first place if it did not also have the tree, and assuming it is using the same "keep recent chunks of history" scheme, then it would itself keep the tree, as well. So checking the alternate objects is not worth doing, and come with a significant performance impact. In both cases, we skip any recent objects that have already been marked SEEN (i.e., that we know are already reachable for prune, or included in the pack for a repack). So there is a slight waste of time in opening the alternate packs at all, only to notice that we have already considered each object. But much worse, the alternate repository may have a large number of objects that are not reachable from the local repository at all, and we end up adding them to the traversal. We can fix this by considering only local unseen objects. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano