git/object.c, branch v2.41.2

object-file.h: move declarations for object-file.c functions from cache.h

2023-04-11T15:52:10Z

Signed-off-by: Elijah Newren Acked-by: Calvin Wan Signed-off-by: Junio C Hamano

treewide: be explicit about dependence on gettext.h

2023-03-21T17:56:51Z

Dozens of files made use of gettext functions, without explicitly including gettext.h. This made it more difficult to find which files could remove a dependence on cache.h. Make C files explicitly include gettext.h if they are using it. However, while compat/fsmonitor/fsm-ipc-darwin.c should also gain an include of gettext.h, it was left out to avoid conflicting with an in-flight topic. Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

cache.h: remove dependence on hex.h; make other files include it explicitly

2023-02-24T01:25:29Z

Signed-off-by: Elijah Newren Signed-off-by: Junio C Hamano

blob: drop unused parts of parse_blob_buffer()

2022-12-13T13:16:22Z

Our parse_blob_buffer() takes a ptr/len combo, just like parse_tree_buffer(), etc, and returns success or failure. But it doesn't actually do anything with them; we just set the "parsed" flag in the object and return success, without even looking at the contents. There could be some value to keeping these unused parameters: - it's consistent with the parse functions for other object types. But we already lost that consistency in 837d395a5c (Replace parse_blob() with an explanatory comment, 2010-01-18). - As the comment from 837d395a5c explains, callers are supposed to make sure they have the object content available. So in theory asking for these parameters could serve as a signal. But there are only two callers, and one of them always passes NULL (after doing a streaming check of the object hash). This shows that there aren't likely to be a lot of callers (since everyone either uses the type-generic parse functions, or handles blobs individually), and that they need to take special care anyway (because we usually want to avoid loading whole blobs in memory if we can avoid it). So let's just drop these unused parameters, and likewise the useless return value. While we're touching the header file, let's move the declaration of parse_blob_buffer() right below that explanatory comment, where it's more likely to be seen by people looking for the function. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

parse_object(): simplify blob conditional

2022-11-22T01:13:54Z

Commit 8db2dad7a0 (parse_object(): check on-disk type of suspected blob, 2022-11-17) simplified the conditional for checking if we might have a blob. But we can simplify it further. In: !obj || (obj && obj->type == OBJ_BLOB) the short-circuit "OR" means "obj" will always be true on the right-hand side. The compiler almost certainly optimized that out anyway, but dropping it makes the conditional easier to understand for humans. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

parse_object(): check on-disk type of suspected blob

2022-11-18T18:59:31Z

In parse_object(), we try to handle blobs by streaming rather than loading them entirely into memory. The most common case here will be that we haven't seen the object yet and check oid_object_info(), which tells us we have a blob. But we trigger this code on one other case: when we have an in-memory object struct with type OBJ_BLOB (and without its "parsed" flag set, since otherwise we'd return early from the function). This indicates that some other part of the code suspected we have a blob (e.g., it was mentioned by a tree or tag) but we haven't yet looked at the on-disk copy. In this case before hitting the streaming path, we check if we have the object on-disk at all. This is mostly pointless extra work, as the streaming path would complain if it couldn't open the object (albeit with the message "hash mismatch", which is a little misleading). But it's also insufficient to catch all problems. The streaming code will only tell us "yes, the on-disk object matches the oid". But it doesn't actually confirm that what we found was indeed a blob, and neither does repo_has_object_file(). One way to improve this would be to teach stream_object_signature() to check the type (either by returning it to us to check, or taking an "expected" type). But there's an even simpler fix here: if we suspect the object is a blob, just call oid_object_info() to confirm that we have it on-disk, and that it really is a blob. This is slightly less efficient than teaching stream_object_signature() to do it (since it has to open the object already). But this case very rarely comes up. In practice, we usually don't have any clue what the type is, in which case we already call oid_object_info(). This "suspected" case happens only when some other code created an object struct but didn't actually parse the blob, which is actually tricky to trigger at all (see the discussion of the test below). I reworked the conditional a bit so that instead of: if ((suspected_blob && oid_object_info() == OBJ_BLOB) (no_clue && oid_object_info() == OBJ_BLOB) we have the simpler: if ((suspected_blob || no_clue) && oid_object_info() == OBJ_BLOB) This is shorter, but also reflects what we really want say, which is "have we ruled out this being a blob; if not, check it on-disk". In either case, if oid_object_info() fails to tell us it's a blob, we'll skip the streaming code path and call repo_read_object_file(), just as before. And if we really do have a mismatch with the existing object struct, we'll eventually call lookup_commit(), etc, via parse_object_buffer(), which will complain that it doesn't match our existing obj->type. So this fixes one of the lingering expect_failure cases from 0616617c7e (t: introduce tests for unexpected object types, 2019-04-09). That test works by peeling a tag that claims to point to a blob (triggering us to create the struct), but really points to something else, which we later discover when we call parse_object() as part of the actual traversal). Prior to this commit, we'd quietly check the sha1 and mark the blob as "parsed". Now we correctly complain about the mismatch. Signed-off-by: Jeff King Signed-off-by: Taylor Blau

parse_object(): drop extra "has" check before checking object type

2022-11-18T18:59:31Z

When parsing an object of unknown type, we check to see if it's a blob, so we can use our streaming code path. This uses oid_object_info() to check the type, but before doing so we call repo_has_object_file(). This latter is pointless, as oid_object_info() will already fail if the object is missing. Checking it ahead of time just complicates the code and is a waste of resources (albeit small). Let's drop the redundant check. Signed-off-by: Jeff King Signed-off-by: Taylor Blau

Merge branch 'jk/fsck-on-diet'

2022-10-10T17:08:39Z

"git fsck" failed to release contents of tree objects already used from the memory, which has been fixed. * jk/fsck-on-diet: parse_object_buffer(): respect save_commit_buffer fsck: turn off save_commit_buffer fsck: free tree buffers after walking unreachable objects

parse_object_buffer(): respect save_commit_buffer

2022-09-22T18:40:47Z

If the global variable "save_commit_buffer" is set to 0, then parse_commit() will throw away the commit object data after parsing it, rather than sticking it into a commit slab. This goes all the way back to 60ab26de99 ([PATCH] Avoid wasting memory in git-rev-list, 2005-09-15). But there's another code path which may similarly stash the buffer: parse_object_buffer(). This is where we end up if we parse a commit via parse_object(), and it's used directly in a few other code paths like git-fsck. The original goal of 60ab26de99 was avoiding extra memory usage for rev-list. And there it's not all that important to catch parse_object(). We use that function only for looking at the tips of the traversal, and the majority of the commits are parsed by following parent links, where we use parse_commit() directly. So we were wasting some memory, but only a small portion. It's much easier to see the effect with fsck. Since we now turn off save_commit_buffer by default there, we _should_ be able to drop the freeing of the commit buffer in fsck_obj(). But if we do so (taking the first hunk of this patch without the rest), then the peak heap of "git fsck" in a clone of git.git goes from 136MB to 194MB. Teaching parse_object_buffer() to respect save_commit_buffer brings that down to 134.5MB (it's hard to tell from massif's output, but I suspect the savings comes from avoiding the overhead of the mostly-empty commit slab). Other programs should see a small improvement. Both "rev-list --all" and "fsck --connectivity-only" improve by a few hundred kilobytes, as they'd avoid loading the tip objects of their traversals. Most importantly, no code should be hurt by doing this. Any program that turns off save_commit_buffer is already making the assumption that any commit it sees may need to have its object data loaded on demand, as it doesn't know which ones were parsed by parse_commit() versus parse_object(). Not to mention that anything parsed by the commit graph may be in the same boat, even if save_commit_buffer was not disabled. This should be the only spot that needs to be fixed. Grepping for set_commit_buffer() shows that this and parse_commit() are the only relevant calls. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano

parse_object(): check commit-graph when skip_hash set

2022-09-07T19:27:02Z

If the caller told us that they don't care about us checking the object hash, then we're free to implement any optimizations that get us the parsed value more quickly. An obvious one is to check the commit graph before loading an object from disk. And in fact, both of the callers who pass in this flag are already doing so before they call parse_object()! So we can simplify those callers, as well as any possible future ones, by moving the logic into parse_object(). There are two subtle things to note in the diff, but neither has any impact in practice: - it seems least-surprising here to do the graph lookup on the git-replace'd oid, rather than the original. This is in theory a change of behavior from the earlier code, as neither caller did a replace lookup itself. But in practice it doesn't matter, as we disable the commit graph entirely if there are any replace refs. - the caller in get_reference() passes the skip_hash flag only if revs->verify_objects isn't set, whereas it would look in the commit graph unconditionally. In practice this should not matter as we should disable the commit graph entirely when using verify_objects (and that was done recently in another patch). So this should be a pure cleanup with no behavior change. Signed-off-by: Jeff King Signed-off-by: Junio C Hamano