git/pack-write.c, branch v2.32.2

Merge branch 'tb/reverse-midx'

2021-04-08T20:23:25Z

An on-disk reverse-index to map the in-pack location of an object back to its object name across multiple packfiles is introduced. * tb/reverse-midx: midx.c: improve cache locality in midx_pack_order_cmp() pack-revindex: write multi-pack reverse indexes pack-write.c: extract 'write_rev_file_order' pack-revindex: read multi-pack reverse indexes Documentation/technical: describe multi-pack reverse indexes midx: make some functions non-static midx: keep track of the checksum midx: don't free midx_name early midx: allow marking a pack as preferred t/helper/test-read-midx.c: add '--show-objects' builtin/multi-pack-index.c: display usage on unrecognized command builtin/multi-pack-index.c: don't enter bogus cmd_mode builtin/multi-pack-index.c: split sub-commands builtin/multi-pack-index.c: define common usage with a macro builtin/multi-pack-index.c: don't handle 'progress' separately builtin/multi-pack-index.c: inline 'flags' with options

pack-write.c: extract 'write_rev_file_order'

2021-04-01T20:07:37Z

Existing callers provide the reverse index code with an array of 'struct pack_idx_entry *'s, which is then sorted by pack order (comparing the offsets of each object within the pack). Prepare for the multi-pack index to write a .rev file by providing a way to write the reverse index without an array of pack_idx_entry (which the MIDX code does not have). Instead, callers can invoke 'write_rev_index_positions()', which takes an array of uint32_t's. The ith entry in this array specifies the ith object's (in index order) position within the pack (in pack order). Expose this new function for use in a later patch, and rewrite the existing write_rev_file() in terms of this new function. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

Merge branch 'jt/transfer-fsck-across-packs'

2021-03-01T22:02:57Z

The approach to "fsck" the incoming objects in "index-pack" is attractive for performance reasons (we have them already in core, inflated and ready to be inspected), but fundamentally cannot be applied fully when we receive more than one pack stream, as a tree object in one pack may refer to a blob object in another pack as ".gitmodules", when we want to inspect blobs that are used as ".gitmodules" file, for example. Teach "index-pack" to emit objects that must be inspected later and check them in the calling "fetch-pack" process. * jt/transfer-fsck-across-packs: fetch-pack: print and use dangling .gitmodules fetch-pack: with packfile URIs, use index-pack arg http-fetch: allow custom index-pack args http: allow custom index-pack args

fetch-pack: print and use dangling .gitmodules

2021-02-22T20:07:40Z

Teach index-pack to print dangling .gitmodules links after its "keep" or "pack" line instead of declaring an error, and teach fetch-pack to check such lines printed. This allows the tree side of the .gitmodules link to be in one packfile and the blob side to be in another without failing the fsck check, because it is now fetch-pack which checks such objects after all packfiles have been downloaded and indexed (and not index-pack on an individual packfile, as it is before this commit). Signed-off-by: Jonathan Tan Signed-off-by: Junio C Hamano

pack-write.c: prepare to write 'pack-*.rev' files

2021-01-26T02:32:43Z

This patch prepares for callers to be able to write reverse index files to disk. It adds the necessary machinery to write a format-compliant .rev file from within 'write_rev_file()', which is called from 'finish_tmp_packfile()'. Similar to the process by which the reverse index is computed in memory, these new paths also have to sort a list of objects by their offsets within a packfile. These new paths use a qsort() (as opposed to a radix sort), since our specialized radix sort requires a full revindex_entry struct per object, which is more memory than we need to allocate. The qsort is obviously slower, but the theoretical slowdown would require a repository with a large amount of objects, likely implying that the time spent in, say, pack-objects during a repack would dominate the overall runtime. Signed-off-by: Taylor Blau Signed-off-by: Junio C Hamano

pack-write: die on error in write_promisor_file()

2021-01-15T01:02:22Z

write_promisor_file() already uses xfopen(), so it would die if the file cannot be opened for writing. To be consistent with this behavior and not overlook issues, let's also die if there are errors when we are actually writing to the file. Suggested-by: Jeff King Suggested-by: Taylor Blau Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano

fetch-pack: refactor writing promisor file

2021-01-13T00:01:07Z

Let's replace the 2 different pieces of code that write a promisor file in 'builtin/repack.c' and 'fetch-pack.c' with a new function called 'write_promisor_file()' in 'pack-write.c' and 'pack.h'. This might also help us in the future, if we want to put back the ref names and associated hashes that were in the promisor files we are repacking in 'builtin/repack.c' as suggested by a NEEDSWORK comment just above the code we are refactoring. Signed-off-by: Christian Couder Signed-off-by: Junio C Hamano

Merge branch 'rs/hashwrite-be64'

2020-11-25T23:24:52Z

Code simplification. * rs/hashwrite-be64: pack-write: use hashwrite_be64() midx: use hashwrite_be64() csum-file: add hashwrite_be64()

pack-write: use hashwrite_be64()

2020-11-12T17:40:10Z

Call hashwrite_be64() to write a 64-bit value instead of open-coding it using htonl() and hashwrite(). This shortens the code, gets rid of a buffer and several magic numbers, and makes the intent clearer. Signed-off-by: René Scharfe Signed-off-by: Junio C Hamano

pack-write: use hashwrite_be32() instead of double-buffering array

2020-11-01T23:52:51Z

hashwrite() already buffers writes, so pass the fanout table entries individually via hashwrite_be32(), which also does the endianess conversion for us. This avoids a memory copy, shortens the code and reduces the number of magic numbers. Signed-off-by: René Scharfe Signed-off-by: Junio C Hamano