coreutils/src/sort.c, branch v8.8

maint: fix a typo in sort --parallel help message

2010-12-20T10:56:04Z

Also fix up Chen Guo's contacts * src/sort.c (usage): Add a missing "of" * THANKS: Add Chen Guo * .mailmap: Add Chen Guo's UCLA address

sort: use at most 8 threads by default

2010-12-19T00:33:45Z

* src/sort.c (main): If --parallel isn't specified, restrict the number of threads to 8 by default. If the --parallel option is specified, then allow any number of threads to be set, independent of the number of processors on the system. * doc/coreutils.texi (sort invocation): Document the changes to determining the number of threads to use. Mention the memory overhead when using multiple threads. * tests/misc/sort-spinlock-abuse: Allow single core systems that support pthreads. * tests/misc/sort-stale-thread-mem: Likewise. * tests/misc/sort-unique-segv: Likewise. * NEWS: Mention the change in behaviour.

sort: do not generate thousands of subprocesses for 16-way merge

2010-12-17T06:32:06Z

Without this change, tests/misc/sort-compress-hang would consume more than 10,000 process slots on my RHEL 5.5 x86-64 server, making it likely for other applications to fail due to lack of process slots. With this change, the same benchmark causes 'sort' to consume at most 19 process slots. The change also improved wall-clock time by 2% and user+system time by 14% on that benchmark. * NEWS: Document this. * src/sort.c (MAX_PROCS_BEFORE_REAP): Remove. (reap_exited): Renamed from reap_some; this is a more accurate name, since "some" incorrectly implies that it reaps at least one process. All uses changed. (reap_some): New function: it *does* reap at least one process. (pipe_fork): Do not allow more than NMERGE + 2 subprocesses. (mergefps, sort): Omit check for exited processes: no longer needed, and anyway the code consumed too much CPU per line when 2 < nprocs.

sort: fix hang with sort --compress

2010-12-16T21:55:31Z

* NEWS: Document this. * src/sort.c (UNCOMPRESSED, UNREAPED, REAPED): New constants. (struct tempnode): New member 'state', to hold these constants. The pid member is now undefined if state == UNCOMPRESSED. (struct sortfile): Replace member 'pid' with member 'temp'. (uintptr): Remove. (proctab_hasher, proctab_comparator, register_proc, delete_proc): Proctab entries are now struct tempnode *, not pid_t, to handle the case where multiple tempnode objects correspond to the same pid. This avoids a race condition that can cause a hang. (register_proc): Arg is now struct tempnode *, not pid_t. All callers changed. (delete_proc): Set tempnode state to REAPED. (create_temp_file): No need to set pid member here; it's now done when the pid is known. (maybe_create_temp, create_temp): Remove PPID arg. Return struct tempnode *, not char *. All callers changed. (maybe_create_temp): Set node state to UNCOMPRESSED or UNREAPED. No need to set node->pid to 0. (open_temp): Replace NAME and PID args with a single TEMP arg. All callers changed. Wait only for unreaped children. (zaptemp): Wait for decompressor to finish before removing its temporary-file input. This avoids .nfsXXXX hassles with NFS and fixes a race (leading to a hang) regardless of NFS. (open_input_files): Adjust to new way of dealing with temp files and their subprocesses. * tests/Makefile.am (TESTS): Add misc/sort-compress-hang. * tests/misc/sort-compress-hang: New file.

sort: don't dump core when merging from input twice

2010-12-16T08:04:32Z

* NEWS: Document this. * src/sort.c (avoid_trashing_input): The previous fix to this function didn't fix all the problems with this code. Replace it with something simpler: just copy the input file. This doesn't change the number of files, so return void instead of the updated file count. Caller changed. * tests/misc/sort-merge-fdlimit: Test for the bug.

sort: fix very-unlikely buffer overrun when merging to input file

2010-12-14T19:10:45Z

* src/sort.c (avoid_trashing_input): Fix a typo that could cause a buffer overrun in theory. In practice this is extremely unlikely, as it requires running out of file descriptors in a small merge, presumably because some other process is hogging all the OS's file descriptors.

sort: fix some --compress reaper bugs

2010-12-14T07:23:47Z

* src/sort.c (uintptr): New type. (enum procstate, struct procnode, update_proc): Remove. (proctab_hasher, proctab_comparator, register_proc, wait_proc): (reap_some): The proctab is now simply a hash of process-IDs rather than of pointers to objects with reference counts and states; this is smaller and faster and easier to understand. (nprocs): Now pid_t, not size_t, since one cannot have more than PID_MAX children. (reap): If the argument is -1, wait; if 0 (a new value), do not. Delete pid from proctab as needed. Ignore children that are not in proctab, as they are from the program that exec'ed us and are irrelevant to our success or failure. (delete_proc, reap_all): New functions. (open_temp): Register the child. (sort): Clean up all children afterwards; without this patch, 'sort' sometimes missed failures in children due to race conditions. * tests/Makefile.am (TESTS): Add misc/sort-compress-proc. * tests/misc/sort-compress-proc: New file, to test for the bugs fixed above.

sort: syntax cleanup

2010-12-11T10:29:38Z

* src/sort.c (xfopen, debug_key, sortlines, sort, main): Adjust formatting: fix misplaced braces, use consistent spacing, split a 2-stmt line.

sort: integer overflow checks in thread counts, etc.

2010-12-11T08:29:13Z

* src/sort.c (specify_nthreads, merge_tree_init, init_node): (queue_init, sortlines, struct thread_args, sort, main): Use size_t, not unsigned long int, for thread counts, since thread counts are now used to compute sizes. (specify_nthreads): Check for size_t overflow. (merge_tree_init, sort): Shorten name of local variable, for readability. (merge_tree_init): Move constants next to each other in product, so that the constant folding is easier to see. (init_node): Now static. Add 'restrict' only where it might be helpful for compiler optimization. (queue_init): 2nd arg is now nthreads, not "reserve", which is a bit harder to follow. All uses changed. (struct thread_args): Rename lo_child to is_lo_child, so that it's obvious to the reader when we're talking about this boolean as opposed to the new lo_child member of the other structure. All uses changed. (sort): Remove unused local variable end_node. (main): Don't allow large thread counts to cause undefined behavior later, due to integer overflow.

sort: preallocate merge tree nodes to heap.

2010-12-11T08:29:13Z

* src/sort.c: (merge_tree_init) New function. Allocates memory for merge tree nodes. (merge_tree_destory) New function. (init_node) New function. (sortlines) Refactor node creation code to init_node. Remove now superfluous arguments. All callers changed. (sort) Initialize/destory merge tree. Refactor root node creation to merge_tree_init.