coreutils/gl/modules, branch v8.6

mbsalign: avoid warning with Solaris 10 cc

2010-10-14T05:41:17Z

* gl/lib/mbsalign.c: Include for SIZE_MAX. Use SIZE_MAX rather than (size_t) -1, to avoid warning for "size_t x = -1" on Solaris 10 cc. * gl/modules/mbsalign (Depends-on): Add stdint.

randread: run 2x faster on 64-bit hosts, don't assume no padding bits

2010-07-23T22:08:36Z

* gl/lib/rand-isaac.c: Remove the I/O; this belongs elsewhere. Add support for ISAAC64. Port to hosts with padding bits. Add self to author list. Include , for CHAR_BIT. Don't include string.h, sys/time.h, unistd.h. (min, just): New functions. (IF32): New macros. (ind, ISAAC_STEP, isaac_refill, mix, isaac_init, isaac_seed): Add support for ISAAC64. Port to hosts with padding bits. (ind): Now an inline function rather than a macro; no need for it to be a macro with modern compilers. (ISAAC_STEP): Renamed from isaac_step, since it's not function-like. Don't bother to pass args that are always the same. All uses changed. (ISAAC_STEP, ISAAC_SEED): Move to inside the only function body that can use it. (ISAAC_MIX): Renamed from isaac_mix, since it's now a macro and is no longer function-like. Don't bother saving and restoring state; no longer needed now that we're not a function. All uses changed. (isaac_seed_start, isaac_seed_data, isaac_seed_finish): Remove. (isaac_seed): Take just the one arg; the caller now sets s->m. * gl/lib/rand-isaac.h: Use _GL_RAND_ISAAC_H to protect, instead of RAND_ISAAC_H. Try out " #" rather than "# " for indenting. (ISAAC_BITS_LOG, ISAAC_BITS): New macros. (ISAAC_WORDS_LOG): Renamed from ISAAC_LOG. (isaac_word): New type. All uses of uint32_t changed to isaac_word, to support ISAAC64. (struct isaac_state): Rename member MM to M, and make it public. (isaac_seed, isaac_refill): Adjust to new API. * gl/lib/randread.c: Include sys/time.h. (get_nonce): New function, containing the nonce stuff that used to be in rand-isaac.c but better belongs here. (randread_new): Use it. * gl/modules/randread (Depends-on): Add inline. * gl/modules/randread-tests: New file. * gl/tests/test-rand-isaac.c: New file.

fadvise: new module providing a simpler interface to posix_fadvise

2010-07-22T00:03:48Z

* bootstrap.conf: Include the new module * gl/lib/fadvise.c: Provide a simpler interface to posix_fadvise. (fadvise): Provide hint to the whole file associated with a stream. (fdadvise): Provide hint to the specific portion of a file associated with a file descriptor. * gl/lib/fadvise.h: Redefine POSIX_FADV_* to FADVISE_* enums. * gl/modules/fadvise: New file. * m4/jm-macros.m4: Remove the no longer needed posix_fadvise check. * .x-sc_program_name: Exclude test-fadvise.c from this check. * gl/tests/test-fadvise (main): New test program. * gl/modules/fadvise-testss: A new index to reference the tests. * src/sort.c (stream_open): Use the new interface. * src/dd.c (iwrite): Likewise.

randread: don't require -lrt

2010-07-16T22:49:05Z

Programs like 'sort' were linking to -lrt in order to get clock_gettime, but this was misguided: it wasted considerable resources while gaining at most 10 bits of entropy. Almost nobody needs the entropy, and there are better ways to get much better entropy for people who do need it. * gl/lib/rand-isaac.c (isaac_seed): Include not "gethrxtime.h". (isaac_seed): Use gettimeofday rather than gethrxtime. * gl/modules/randread (Depends-on): Depend on gettimeofday and not gethrxtime. * src/Makefile.am (mktemp_LDADD, shred_LDADD, shuf_LDADD, sort_LDADD): (tac_LDADD): Omit $(LIB_GETHRXTIME); no longer needed.

sort: parallelize internal sort

2010-07-13T00:44:46Z

This patch is by Gene Auyeung, Chris Dickens, Chen Guo, and Mike Nichols, based off of a patch by Paul Eggert, Glen Lenker, et. al., with a basic heap implementation based off of the GDSL heap, originally by Nicolas Darnis. The number of sorts done in parallel is limited to the number of available processors by default, or can be further restricted with the --parallel option. On a dual-die, 8 core Intel Xeon, results show sorting with 8 threads is almost 4 times faster than using a single thread. Timings when sorting a 96MB file: THREADS TIME (s) 1 5.10 2 2.87 4 1.75 8 1.31 Single threaded sorting has also been improved, especially for cheaper comparison operations: COMMAND BEFORE (s) AFTER (s) sort 8.822 8.716 sort -g 10.336 10.222 sort -n 3.077 2.961 LANG=C sort 2.169 2.066 * bootstrap.conf: Add heap, pthread. * coreutils.texi (sort): Describe the new --parallel option. * gl/lib/heap.c: New file. Very basic heap implementation. * gl/lib/heap.h: New file. * gl/modules/heap: New file. * src/Makefile.am: Add LIB_PTHREAD. * src/sort.c: Include heap.h, nproc.h, pthread.h. (MAX_MERGE): New macro. (SUBTHREAD_LINES_HEURISTIC, PARALLEL_OPTION): New constants. (MERGE_END, MERGE_ROOT): New constants. (struct merge_node): New struct. (struct merge_node_queue): New struct. (sortlines temp): Remove declaration. (usage, long_options, main): New option, --parallel. (specify_nthreads): New function. (mergelines): New signature, to emphasize the fact that the HI area must be part of the destination. All callers changed. (sequential_sort): New function, renamed from sortlines. Merge in the functionality of sortlines_temp. (compare_nodes): New function. (lock_node, unlock_node): New functions. (queue_destroy): New function. (queue_init): New function. (queue_insert): New function. (queue_pop): New function. (write_unique): New function. (mergelines_node): New function. (check_insert): New function. (update_parent): New function. (merge_loop): New function. (sortlines): Rewrite to support and use parallelism, with a new signature. All callers changed. (struct thread_args): New struct. (sortlines_thread): New function. (sortlines_temp): Remove. (sort): New argument NTHREADS. All uses changed. Output moved to mergelines_node. (main): disable threading if we are sorting at random. * tests/Makefile.am (TESTS): Add misc/sort-benchmark-random. * tests/misc/sort-benchmark-random: New file. Signed-off-by: Pádraig Brady

du: Hash with a mechanism that's simpler and takes less memory.

2010-07-06T21:58:48Z

* gl/lib/dev-map.c, gl/lib/dev-map.h, gl/modules/dev-map: Remove. * gl/lib/ino-map.c, gl/lib/ino-map.h, gl/modules/ino-map: New files. * gl/modules/dev-map-tests, gl/tests/test-dev-map.c: Remove. * gl/modules/ino-map-tests, gl/tests/test-ino-map.c: New files. * gl/lib/di-set.h (struct di_set): Renamed from struct di_set_state, and now private. All uses changed. (_ATTRIBUTE_NONNULL_): Don't assume C99. (di_set_alloc): Renamed from di_set_init, with no size arg. Now allocates the object rather than initializing it. For now, this no longer takes an initial size; we can put this back later if it is needed. * gl/lib/di-set.c: Include hash.h, ino-map.h, and limits.h instead of stdio.h, assert.h, stdint.h, sys/types.h (di-set.h includes that now), sys/stat.h, and verify.h. (N_DEV_BITS_4, N_INO_BITS_4, N_DEV_BITS_8, N_INO_BITS_8): Remove. (struct dev_ino_4, struct dev_ino_8, struct dev_ino_full): Remove. (enum di_mode): Remove. (hashint): New typedef. (HASHINT_MAX, LARGE_INO_MIN): New macros. (struct di_ent): Now maps a dev_t to a inode set, instead of containing a union. (struct dev_map_ent): Remove. (struct di_set): New type. (is_encoded_ptr, decode_ptr, di_ent_create): Remove. (di_ent_hash, di_ent_compare, di_ent_free, di_set_alloc, di_set_free): (di_set_insert): Adjust to new representation. (di_ino_hash, map_device, map_inode_number): New functions. * gl/modules/di-set (Depends-on): Replace dev-map with ino-map. Remove 'verify'. * gl/tests/test-di-set.c: Adjust to the above changes to API. * src/du.c (INITIAL_DI_SET_SIZE): Remove. (hash_ins, main): Adjust to new di-set API.

di-set: manipulate sets of dev/inode pairs efficiently

2010-07-04T06:40:40Z

* gl/lib/di-set.c: Implementation. * gl/lib/di-set.h: Declarations. * gl/modules/di-set: Define module. * gl/modules/di-set-tests: Define test module. * gl/tests/test-di-set.c: Likewise.

dev-map: map device number to small non-negative

2010-07-04T06:40:40Z

* gl/lib/dev-map.c: New file. * gl/lib/dev-map.h: Declarations. * gl/modules/dev-map: Define primary modules. * gl/modules/dev-map-tests: Define test module. * gl/tests/test-dev-map.c: Test it.

maint: update the mbsalign module

2010-03-19T19:23:45Z

* gl/lib/mbsalign.c (mbsalign): Support the MBA_UNIBYTE_FALLBACK flag which reverts to unibyte mode if one can't allocate memory or if there are invalid multibyte characters present. Note memory is no longer dynamically allocated in unibyte mode so one can assume that mbsalign() will not return an error if this flag is present. Don't calculate twice, the number of spaces, when centering. Suppress a signed/unsigned comparison warning. (ambsalign): A new wrapper function to dynamically allocate the minimum memory required to hold the aligned string. * gl/lib/mbsalign.h: Add the MBA_UNIBYTE_FALLBACK flag and also document others that may be implemented in future. (ambsalign): A prototype for the new wrapper. * gl/tests/test-mbsalign.c (main): New test program. * gl/modules/mbsalign-tests: A new index to reference the tests. * .x-sc_program_name: Exclude test-mbsalign.c from this check.

tests: fix link failure on cygwin

2009-11-24T13:36:13Z

Counterpart to commit 8fe40b84bd8, since test-link.c uses rename, and we override gnulib with a rename() replacement that can xalloc_die. * gl/modules/link-tests.diff: New file.