Change #265906

Comments

sort: use more dynamic memory allocation with pipes
The default memory allocation with pipes was too passive/static,
resulting in not allocating enough memory to enable threading.
By dynamically reallocating the buffer when reading from
unknown sized inputs we better use available memory and threads.

  $ time seq 10000000 -1 0 | sort-old >/dev/null
  real	0m16.523s
  user	0m16.900s
  sys	0m0.167s

  $ time seq 10000000 -1 0 | sort-old -S1G >/dev/null
  real	0m12.263s
  user	0m29.646s
  sys	0m0.527s

  $ time seq 10000000 -1 0 | sort-new >/dev/null
  real	0m12.994s
  user	0m31.266s
  sys	0m0.716s

It also avoids the overhead of writing to temp files
for modestly sized inputs. For example the following
input would induce interaction with temp storage:

  $ seq 125000 | wc -c
  763895

* src/sort.c (sort_buffer_size): Rename to ...
(sort_buffer_policy): ... here, and adjust to set
an initial size and limit, rather than just a size.
(fillbuf): Add a POLICY parameter, and use that
to call maybe_growbuf() as needed.
(maybe_growbuf): Return true if POLICY dictates we
should grow the buffer, and try_growbuf() was
able to reallocate the larger buffer.
* tests/sort/sort-buffer-size.sh: Add a new test.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
Related to https://bugs.gnu.org/10877

Changed files

• NEWS
• src/sort.c
• tests/local.mk
• tests/sort/sort-buffer-size.sh