Concatenate text files, separating them with a new line - database

Question

I have just over 100 text files in a directory, functioning as a simple database with each line containing one record. In total, these files add up to around 25GB. However, the records are not sorted alphabetically and there are many duplicates, so in order to alphabetise the contents of all ~100 text files using something like sort -u, I'm first trying to combine all of these files into a single large text file. A simple cat would be unsuitable because the beginnings and ends of the 100 text files do not contain new lines, which (to my understanding) would cause the last record in a file to become merged with the first record of the next file.
What solutions are there that would allow me to concatenate my text files while ensuring there is a single new line character separating them?

Answer 1

A simple
sort -u *.db > uniquified # adjust glob as needed
should do it; sort will interpose newlines between files should it be necessary.
cat *.db | sort -u
is a classic UUoC and the glitch with files lacking trailing newlines is not the only issue.
Having said that, 25GB probably won't fit in your RAM, so sort will end up creating temporary files anyway. It might turn out to be faster to sort the files in four or five groups, and then merge the results. That could take better advantage of the large number of duplicates. But I'd only experiment if the simple command really takes an exorbitant amount of time.
Even so, sorting the files individually is probably even slower; usually the best bet is to max out your memory resources for each invocation of sort. You could, for example, use xargs with the -n option to split the filelist into groups of a couple of dozen files each. Once you have each group sorted, you could use sort -m to merge the sorted temporaries.
A couple of notes on how to improve sorting speed:
Use LC_COLLATE=C sort if you don't need locale-aware sorting of alphabetic data. That typically speeds sort up by a factor of three or four.
Avoid using RAM disks for temporary space. (On many Linux distros, /tmp is a RAM disk.) Since sort uses temporary disks when it runs out of RAM, putting the temporary in a RAMdisk is counterproductive. For the same reason, don't put your own temporary output files in /tmp. /var/tmp should be real disk; even better, if possible, use a second disk drive (not a slow USB drive, of course).
Avoid slugging your machine down with excessive swapping while you're doing the sort, by turning swap off:
sudo swapoff -a
You can turn it back on afterwards, although I personally run my machine like this all the time because it avoids diving into complete unresponsiveness under memory pressure.
The ideal is to adjust -S so that sort uses as much memory as you can spare, and avoid the use of internal temporaries by sorting in chunks which fit into that amount of memory. (Merging the sorted chunks is a lot faster than sorting, and it reads and writes sequentially without needing extra disk space.) You'll probably need to do some experimentation to find a good chunk size.

Answer 2

I would tell you to create that file by concatenating all input files and intercalating a new line in the middle:
out=newfile.txt
rm -f "$out"
for f in *.txt
do
cat "$f" >> "$out"
echo >> "$out"
done
Now you can sort it. Or remove empty lines, in case you think there could be some input file with a new line at the end.

Answer 3

You can use awk.
$ od -t x1 file1
0000000 72 65 63 6f 72 64 31 0a 72 65 63 6f 72 64 32
0000017
$ od -t x1 file2
0000000 72 65 63 6f 72 64 31 0a 72 65 63 6f 72 64 32 0a
0000020 72 65 63 6f 72 64 33
0000027
$ awk 1 file1 file2
record1
record2
record1
record2
record3
1 is awk script here, which means print all records

Answer 4

sort * should be all you need but just in case you every do need to append newlines to file contents for processing by a subsequent tool, here's how to do that:
$ ls
file1 file2
$ cat file1
foo$
$ cat file2
bar$
$ cat file1 file2
foobar$
$ find . -type f -exec sh -c '(cat {}; printf "\\n")' \;
foo
bar
That is, of course, assuming that your cat can handle files that don't end in newlines!