I have been looking at an old post about sorting an array by using a regular expression in Perl. The original post is here
I am struggling to totally understand the script that was voted as the ‘correct’ answer.
The original post was regarding sorting the array below:
my #array = (
"2014 Computer Monitor 200",
"2010 Keyboard 30",
"2012 Keyboard 80",
"2011 Study Desk 100"
);
The question was how to use regular expressions in perl to sort the entire array by year, item name, and price? For example, if the user wants to sort by price they type 'price' and it sorts like this:
2010 Keyboard 30
2012 Keyboard 80
2011 Study Desk 100
2014 Computer Monitor 200
A solution was proposed, that uses a Schwartzian transform. I have just started to learn about this, and this script is a little different to the other examples I've seen. The script that was voted as the correct answer is below. I am looking for advice on how it works.
my $order = "price";
my #array = (
"2014 Computer Monitor 200",
"2010 Keyboard 30",
"2012 Keyboard 80",
"2011 Study Desk 100"
);
my %sort_by = (
year => sub { $a->{year} <=> $b->{year} },
price => sub { $a->{price} <=> $b->{price} },
name => sub { $a->{name} cmp $b->{name} },
);
#array = sort {
local ($a, $b) = map {
my %h;
#h{qw(year name price)} = /(\d+) \s+ (.+) \s+ (\S+)/x;
\%h;
} ($a, $b);
$sort_by{$order}->();
} #array;
# S. transform
# #array =
# map { $_->{line} }
# sort { $sort_by{$order}->() }
# map {
# my %h = (line => $_);
# #h{qw(year name price)} = /(\d+) \s+ (.+) \s+ (\S+)/x;
# $h{name} ? \%h : ();
# } #array;
use Data::Dumper; print Dumper \#array;
I know the script is using the regular expression /(\d+) \s+ (.+) \s+ (\S+)/x to match on year name and price.
I think the rest of the script works as below:
• The initial sort on line 14 takes in items from #array two at a time, one in $a and one in $b
• The map function then takes items $a and $b and maps each to a hash - each item becomes a hash with keys 'year', 'price', and 'name. This is based on the regex /(\d+) \s+ (.+) \s+ (\S+)/x
• Map returns the two hashes, as references, to local variables $a and $b
• I think it is necessary to use local $a and $b otherwise sort will use the default $a and $b taken in at the start of the sort on line 17?
• The 'price' sort function is stored as an coderef in the %sort_by hash
• This is called at line 26 by the code $sort_by{$order}->() on the local versions of $a and $b
This repeated until all items are returned to #array in line 14
Please can anyone tell me if I'm on the right lines here, or correct any misunderstandings. Also can you advise on the use of the local $a and $b variables.
thanks
J
A Schwartzian transform is a way to avoid computing the sorting keys too many times, like in the solution - the one with the local ($a,$b)
The steps of a S. tranform are basically:
use a Map to enrich the list elements with computed sorted keys. Here, %h is used as the new element, containing the original line as line
use a Sort to sort this rich list. The sort with a bit of dirty $a $b magic.
use a Map to extract the original list elements. Here by extracting the line key.
A note on $a $b
Very sadly, $a and $b are global variables in Perl. They usually get automagically assigned inside a sort block. Like in sort { $a <=> $b } (3,2,1)
This explains why the S. solution works even though the compared elements are not given as arguments to the sorting subs. And it also explains the need for local (another Perl horror to pretend a global variable is local) so the naive solution's sort function get the right values in $a, $b.
I strongly encourage you to forget about this and avoid implicit use of $a , $b deeper than the sort block itself.
A slightly more understandable version would be:
my $order = "price";
my #array = (
"2014 Computer Monitor 200",
"2010 Keyboard 30",
"2012 Keyboard 80",
"2011 Study Desk 100"
);
my %sort_by = (
year => sub { shift->{year} <=> shift->{year} },
price => sub { shift->{price} <=> shift->{price} },
name => sub { shift->{name} cmp shift->{name} },
);
my #sorted =
map { $_->{line} }
sort { $sort_by{$order}->($a, $b) }
map {
my %h = (line => $_); # $_ is the array element (the input line)
#h{qw(year name price)} = ( $_ =~ /(\d+) \s+ (.+) \s+ (\S+)/x );
# Did the regex capture a name, i.e. did it work?
if( $h{name} ){
\%h
} else{
(); # Empty array will cause the invalid line to disappear, but you can choose to do something else with it.
}
} #array;
print(join("\n", #sorted))
Related
So I have a problem and I can't solve it. If I read some words from a file in Perl, in that file the words aren't in order, but have a number (as a first character) that should be the element's position to form a sentence.The 0 means that position is correct, 1 means that the word should be in position [1] etc.
The file looks like: 0This 3a 4sentence 2be 1should, and the solution should look like 0This 1should 2be 3a 4sentence.
In a for loop I get through the words array that i get from the file, and this is how i get the first character(the number) $firstCharacter = substr $words[$i], 0, 1;, but i don't know how to properly change the array.
Here's the code that I use
#!/usr/bin/perl -w
$arg = $ARGV[0];
open FILE, "< $arg" or die "Can't open file: $!\n";
$/ = ".\n";
while($row = <FILE>)
{
chomp $row;
#words = split(' ',$row);
}
for($i = 0; $i < scalar #words; $i++)
{
$firstCharacter = substr $words[$i], 0, 1;
if($firstCharacter != 0)
{
}
}
Just use sort. You can use a match in list context to extract the numbers, using \d+ will work even for numbers > 9:
#! /usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
my #words = qw( 0This 3a 4sentence 2be 1should );
say join ' ', sort { ($a =~ /\d+/g)[0] <=> ($b =~ /\d+/g)[0] } #words;
If you don't mind the warnings, or you are willing to turn them off, you can use numeric comparison directly on the words, Perl will extract the numeric prefixes itself:
no warnings 'numeric';
say join ' ', sort { $a <=> $b } #words;
Assuming you have an array like this:
my #words = ('0This', '3a', '4sentence', '2be', '1should');
And you want it sorted like so:
('0This', '1should', '2be', '3a', '4sentence');
There's two steps to this. First is extracting the leading number. Then sorting by that number.
You can't use substr, because you don't know how long the number might be. For example, ('9Second', '12345First'). If you only looked at the first character you'd get 9 and 1 and sort them incorrectly.
Instead, you'd use a regex to capture the number.
my($num) = $word =~ /^(\d+)/;
See perlretut for more on how that works, particularly Extracting Matches.
Now that you can capture the numbers, you can sort by them. Rather than doing it in loop yourself, sort handles the sorting for you. All you have to do is supply the criterion for the sorting. In this case we capture the number from each word (assigned to $a and $b by sort) and compare them as numbers.
#words = sort {
# Capture the number from each word.
my($anum) = $a =~ /^(\d+)/;
my($bnum) = $b =~ /^(\d+)/;
# Compare the numbers.
$anum <=> $bnum
} #words;
There are various ways to make this more efficient, in particular the Schwartzian Transform.
You can also cheat a bit.
If you ask Perl to treat something as a number, it will do its damnedest to comply. If the string starts with a number, it will use that and ignore the rest, though it will complain.
$ perl -wle 'print "23foo" + "42bar"'
Argument "42bar" isn't numeric in addition (+) at -e line 1.
Argument "23foo" isn't numeric in addition (+) at -e line 1.
65
We can take advantage of that to simplify the sort by just comparing the words as numbers directly.
{
no warnings 'numeric';
#words = sort { $a <=> $b } #words;
}
Note that I turned off the warning about using a word as a number. use warnings and no warnings only has effect within the current block, so by putting the no warnings 'numeric' and the sort in their own block I've only turned off the warning for that one sort statement.
Finally, if the words are in a file you can use the Unix sort utility from the command line. Use -n for "numeric sorting" and it will do the same trick as above.
$ cat test.data
00This
3a
123sentence
2be
1should
$ sort -n test.data
00This
1should
2be
3a
123sentence
You should be able to split on the spaces, which will make the numbers the first character of the word. With that assumption, you can simply compare using the numerical comparison operator (<=>) as opposed to the string comparison (cmp).
The operators are important because if you compare strings, the first character is used, meaning 10, 11, and 12 would be out of order, and listed near the 1 (1,10,11,12,2,3,4… instead of 1,2,3,4…10,11,12).
Split, Then Sort
Note: #schwern commented an important point. If you use warnings -- and you should -- you will receive warnings. This is because the values of the internal comparison variables, $a and $b, aren't numbers, but strings (e.g., `"0this", "3a"). I've update the following Codepad and provided more suitable alternatives to avoid this issue.
http://codepad.org/xs2GH9xT
use strict;
use warnings;
my $line = q{0This 3a 4sentence 2be 1should};
my #words = split /\s/,$line;
my #sorted = sort {$a <=> $b} #words;
print qq{
Line: $line
Words: #words
Sorted: #sorted
};
Alternatives
One method is to ignore the warning using no warnings 'numeric' as in Schwern's answer. As he has shown, turning off the warnings in a block will re-enable it afterwards, which may be a little foolproof compared to Choroba's answer, which applies it to the broader scope.
Choroba's solution works by parsing the digits from the those values internally. This is much fewer lines of code, but I would generally advise against that for performance reasons. The regex isn't only run once per word, but multiple times over the sorting process.
Another method is to strip the numbers out and use them for the sort comparison. I attempt to do this below by creating a hash, where the key will be the number and the value will be the word.
Hash Mapping / Key Sort
Once you have an array where the values are the words prefixed by the numbers, you could just as easily split those number/word combo into a hash that has the key as the number and value as the word. This is accomplished by using split.
The important thing to note about the split statement is that a limit is passed (in this case 2), which limits the maximum number of fields the string is split into.
The two values are then used in the map to build the key/value assignment. Thus "0This" is split into "0" and "This" to be used in the hash as "0"=>"This"
http://codepad.org/kY8wwajc
use strict;
use warnings;
my $line = q{0This 3a 4sentence 2be 1should};
my #words = split /\s/, $line; # [ '0This', '3a', ... ]
my %mapped = map { split /(?=\D)/, $_, 2 } #words; # { '0'=>'This, '3'=>'a', ... }
my #sorted = #mapped{ sort { $a <=> $b } keys %mapped }; # [ 'This', 'should', 'be', ... ]
print qq{
Line: $line
Words: #words
Sorted: #sorted
};
This also can be further optimized, but uses multiple variables to illustrate the steps in the process.
I want to write a Perl subroutine like this:
use List::MoreUtils qw{pairwise};
sub multiply {
my ($a, $b) = #_;
return [ pairwise {$a * $b} #$a, #$b ];
}
(multiplication is just an example, I'm gonna do something else)
However, this gives me nonsensical results, because Perl gets confused and tries to use the outer $a and $b instead of the list items.
If I try this code in a REPL (such as reply):
0> use List::MoreUtils qw{pairwise};
1> my $a = [1, 2, 3];
…
2> my $b = [3, 2, 1];
…
3> pairwise {$a * $b} #$a, #$b;
Can't use lexical $a or $b in pairwise code block at reply input line 1.
4> sort {$a <=> $b} #$a;
"my $a" used in sort comparison at reply input line 1.
"my $b" used in sort comparison at reply input line 1.
$res[2] = [
1,
3,
2
]
So far, my solution has been replacing my ($a, $b) = #_; with my ($a_, $b_) = #_; (i.e. renaming the troublesome variables).
Is there any other solution?
Well, first off - $a is an awful variable name. Why are you doing that? Single letters are almost always a bad choice, and just about ok if it's just an iterator or some other simple use.
So the solution would be 'don't call them $a and $b'.
I mean, if you really don't want to use a different name:
sub multiply {
return [ pairwise { $a * $b } #{$_[0]}, #{$_[1]} ];
}
But what you've done is really highlighted why namespace clashes are a bad idea, and thus using $a and $b in your code as actual variables is asking for trouble.
I don't think there's a way to make it work they way you're trying to, and even if you could - you'd end up with some code that's really confusing.
I mean, something like this should work:
sub multiply {
my ( $a, $b ) = #_;
return [ pairwise { $::a * $::b } #$a, #$b ];
}
Because then you're explicitly using the package $a and $b not the lexical one. But note - that won't work the same way if they're imported from another package, and it just generally gets messy.
But it's pretty filthy. perlvar tells you outright that you shouldn't do it:
$a
$b
Special package variables when using sort(), see sort. Because of this specialness $a and $b don't need to be declared (using use vars , or our()) even when using the strict 'vars' pragma. Don't lexicalize them with my $a or my $b if you want to be able to use them in the sort() comparison block or function.
And that's before you get into the territory of 'single letter variable names are pretty much always a bad idea'.
So seriously. Is:
my ( $first, $second ) = #_;
Actually so bad?
pairwise sets the $a and $b found in its caller's package, so you could use fully qualified variable names.
Assuming this code is in package main,
use List::MoreUtils qw{pairwise};
sub multiply {
my ($a, $b) = #_;
return [ pairwise { $main::a * $main::b } #$a, #$b ];
}
Alternatively, our creates a lexical variable that is aliased to the current package's variable with the same name, and it will override the my declaration since the most recent lexical-variable declaration wins out.
use List::MoreUtils qw{pairwise};
sub multiply {
my ($a, $b) = #_;
return [ pairwise { our $a * our $b } #$a, #$b ];
}
The second approach is obviously much less fragile than the first one, but you know what would be even less fragile? Not declaring $a and $b as lexical vars in the first place! :) It would be far simpler just to use different variables. Even $A and $B or $x and $y would be better.
I have a file named all_energy.out and I am trying to sort it in a way so I can renumber files in the directory based on the lowest energy in the all_energy.out file. So I want to create an array with the file names and energy starting at the lowest energy and going to increasing energy like name and age.
Analogous Example:
Don 24
Jan 30
Sue 19
sorted to
Sue 19
Don 24
Jan 30
Example of all_energy.out file: The highest negative value is the lowest energy.
Energy
0001_IP3_fullBinding_Rigid0001 -219.209742
0001_IP3_fullBinding_Rigid0002 -219.188106
0001_IP3_fullBinding_Rigid0003 -219.064542
0001_IP3_fullBinding_Rigid0004 -219.050730
0001_IP3_fullBinding_Rigid0005 -219.044573
0001_IP3_fullBinding_Rigid0006 -218.927479
0001_IP3_fullBinding_Rigid0007 -218.919717
0001_IP3_fullBinding_Rigid0008 -218.900923
0001_IP3_fullBinding_Rigid0009 -218.898945
0001_IP3_fullBinding_Rigid0010 -218.889269
0001_IP3_fullBinding_Rigid0011 -218.871619
0001_IP3_fullBinding_Rigid0012 -218.859429
0001_IP3_fullBinding_Rigid0013 -218.848516
0001_IP3_fullBinding_Rigid0014 -218.835355
0001_IP3_fullBinding_Rigid0015 -218.822244
0001_IP3_fullBinding_Rigid0016 -218.819328
0001_IP3_fullBinding_Rigid0017 -218.818431
0001_IP3_fullBinding_Rigid0018 -218.815494
0001_IP3_fullBinding_Rigid0019 -218.798388
0001_IP3_fullBinding_Rigid0020 -218.792151
Energy
0002_IP3_fullBinding_Rigid0001 -226.007998
0002_IP3_fullBinding_Rigid0002 -225.635657
The file names are given before the energy value, for example 0001_IP3_fullBinding_Rigid0001.mol2 is the name of the first file.
Example solution:
0002_IP3_fullBinding_Rigid0001 -226.007998
0002_IP3_fullBinding_Rigid0002 -225.635657
0001_IP3_fullBinding_Rigid0001 -219.209742
0001_IP3_fullBinding_Rigid0002 -219.188106
0001_IP3_fullBinding_Rigid0003 -219.064542
My current script is:
#!/usr/bin/perl
use strict;
use warnings;
print "Name of all total energy containing file:\n";
my $energy_file = <STDIN>;
chomp $energy_file;
my $inputfile_energy = $energy_file;
open (INPUTFILE_ENERGY, "<", $inputfile_energy) or die $!;
print map $inputfile_energy->[0],
sort { $a->[1] <=> $b->[1] }
map { [ $_, /(\d+)$/ ] }
<INPUTFILE_ENERGY>;
close $inputfile_energy;
At this point I am just trying to get the energy with their names to print to the correct order. Then I will loop through the files in the directory and when the name matches with the sorted energy names it will be renumber.
Problems with your script:
/(\d+)$/ only matches digits (0-9) at the end of a line. Your file contains floating point numbers, so only digits after the decimal point will be matched. You could get away with /(\S+)$/ instead. (Actually, in your sample input there is a line with a trailing space, so let's make that /(\S+)\s*$/ instead)
$inputfile_energy is a filename, a scalar, and not a reference, so $inputfile_energy->[0] doesn't make sense. You use it as the expression in a map construction, and in a map EXPR, LIST construction, $_ refers to the current element of the list that is being iterated through, so you probably just meant to say $_->[0].
Your input contains a few lines -- all with the keyword Energy -- that don't have the same format as the other lines you want to sort and should be filtered out.
Putting this all together, I get working code when the penultimate statement looks like:
print map $_->[0],
sort { $a->[1] <=> $b->[1] }
map { [ $_, /(\S+)\s*$/ ] }
grep /\d/,
<INPUTFILE_ENERGY>;
you can use oneliner like this and run it from command line:
perl -lnae 'push #arr, [$_, $F[1]] if $F[1]; END { print join "\n", map {$_->[0]} sort {$a->[1] <=> $b->[1]} #arr }' energy_file.txt
1) special key -n makes the loop over all lines in input file (energy_file.txt); current line is available in $_ variable.
2) then key -a splits each line by whitespaces and puts nonempty values into #F array.
A less "idiomatic" solution could be :
#data = <DATA>;
my #table;
foreach(#data){
chomp;
next unless /^0/; # skip Energy lines (or any other cleaning test)
#line = split /\s+/;
push #table,[#line]; # build a 2d array
}
my #sortedTable = sort { $a->[1] <=> $b->[1] } #table;
foreach(#sortedTable){
printf(
"%5s,%25s\n",
$_->[0],
$_->[1]
) # some pretty printing
}
__DATA__
Energy
0001_IP3_fullBinding_Rigid0001 -219.209742
0001_IP3_fullBinding_Rigid0002 -219.188106
0001_IP3_fullBinding_Rigid0003 -219.064542
0001_IP3_fullBinding_Rigid0004 -219.050730
....
Try this:
print join "\n", sort {(split /\s+/,$a)[1] <=> (split /\s+/,$b)[1]} map{chomp $_; $_} <INPUTFILE_ENERGY>;
I understand that the default sort in Perl is an ASCII sort, not numerical. But how can I numerically sort strings that have numbers?
For example, I have a hash of arrays, like so:
myhash{ANN20021015_0101_XML_71.9} = ("anta", "hunna", "huma");
myhash{ANN20021115_0049_XML_14.1} = ("lqd", "qAl", "arrajul");
myhash{ANN20021115_0049_XML_14.2} = ("anna", "alwalada");
I just need the keys to be sorted...but the sorting is numerical within strings. I can't do a string sort, because I end up with "10" following "1", but I can't do a numerical sort either!
First of all your code isn't valid Perl and may not do what you think it does. Always
use strict;
use warnings;
at the head of your program to resolve any simple mistakes. The code should look like
$myhash{'ANN20021015_0101_XML_71.9'} = ["anta", "hunna", "huma"];
$myhash{'ANN20021115_0049_XML_14.1'} = ["lqd", "qAl", "arrajul"];
$myhash{'ANN20021115_0049_XML_14.2'} = ["anna", "alwalada"];
To sort on something other than the entire value, you can transform $a and $b within the sort block, and sort the result numerically <=> instead of stringwise <=>. This code does what you need
my #sorted = sort {
my ($aa) = $a =~ /.*_(.+)/;
my ($bb) = $b =~ /.*_(.+)/;
$aa <=> $bb;
} keys %myhash;
But if you have a large amount of data it may be profitable to use the Schwartzian Transform which will avoid extracting the numeric part of your strings every time they are compared
my #sorted = map { $_->[0] }
sort { $a->[1] cmp $b->[1] }
map { /.*_(.+)/ and [$_, $1] }
keys %myhash;
You need to do a custom sort: cut your strings into parts you know are literals/numbers and compare those as needed.
From your example it looks like you want literal.digits, but you can change the regular expression so that it fits you.
my $cut = qr/(.*?\.)(\d+)(.*)/;
sort {
my #a = $a =~ $cut; my #b = $b =~ $cut;
$a[0] cmp $b[0] || $a[1] <=> $b[1] || $a[2] cmp $b[2]
} keys %myhash;
See also Borodin's answer.
I have a text file layed out like this:
1 a, b, c
2 c, b, c
2.5 a, c
I would like to reverse the keys (the number) and values (CSV) (they are separated by a tab character) to produce this:
a 1, 2.5
b 1, 2
c 1, 2, 2.5
(Notice how 2 isn't duplicated for c.)
I do not need this exact output. The numbers in the input are ordered, while the values are not. The output's keys must be ordered, as well as the values.
How can I do this? I have access to standard shell utilities (awk, sed, grep...) and GCC. I can probably grab a compiler/interpreter for other languages if needed.
If you have python (if you're on linux you probably already have) i'd use a short python script to do this. Note that we use sets to filter out "double" items.
Edited to be closer to requester's requirements:
import csv
from decimal import *
getcontext().prec = 7
csv_reader = csv.reader(open('test.csv'), delimiter='\t')
maindict = {}
for row in csv_reader:
value = row[0]
for key in row[1:]:
try:
maindict[key].add(Decimal(value))
except KeyError:
maindict[key] = set()
maindict[key].add(Decimal(value))
csv_writer = csv.writer(open('out.csv', 'w'), delimiter='\t')
sorted_keys = [x[1] for x in sorted([(x.lower(), x) for x in maindict.keys()])]
for key in sorted_keys:
csv_writer.writerow([key] + sorted(maindict[key]))
I would try perl if that's available to you. Loop through the input a row at a time. Split the line on the tab then the right hand part on the commas. Shove the values into an associative array with letters as the keys and the value being another associative array. The second associative array will be playing the part of a set so as to eliminate duplicates.
Once you read the input file, sort based on the keys of the associative array, loop through and spit out the results.
here's a small utility in php:
// load and parse the input file
$data = file("path/to/file/");
foreach ($data as $line) {
list($num, $values) = explode("\t", $line);
$newData["$num"] = explode(", ", trim($values));
}
unset($data);
// reverse the index/value association
foreach ($newData as $index => $values) {
asort($values);
foreach($values as $value) {
if (!isset($data[$value]))
$data[$value] = array();
if (!in_array($index, $data[$value]))
array_push($data[$value], $index);
}
}
// printout the result
foreach ($data as $index => $values) {
echo "$index\t" . implode(", ", $values) . "\n";
}
not really optimized or good looking, but it works...
# use Modern::Perl;
use strict;
use warnings;
use feature qw'say';
our %data;
while(<>){
chomp;
my($number,$csv) = split /\t/;
my #csv = split m"\s*,\s*", $csv;
push #{$data{$_}}, $number for #csv;
}
for my $number (sort keys %data){
my #unique = sort keys %{{ map { ($_,undef) } #{$data{$number}} }};
say $number, "\t", join ', ', #unique;
}
Here is an example using CPAN's Text::CSV module rather than manual parsing of CSV fields:
use strict;
use warnings;
use Text::CSV;
my %hash;
my $csv = Text::CSV->new({ allow_whitespace => 1 });
open my $file, "<", "file/to/read.txt";
while(<$file>) {
my ($first, $rest) = split /\t/, $_, 2;
my #values;
if($csv->parse($rest)) {
#values = $csv->fields()
} else {
warn "Error: invalid CSV: $rest";
next;
}
foreach(#values) {
push #{ $hash{$_} }, $first;
}
}
# this can be shortened, but I don't remember whether sort()
# defaults to <=> or cmp, so I was explicit
foreach(sort { $a cmp $b } keys %hash) {
print "$_\t", join(",", sort { $a <=> $b } #{ $hash{$_} }), "\n";
}
Note that it will print to standard output. I recommend just redirecting standard output, and if you expand this program at all, make sure to use warn() to print any errors, rather than just print()ing them. Also, it won't check for duplicate entries, but I don't want to make my code look like Brad Gilbert's, which looks a bit wack even to a Perlite.
Here's an awk(1) and sort(1) answer:
Your data is basically a many-to-many data set so the first step is to normalise the data with one key and value per line. We'll also swap the keys and values to indicate the new primary field, but this isn't strictly necessary as the parts lower down do not depend on order. We use a tab or [spaces],[spaces] as the field separator so we split on the tab between the key and values, and between the values. This will leave spaces embedded in the values, but trim them from before and after:
awk -F '\t| *, *' '{ for (i=2; i<=NF; ++i) { print $i"\t"$1 } }'
Then we want to apply your sort order and eliminate duplicates. We use a bash feature to specify a tab char as the separator (-t $'\t'). If you are using Bourne/POSIX shell, you will need to use '[tab]', where [tab] is a literal tab:
sort -t $'\t' -u -k 1f,1 -k 2n
Then, put it back in the form you want:
awk -F '\t' '{
if (key != $1) {
if (key) printf "\n";
key=$1;
printf "%s\t%s", $1, $2
} else {
printf ", %s", $2
}
}
END {printf "\n"}'
Pipe them altogether and you should get your desired output. I tested with the GNU tools.