Perl script problems - arrays

The purpose of the script is to process all words from a file and output ALL words that occur the most. So if there are 3 words that each occur 10 times, the program should output all the words.
The script now runs, thanks to some tips I have gotten here. However, it does not handle large text files (i.e. the New Testament). I'm not sure if that is a fault of mine or just a limitation of the code. I am sure there are several other problems with the program, so any help would be greatly appreciated.
#!/usr/bin/perl -w
require 5.10.0;
print "Your file: " . $ARGV[0] . "\n";
#Make sure there is only one argument
if ($#ARGV == 0){
#Make sure the argument is actually a file
if (-f $ARGV[0]){
%wordHash = (); #New hash to match words with word counts
$file=$ARGV[0]; #Stores value of argument
open(FILE, $file) or die "File not opened correctly.";
#Process through each line of the file
while (<FILE>){
chomp;
#Delimits on any non-alphanumeric
#words=split(/[^a-zA-Z0-9]/,$_);
$wordSize = #words;
#Put all words to lowercase, removes case sensitivty
for($x=0; $x<$wordSize; $x++){
$words[$x]=lc($words[$x]);
}
#Puts each occurence of word into hash
foreach $word(#words){
$wordHash{$word}++;
}
}
close FILE;
#$wordHash{$b} <=> $wordHash{$a};
$wordList="";
$max=0;
while (($key, $value) = each(%wordHash)){
if($value>$max){
$max=$value;
}
}
while (($key, $value) = each(%wordHash)){
if($value==$max && $key ne "s"){
$wordList.=" " . $key;
}
}
#Print solution
print "The following words occur the most (" . $max . " times): " . $wordList . "\n";
}
else {
print "Error. Your argument is not a file.\n";
}
}
else {
print "Error. Use exactly one argument.\n";
}

Your problem lies in the two missing lines at the top of your script:
use strict;
use warnings;
If they had been there, they would have reported lots of lines like this:
Argument "make" isn't numeric in array element at ...
Which comes from this line:
$list[$_] = $wordHash{$_} for keys %wordHash;
Array elements can only be numbers, and since your keys are words, that won't work. What happens here is that any random string is coerced into a number, and for any string that does not begin with a number, that will be 0.
Your code works fine reading the data in, although I would write it differently. It is only after that that your code becomes unwieldy.
As near as I can tell, you are trying to print out the most occurring words, in which case you should consider the following code:
use strict;
use warnings;
my %wordHash;
#Make sure there is only one argument
die "Only one argument allowed." unless #ARGV == 1;
while (<>) { # Use the diamond operator to implicitly open ARGV files
chomp;
my #words = grep $_, # disallow empty strings
map lc, # make everything lower case
split /[^a-zA-Z0-9]/; # your original split
foreach my $word (#words) {
$wordHash{$word}++;
}
}
for my $word (sort { $wordHash{$b} <=> $wordHash{$a} } keys %wordHash) {
printf "%-6s %s\n", $wordHash{$word}, $word;
}
As you'll note, you can sort based on hash values.

Here is an entirely different way of writing it (I could have also said "Perl is not C"):
#!/usr/bin/env perl
use 5.010;
use strict; use warnings;
use autodie;
use List::Util qw(max);
my ($input_file) = #ARGV;
die "Need an input file\n" unless defined $input_file;
say "Input file = '$input_file'";
open my $input, '<', $input_file;
my %words;
while (my $line = <$input>) {
chomp $line;
my #tokens = map lc, grep length, split /[^A-Za-z0-9]+/, $line;
$words{ $_ } += 1 for #tokens;
}
close $input;
my $max = max values %words;
my #argmax = sort grep { $words{$_} == $max } keys %words;
for my $word (#argmax) {
printf "%s: %d\n", $word, $max;
}

why not just get the keys from the hash sorted by their value and extract the first X?
this should provide an example: http://www.devdaily.com/perl/edu/qanda/plqa00016

Related

Perl: Adding an exception to a foreach loop

I'm new to Stack Overflow and I would like to ask for some advice with regard to a minor problem I have with my Perl code.
In short, I have written a small programme that opens text files from a pre-defined array, then searches for certain strings in them and finally prints out the line containing the string.
my #S1A_SING_Files = (
'S1A-001_SING_annotated.txt',
'S1A-002_SING_annotated.txt',
'S1A-003_SING_annotated.txt',
'S1A-004_SING_annotated.txt',
'S1A-005_SING_annotated.txt'
);
foreach (#S1A_SING_Files) {
print ("\n");
print ("Search results for $_:\n\n");
open (F, $_) or die("Can't open file!\n");
while ($line = <F>) {
if ($line =~ /\$(voc)?[R|L]D|\$Rep|\/\//) {
print ($line);
}
}
}
close (F);
I was wondering whether it is possible to create an exception to the foreach loop, so that the line containing
print ("\n");
not be executed if the file is $S1A_SING_Files[0]. It should then be normally executed if the file is any of the following ones. Do you think this could be accomplished?
Thank you very much in advance!
Yes. Just add a check for the first file. Change:
print ("\n");
to:
print ("\n") if $_ ne $S1A_SING_Files[0];
If the array contains unique strings, you can use the following:
print("\n") if $_ ne $S1A_SING_Files[0]; # Different stringification than 1st element?
The following will work even if the array contains non-strings or duplicate values (and it's faster too):
print("\n") if \$_ != \$S1A_SING_Files[0]; # Different scalar than 1st element?
Both of the above could fail for magical arrays. The most reliable solution is to iterate over the indexes.
for my $i (0..$#S1A_SING_Files) {
my $file = $S1A_SING_Files[$i];
print("\n") if $i; # Different index than 1st element?
...
}
Your code can be written in following form
use strict;
use warnings;
my #S1A_SING_Files = (
'S1A-001_SING_annotated.txt',
'S1A-002_SING_annotated.txt',
'S1A-003_SING_annotated.txt',
'S1A-004_SING_annotated.txt',
'S1A-005_SING_annotated.txt'
);
foreach (#S1A_SING_Files) {
print "\n" unless $_ ne $S1A_SING_Files[0];
print "Search results for $_:\n\n";
open my $fh, '<', $_ or die("Can't open file!\n");
m!\$(voc)?[R|L]D|\$Rep|//! && print while <$fh>;
close $fh;
}

Adding a key value pair in hash, by assigning an array in the value => Can't use an undefined value as an ARRAY reference

I'm trying to assign an array in a hash key-value pair as a value of a key. After assigning it i'm trying to dereference it and print the array values from the specific key in an output file as you can see from the code below.
The code is not working well on the array manipulation part. Can someone tell me what I'm doing wrong?
use strict;
use warnings;
use Data::Dumper;
# File input
my $in_file = 'log.txt';
# Output file
my $out_file_name = 'output.csv';
open(my $fout, '>', $out_file_name);
# print header csv
print $fout "Col1\,Col2\,Col3\,Col4\,Col5\n";
# Read the input file
open(FH, '<', $in_file) or die "Could not open file '$in_file' $!";
my #log_file = <FH>;
# print Dumper(#log_file),"\n";
close (FH);
# my #test_val;
my ($read, $ref, $val_county, $val_rec, $val_tar, $val_print, #test_values, $status);
foreach(#log_file) {
# print $_;
if ($_ =~ /\t+(?<county_name>(?!Total).+)\s+/i) {
$ref->{code} = $+{county_name};
$val_county = $ref->{code};
} elsif ($_ =~ /^Total\s+records\s+in\s+TAR\s+\(pr.+\)\:\s+(?<tar_records>.+)$/i) {
$ref->{code} = $val_county;
push(#test_values, $+{tar_records});
$ref->{tar_rec} = \#test_values;
# $val_rec = $ref->{tar_rec};
# $val_rec =~ s/\.//g;
}
&print_file($ref);
}
sub print_file {
my $ref = shift;
my $status = shift;
print $fout join(",", $ref->{code}, [#{$ref->{tar_rec}}]), "\n"; # Line 68
print Dumper($ref);
}
close $fout;
print "Done!","\n";
The code is a providing an error like:
"Can't use an undefined value as an ARRAY reference at test_array_val_hash.pl line 68."
Until the second regex in your forloop block is matched, the $ref->{tar_rec} key will not be assigned a value - and will be undefined. The following snippet - based on your own code - highlights the issue.
#!/usr/bin/perl -w
my #tar_records = (15,35,20);
my $ref = {
code => 'Cork',
tar_rec => \#tar_records,
};
sub print_info {
my $ref = shift;
print join(", ", $ref->{code}, (#{$ref->{tar_rec}})), $/;
}
print_info($ref);
# Once we 'undefine' the relevant key, we witness the afore-
# mentioned error.
undef $ref->{tar_rec};
print_info($ref);
To avoid this error, you could assign an anonymous array reference to $ref->{tar_rec} key before the for loop (since $ref->{tar_rec} is a cumulative value).
# Be sure not to declare $ref twice!
my ($read, $val_county, $val_rec, $val_tar, $val_print, #test_values, $status);
my $ref = {
code => '',
tar_rec => [],
}
P.S. Notice also that I used round brackets rather than square brackets in the join() function (although you actually don't need either).
The problem is that you're calling print_file in the wrong place.
Imagine that you're parsing the file a line at a time. Your code parses the first line and that populates $ref->{code}. But then you call print_file on a partially populated $ref so it doesn't work.
Your code is also not resetting any of the variables used, so as it progresses through the file, the contents of $ref are going to grow.
The code below fixes the first problem by implicitly setting an empty array in $ref->{tar_rec} and only printing out the record when it's starting a new one or when it's finished reading in the file. Since $ref->{tar_rec} is an array it solves the other problem by allowing you to directly push into it rather than relying upon #test_values. Just for added safety it assigns an empty hash to $ref.
if(open(my $fh, '<', $in_file)) {
my $ref;
my $val_county;
foreach(<$fh>) {
# print $_;
if ($_ =~ /\t+(?<county_name>(?!Total).+)\s+/i) {
if(defined($val_county)) {
print_file($ref);
}
$ref={};
$val_county = $+{county_name};
$ref->{code} = $val_county;
$ref->{tar_rec} = [];
} elsif ($_ =~ /^Total\s+records\s+in\s+TAR\s+\(pr.+\)\:\s+(?<tar_records>.+)$
push #{$ref->{tar_rec}}, $+{tar_records};
}
}
if(defined($ref)) {
print_file($ref);
}
close($fh);
} else {
die "Could not open file '$in_file' $!";
}
You're also printing out the array incorrectly
print $fout join(",", $ref->{code}, [#{$ref->{tar_rec}}]), "\n";
you don't need any brackets around #{$ref->{tar_rec}} - it'll be treated as a list of values to pass to join as is.
print $fout join(",", $ref->{code}, #{$ref->{tar_rec}}), "\n";

Perl Program to Count Two Character Frequencies

I am trying to find two character strings in a text file and print them and their frequencies out.
#!/usr/bin/perl
#digram finder
use strict; use warnings;
#finds digrams in a file and prints them and their frequencies out
die "Must input file\n" if (#ARGV != 1);
my ($file) = #ARGV;
my %wordcount;
open (my $in, "<$file") or die "Can't open $file\n";
while (my $words = <$in>){
chomp $words;
my $length = length($words);
for (my $i = 0; $i<$length; $i++){
my $duo = substr($words, $i; 2);
if (not exists $wordcount{$duo}){
$wordcount{$duo} = 1;
}
else {
$wordcount{$duo}++;
}
}
}
foreach my $word (sort {$wordcount{$b} cmp $wordcount{$a}} keys %wordcount){
print "$word\t$wordcount{$duo}\n";
}
close($in);
First I set the text file to a string $words.
Then, I run a for loop and create a substring $duo at each position along $words
If $duo doesn't exist within the hash %wordcount, then the program creates the key $duo
If $duo does exist, then the count for that key goes up by 1
Then the program prints out the digrams and their frequencies, in order of decreasing frequency
When I try to run the code, I get the error message that I forgot to declare $word on line 17 but I do not even have the string $word. I am not sure where this error message is coming from. Can someone help me find where the error is coming from?
Thank you
My best guess is that you actually have $word instead of $words; a typo. If the compilation found the symbol $word in the text then it's probably there.
However, I'd also like to comment on the code. A cleaned up version
while (my $words = <$in>) {
chomp $words;
my $last_duo_idx = length($words) - 2;
for my $i (0 .. $last_duo_idx) {
my $duo = substr($words, $i, 2);
++$wordcount{$duo};
}
}
my #skeys = sort { $wordcount{$b} <=> $wordcount{$a} } keys %wordcount;
foreach my $word (#skeys) {
print "$word\t$wordcount{$word}\n";
}
This runs correctly on a made-up file. (I sort separately only so to not run off of the page.)
Comments
Need to stop one before last in the line, and substr starts from 0; thus -2
One almost never needs a C-style loop
There is no need here to test for existence of a key. If it doesn't exist it is autovivified (created), then incremented to 1 with ++; otherwise the count is incremented.
To sort numerically use <=>, not cmp
Typos:
substr($words, $i; 2) needs a , not ;, so substr($words, $i, 2)
$wordcount{$duo} in print should be $wordcount{$word}.
I am not sure about naming: why is a line of text called $words?

Split an array into two. Predetermined sizes, might not always be equal

I need to iterate over many files in a directory and split each file into two parts. I need to keep lines intact (I can't split on bite size). I also can't always assume that the file has an equal number of lines. I could use the "split" function, but am looking for a faster way of going through my files and to avoid the standard output names "xaa" and "xab" it generates.
The easiest would be to make two subsequent substrings of an array in the sizes specified ($number_of_group_one and $number_of_group_two). I can't find out how to do this. Instead I am trying to push the lines into different arrays- filling one up until a certain number of lines and then "spill over" into the other array until there are no more lines left to push. However, this approach yields two output arrays that both have exactly double the number of input lines. Here is my code:
#!/usr/bin/perl
use warnings;
use strict;
my ($directory) = #ARGV;
my $dir = "$directory";
my #arrayoffiles = glob "$dir/*";
my #arrayoflines_one;
my #arrayoflines_two;
my $counter = 0;
foreach my $filename(#arrayoffiles){
my #arrayoflines_one;
my #arrayoflines_two;
my #lines = read_lines($filename);
my $NumberofLines = #lines;
my $number_of_group_one = int($NumberofLines/2);
my $number_of_group_two = ($NumberofLines - $number_of_group_one);
foreach my $line (#lines){
$counter++;
push (#arrayoflines_one, $line, "\n");
if ($counter == $number_of_group_one){
push (#arrayoflines_two, $line, "\n");
}
}
}
sub read_lines {
my ($file) = #_;
open my $in, '<', $file or die $!;
local $/ = undef; #slurps the whole file in as one
my $content = <$in>;
return split /\s/, $content;
close $in;
}
I hope this is clear. Thanks for your help!
This is a good use case for splice:
my #lines = read_lines($filename);
my #lines1 = splice #lines, 0, #lines/2;
will put (about) half of your lines from #lines into #lines1, removing them (and leaving about half of the lines) from #lines.

Print regex match Perl

Hello I am in the process of making a program that matches a given set of keywords to a file.
I want to output the matched data to a text file and include the regex keyword that triggered the match.
Below is my code related to my issue:
my $counter = 0;
foreach($words)
{
while($line = <FILE>)
{
if($line =~ /$words/)
{
print "#array[$counter] $line\n";
print OUTPUT $line;
}
}
$counter ++;
}
This does not produce the expected outcome. It works perfectly for the first element in the array but for the rest it just simply prints the first one again. I believe the counter is not being incremented.
Is there a better / easier way to get the current element being used in the loop? or even get the current regex match?
The problem is that <FILE> exhausts the file for the first word. For the next word, <FILE> tries to read at the end of the file, which means the whole loop is skipped.
You can iterate over the words inside the loop over the file, or you can seek
back to the beginning of the file at the end of the loop.
Here is what you should do:
use strict;
use warnings;
use 5.016;
my $fname = 'data.txt';
my #patterns = (
'do.',
'.at',
'.ir.',
);
open my $INFILE, '<', $fname
or die "Couldn't read from $fname: $!";
while (my $line = <$INFILE>) {
for my $pattern (#patterns) {
if ($line =~ /($pattern)/) {
print "$pattern --> $1";
}
}
}
close $INFILE:
Putting parentheses around parts of the regex causes perl to set the match variables $1, $2, $3, etc., which contain the match for each parenthesized group.
$line will have a newline at the end of the line, so if you write print "$line\n", you will add another newline, so your output file will have blank lines between every line you print.

Resources