In this part of the Perl tutorial we are going to see how to make sure we only have distinct values in an array.
Perl 5 does not have a built in function to filter out duplicate values from an array, but there are several solutions to the problem.


List::MoreUtils

Depending on your situation, probably the simplest way is to use the uniq function of the List::MoreUtils module from CPAN. 

    

  1. use List::MoreUtils qw(uniq);
    2. 

  2.  
    3. 

  3. my @words = qw(foo bar baz foo zorg baz);
    4. 

  4. my @unique_words = uniq @words;
    5.
A full example is this: 

    

  1. use strict;
    2. 

  2. use warnings;
    3. 

  3. use 5.010;
    4. 

  4.  
    5. 

  5. use List::MoreUtils qw(uniq);
    6. 

  6. use Data::Dumper qw(Dumper);
    7. 

  7.  
    8. 

  8. my @words = qw(foo bar baz foo zorg baz);
    9. 

  9.  
    10. 

  10. my @unique_words = uniq @words;
    11. 

  11.  
    12. 

  12. say Dumper \@unique_words;
    13.
The result is: 

$VAR1 = [
        'foo',
        'bar',
        'baz',
        'zorg'
      ];
For added fun the same module also provides a function called distinct, which is just an alias of the uniq function.
In order to use this module you'll have to install it from CPAN.

Home made uniq

If you cannot install the above module for whatever reason, or if you think the overhead of loading it is too big, there is a very short expression that will do the same: 

    

  1. my @unique = do { my %seen; grep { !$seen{$_}++ } @data };
    2.
This, of course can look cryptic to someone who does not know it already, so it is recommended to define your own uniq subroutine, and use that in the rest of the code: 

    

  1. use strict;
    2. 

  2. use warnings;
    3. 

  3. use 5.010;
    4. 

  4.  
    5. 

  5. use Data::Dumper qw(Dumper);
    6. 

  6.  
    7. 

  7. my @words = qw(foo bar baz foo zorg baz);
    8. 

  8.  
    9. 

  9. my @unique = uniq( @words );
    10. 

  10.  
    11. 

  11. say Dumper \@unique_words;
    12. 

  12.  
    13. 

  13. sub uniq {
    14. 

  14.   my %seen;
    15. 

  15.   return grep { !$seen{$_}++ } @_;
    16. 

  16. }
    17.

Home made uniq explained

I can't just throw this example here and leave it like that. I'd better explain it. Let's start with an easier version: 

    

  1. my @unique;
    2. 

  2. my %seen;
    3. 

  3.  
    4. 

  4. foreach my $value (@words) {
    5. 

  5.   if (! $seen{$value}) {
    6. 

  6.     push @unique, $value;
    7. 

  7.     $seen{$value} = 1;
    8. 

  8.   }
    9. 

  9. }
    10.
Here we are using a regular foreach loop to go over the values in the original array, one by one. We use a helper hash called %seen. The nice thing about the hashes is that their keys are unique.
We start with an empty hash so when we encounter the first "foo", $seen{"foo"} does not exist and thus its value is undef which is considered false in Perl. Meaning we have not seen this value yet. We push the value to the end of the new @uniq array where we are going to collect the distinct values.
We also set the value of $seen{"foo"} to 1. Actually any value would do as long as it is considered "true" by Perl.
The next time we encounter the same string we already have that key in the %seen hash and its value is true, so the if condition will fail, and we won't push the duplicate value in the resulting array.


Shortening the home made unique function

First of all we replace the assignment of 1 $seen{$value} = 1; by the post-increment operator $seen{$value}++. This does not change the behavior of the previous solution - any positive number is going to be evaluated as TRUE, but it will allow us to include the setting of the "seen flag" within the if condition. It is important that this is a postfix increment (and not a prefix increment) as this means the increment only takes place after the boolean expression was evaluated. The first time we encounter a value the expression will be TRUE and the rest of the times it will be FALSE. 

    

  1. my @unique;
    2. 

  2. my %seen;
    3. 

  3.  
    4. 

  4. foreach my $value (@data) {
    5. 

  5.   if (! $seen{$value}++ ) {
    6. 

  6.     push @unique, $value;
    7. 

  7.   }
    8. 

  8. }
    9.
This is shorter, but we can do even better.

Filtering duplicate values using grep

The grep function in Perl is a generalized form of the well known grep command of Unix.
It is basically a filter. You provide an array on the right hand side and an expression in the block. The grep function will take each value of the array one-by-one, put it in $_, the default scalar variable of Perl and then execute the block. If the block evaluates to TRUE, the value can pass. If the block evaluates to FALSE the current value is filtered out.
That's how we got to this expression: 

    

  1. my %seen;
    2. 

  2. my @unique = grep { !$seen{$_}++ } @words;
    3.

Wrapping it in 'do' or in 'sub'

The last little thing we have to do, is wrapping the above two statements in either a do block 

    

  1. my @unique = do { my %seen; grep { !$seen{$_}++ } @words };
    2.
or, better yet, in a function with an expressive name: 

    

  1. sub uniq {
    2. 

  2.   my %seen;
    3. 

  3.   return grep { !$seen{$_}++ } @_;
    4. 

  4. }
    5.

Home made uniq - round 2

Prakash Kailasa suggested an even shorted version of implementing uniq, for perl version 5.14 and above, if there is no requirement to preserve the order of elements.
Inline: 

    

  1. my @unique = keys { map { $_ => 1 } @data };
    2.
or within a subroutine: 
    

  1. my @unique = uniq(@data);
    2. 

  2. sub uniq { keys { map { $_ => 1 } @_ } };
    3.
Let's take this expression apart:
map has a similar syntax to grep: a block and an array (or a list of values). It goes over all the elements of the array, executes the block and passes the result to the left.
In our case, for every value in the array it will pass the value itself followed by the number 1. Remember =>, aka. fat comma, is just a comma. Assuming @data has ('a', 'b', 'a') in it, this expression will return ('a', 1, 'b', 1, 'a', 1). 

    

  1. map { $_ => 1 } @data
    2.
If we assigned that expression to a hash, we would get the original data as keys, and the number 1-es as values. Try this: 

    

  1. use strict;
    2. 

  2. use warnings;
    3. 

  3.  
    4. 

  4. use Data::Dumper;
    5. 

  5.  
    6. 

  6. my @data = qw(a b a);
    7. 

  7. my %h = map { $_ => 1 } @data;
    8. 

  8. print Dumper \%h;
    9.
and you will get: 
$VAR1 = {
          'a' => 1,
          'b' => 1
        };
If, instead of assigning it to an array we wrap the above expression in curly braces, we will get a reference to an anonymous hash. 

    

  1. { map { $_ => 1 } @data }
    2.
Let's see it in action: 

    

  1. use strict;
    2. 

  2. use warnings;
    3. 

  3.  
    4. 

  4. use Data::Dumper;
    5. 

  5. my @data = qw(a b a);
    6. 

  6. my $hr = { map { $_ => 1 } @data };
    7. 

  7. print Dumper $hr;
    8.
Will print the same output as the previous one, barring any change in order in the dumping of the hash.
Finally, starting from perl version 5.14, we can call the keys function on hash references as well. Thus we can write: 

    

  1. my @unique = keys { map { $_ => 1 } @data };
    2.
and we'll get back the unique values from @data


Exercise

Given the following file print out the unique values:
input.txt: 

foo Bar bar first second
Foo foo another foo
expected output: 

foo Bar bar first second Foo another

Exercise 2

This time filter out duplicates regardless of case.
expected output: 

foo Bar first second another