How to combine columns in Pig FULL OUTER JOIN - database

Question

I have two tables:
1,'hello'
2,'world'
4,'this'
and
1,'john'
3,'king'
and I want to produce a table
1,'hello','john'
2,'world',''
3,'' ,king
4,'this' ,''
I am currently using the Pig command:
JOIN A BY code FULL OUTER,
B BY code;
but this gives me the output:
1,'hello',1,'john'
2,'world',,''
,'' ,3,king
4,'this' ,,''
I need the code columns to combine, how can I do this? Thanks

Answer 1

Yes join will always produce output like this,its a expected behavior in pig. One option could be try group operator instead of join operator.
a.txt
1,'hello'
2,'world'
4,'this'
b.txt
1,'john'
3,'king'
PigScript:
A = LOAD 'a.txt' USING PigStorage(',') AS (code:int,name:chararray);
B = LOAD 'b.txt' USING PigStorage(',') AS (code:int,name:chararray);
C = GROUP A BY code,B BY code;
D = FOREACH C GENERATE group,(IsEmpty(A.name) ? TOTUPLE('') : BagToTuple(A.name)) AS aname,(IsEmpty(B.name) ? TOTUPLE('') : BagToTuple(B.name)) AS bname;
E = FOREACH D GENERATE group,FLATTEN(aname),FLATTEN(bname);
DUMP E;
Output:
(1,'hello','john')
(2,'world',)
(3,,'king')
(4,'this',)
BagToTuple() is not available in native pig, you have to download the pig-0.11.0.jar and set it in your classpath.
Download jar from this link:
http://www.java2s.com/Code/Jar/p/Downloadpig0110jar.htm

Answer 2

A = load 'a' using PigStorage(',') as (code:int,name:chararray);
B = load 'b' using PigStorage(',') as (code:int,name:chararray);
C = join A by code full outer ,B by code;
D = foreach C generate
(A::code IS NULL ? B::code : A::code) AS code,
A::name as aname, B::name as bname;
dump D;
the result is
(1,'hello','john')
(2,'world',)
(3,,'king')
(4,'this,)

Answer 3

You can use union and then do a groupBy
Union A,B will give you:
1,'hello'
2,'world'
4,'this'
1,'john'
3,'king'
Now do a groupBy based on id. This will give you:
1, {'hello', 'john'}
2, {'world'}
3, {'king'}
4, {'this'}
Now you just need a udf to parse the bag. In udf iterate over each key to generate output in your format.
I also ran into same issue. This is how I solved it.

Answer 4

You can use the ternary operator after your join to re-assign a new code, based on whether it was populated in the A or B relations. In this example, if A.code is null then B.code is used, else A.code is used.
C = JOIN A BY code FULL OUTER, B BY code;
D = FOREACH C GENERATE
(A.code IS NULL ? B.code : A.code) AS code,
A.field1,
A.field2,
B.field3,
B.field4;