How to reference subclasses of static Java classes with generics in Scala - google-app-engine

Question

I have this Java code:
public class TestMapper extends AppEngineMapper<Key, Entity, NullWritable, NullWritable> {
public TestMapper() {
}
// [... other overriden methods ...]
#Override
public void setup(Context context) {
log.warning("Doing per-worker setup");
}
}
...which I've converted to:
class TestMapper extends AppEngineMapper[Key, Entity, NullWritable, NullWritable] {
// [... other overriden methods ...]
override def setup(context: Context) {
log.warning("Doing per-worker setup")
}
}
Now the actual problem:
Context is defined as a nested class within the org.apache.hadoop.mapreduce.Mapper class:
public static class Mapper<KEYIN, VALUEIN, KEYOUT, VALUEOUT> {
//[... some other methods ...]
protected void setup(org.apache.hadoop.mapreduce.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>.Context context) throws java.io.IOException, java.lang.InterruptedException { /* compiled code */ }
public class Context extends org.apache.hadoop.mapreduce.MapContext<KEYIN,VALUEIN,KEYOUT,VALUEOUT> {
public Context(org.apache.hadoop.conf.Configuration configuration, org.apache.hadoop.mapreduce.TaskAttemptID conf, org.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN> taskid, org.apache.hadoop.mapreduce.RecordWriter<KEYOUT,VALUEOUT> reader, org.apache.hadoop.mapreduce.OutputCommitter writer, org.apache.hadoop.mapreduce.StatusReporter committer, org.apache.hadoop.mapreduce.InputSplit reporter) throws java.io.IOException, java.lang.InterruptedException { /* compiled code */ }
}
So I can't tell my Scala class where/what Context actually is. If Mapper had no generics I could reference Context via
Mapper#Context
but how can I tell that Mapper has Generics?
Mapper[_,_,_,_]#Context
...didn't work.

Answer 1

You have to supply the exact base type for your type projection, in your case
Mapper[Key, Entity, NullWritable, NullWritable]#Context
so overriding setup would be written as
override def setup(context: Mapper[Key, Entity, NullWritable, NullWritable]#Context)
Usage can be simplified by introducing a type alias
class TestMapper extends AppEngineMapper[Key, Entity, NullWritable, NullWritable] {
type Context = Mapper[Key, Entity, NullWritable, NullWritable]#Context
override def setup(context: Context) = {
// ...
}
}
If you want to write multiple mappers you can refactor this into a trait that can be mixed into your implementation:
trait SMapper[A,B,C,D] extends Mapper[A,B,C,D] {
type Context = Mapper[A,B,C,D]#Context
}
class TestMapper extends AppEngineMapper[Key, Entity, NullWritable, NullWritable]
with SMapper[Key, Entity, NullWritable, NullWritable] {
override def setup(context: Context) = {
// ...
}
}
or for plain hadoop:
class TestMapper extends SMapper[Key, Entity, NullWritable, NullWritable] {
override def setup(context: Context) = {
// ...
}
}

Answer 2

The problems people have sometimes when using the accepted answer are due to a bug in Scala compiler (link).