I'm using flink 1.11 with Scala and i have a question regarding the schema evolution using a POJO.
In the documentation is written, that POJOs are supported for state schema evolution (with some limitations).
Are Scala case clases also considered as POJO and therefore supported?
case class WordCount(word: String, count: Int)
Or have i to write something like this:
class WordCount(var word: String, var count: Int) {
def this() {
this(null, -1)
}
}
Case classes are not POJOs. In particular, they do not satisfy:
The class has a public no-argument constructor
All non-static, non-transient fields in the class (and all superclasses) are either public (and non-final) or have a public getter- and a setter- method that follows the Java beans naming conventions for getters and setters. (afaik case classes have final fields with getters in the generated JVM class)
You can implement all required things in a normal scala class but your IDE might not support you well. An option is to create your class in Java, let your IDE beanify it and convert it to scala (or use it directly).
There is also the option to create evolution support for case classes with a custom serializer. That will eventually be available by Flink. (You could also go ahead and contribute it).
Related
I am trying to follow the steps here to create a basic Flink Aggregate UDF. I've added the dependencies () and implemented
public class MyAggregate extends AggregateFunction<Long, TestAgg> {..}
I've implemented the mandatory methods as well as a few other: accumulate, merge, etc. All this builds without errors. Now according to the docs, I should be able to register this as
StreamExecutionEnvironment sEnv = StreamExecutionEnvironment.getExecutionEnvironment();
StreamTableEnvironment sTableEnv = StreamTableEnvironment.getTableEnvironment(sEnv);
sTableEnv.registerFunction("MyMin", new MyAggregate());
But, the registerFucntion seems to want a ScalarFunction only as input. I am getting an incompatible type error: The method registerFunction(String, ScalarFunction) in the type TableEnvironment is not applicable for the arguments (String, MyAggregate)
Any help would be great.
You need to import the StreamTableEnvironment for your chosen language which is in your case org.apache.flink.table.api.java.StreamTableEnvironment.
org.apache.flink.table.api.StreamTableEnvironment is a common abstract class for the Java and Scala variants of StreamTableEnvironment. We've noticed that this part of the API is confusing for users and we will improve it in the future.
I'm using ScalaJs angular and Upickle and I try to create a filter to transform an unknown class to JSON.
What I tried :
my scope :
var myScope: MyClass = js.native
my filter:
#injectable("copy")
class CopyFilter extends Filter[Any] {
override def filter(any: Any): js.Dynamic = {
val myClass = any.getClass
fromClassToJsValue[myClass](any)
}
}
my function
def fromClassToJsValue[A](value: A)(implicit serializer: Writer[A]): js.Dynamic =
JSON.parse(write(value))
In this case my problem is getClass which returns Class[_] and not MyClass
Is there any solution to find MyClass? (Or maybe any other solution to derive a type Any?)
Broadly speaking, uPickle isn't designed to deal with that; I don't think any of the other JSON serializers are, either. That sort of Any-friendly serialization is usually based on reflection, which mostly isn't available in the JavaScript environment.
I suspect you do need a Filter per case class, albeit probably a one-liner. (Possibly done as a base trait that you mix into the case classes themselves, but I don't know Angular, so I know don't what the constraints look like.)
I'm using Objectify and wish to have its Key<> type passed around in my API. I've created an ApiTransformer, but my questions is where to declare it, since the serialized Key<> class is not available, hence I cannot declare its transformer as a class annotation. I tried declaring it in the #Api annotation, but it doesn't work, I still get the error:
There was a problem generating the API metadata for your Cloud Endpoints classes: java.lang.IllegalArgumentException: Parameterized type com.googlecode.objectify.Key<[my package].User> not supported.
The ApiTransformer looks like:
public class KeyTransformer implements Transformer<Key<?>, String> {
public String transformTo(Key<?> in) {
return in.getString();
}
public Key<?> transformFrom(String in) {
return Key.valueOf(in);
}
}
And in my #Api I have:
#Api(name = "users", version = "v1",transformers = {KeyTransformer.class})
Unfortunately you can't. As you said you need to declare it on the Key class, your only chances to make this work are either.
1) Recompile the Key class for objectify with the #transformer annotation.
2) Extend the Key class with your own implementation and define the transformer there.
I don't really like any of those options so the way i usually resolve this is to hide the key object getter (by using #ApiResourceProperty(ignored=AnnotationBoolean.TRUE)) and only expose the id from that key.
That way you get a Endpoints frendly object, the only downside is you'll have to reconstitute the key using Key.create(YourClass.class, longId) manually whenever you need it.
You can add transforms to 3rd party classes by listing the transform in #Api annotation. I'm not dead sure it'll work parameterized class, but I don't see why not.
https://cloud.google.com/appengine/docs/java/endpoints/javadoc/com/google/api/server/spi/config/Api#transformers()
I need to poll a directory and narrow the files with a case insentive expression.
With version 2.10 camel adds support for antInclude which is what I look into, unfortunately antInclude is case sensitive, so are other filtering expressions. Implementing GenericFileFilter is not an option, since the filtering patterns are not known at compile time as I read them from database at runtime and I have multiple file rules each with a different pattern.
I programmatically create several routes in a loop, where each file route has a different case insensitive filtering pattern. I would appreciate if camel file component supports case insensitive expressions, or is there any other way without creating myself a new file component in camel?
public class MyRouter extends RouteBuilder {
#Override
public void configure() throws Exception {
Vector<FileTransferEntity> list = TransferDAO.getTransferList();
for(FileTransferEntity t : list) {
fromF("ftp://ftpuser#ftpserver/some-directory?antInclude=%s", t.getFileMask()).
toF("mock:result");//depending on t, action will change.
}
}
should be able to use a custom filter instead...see camel-file2 for information or see this example...
https://svn.apache.org/repos/asf/camel/trunk/camel-core/src/test/java/org/apache/camel/component/file/FileConsumerFileFilterTest.java
I've got a repeating pattern in my routes - a certain Processor needs the same 3 Headers set every time I call it, so I've got the following code in my routes about 10+ times:
.whatever()
.setHeader("foo1", "bar1")
.setHeader("foo2", "bar2")
.setHeader("foo3", "bar3")
.processRef("processorBazThatNeedsHeaders")
.whatever()
The headers are populated differently every time, so abstracting this out into a subroute doesn't really buy me anything.
What I love to be able to do is subclass RouteDefinition to have another method in my DSL that would allow me to do this:
.whatever()
.bazProcessor("bar1", "bar2", "bar3")
.whatever()
and in 'bazProcessor', set the headers and call the processor.
I've tried to do this but it seems that it's only possible with some serious probably-not-future-proof surgery, and it seems that others have had similar luck.
I need them to be set as headers as opposed to passing them as parameters directly to the processor because the values are also used after the processor for routing.
Is there some hidden facility to achieve something like this?
By subclassing the RouteDefinition your extension will only be visible direct after from(...). This could be a limitation if you would like to use the DSL extension for example after the filter(...) DSL.
A simpler approach would be to encapsulate the logic somewhere, and use it in a class that implements the org.apache.camel.Processor interface, and then call an overload of .process(...), or bean(...) in the route to use the logic. You will be actually very closed to a DSL extension if you use a meaningful name for the Processor instance or a method, that returns that Processor instance. Here is an example of the suggested approach. At the end, your code could look like:
.whatever()
.process(setTheHeadersForBaz)
.whatever()
Just for reference: if you really need to do a DSL, there is a project that extends the Camel DSL based on Groovy. I guess a Scala way based on the Camel Scala DSL could be also an option.
Though slightly irrelevant, following is an example of extending Scala DSL.
We can create an implicit methods to DSL trait via an implicit class.
object DSLImplicits {
implicit class RichDSL(val dsl: DSL) {
def get = dsl.setHeader(Exchange.HTTP_METHOD, _ => HttpMethods.GET.name)
def post = dsl.setHeader(Exchange.HTTP_METHOD, _ => HttpMethods.POST.name)
}
}
And use it like this.
import DSLImplicits.RichDSL
//----------------------------
from("someWhere")
//Do some processing
.get.to("http://somewhere.com")
More details #
http://siliconsenthil.in/blog/2013/07/11/apache-camel-with-scala-extending-dsl/
So you only set the headers because you want the Processor to have access to those values?
If so then a simple example using a Factory could look like this:
whatever()
.process(BazProcessorFactory.instance("bar1", "bar2", "bar3"))
.whatever()
Where the BazProcessorFactory is just a wrapper around your Processor:
public class BazProcessorFactory {
public Processor instance(final String...vals) {
return new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
//access your array of values here
System.out.println("Foo1 = "+vals[0]);
}
}
}
}