I want to create a flink project written in scala.
when i importing
'flink-scala_2.11',
'flink-streaming-scala_2.11'
'flink-core',
i can run a code like:
val someEventScalaStream = myScalaStreamm.map{x=>x.Part3} (it is using scala map function)
but i see that for each:
flink-connector-rabbitmq_2.11
flink-connector-filesystem_2.11
flink-clients_2.11
all of them depends on 'flink-java'
so when all of them imported into my solution the project is trying to use the map function of the java version & doesn't compile.
any solution?
thanks.
Generally apis for java and scala are put into separate packages. So make sure you use the right one. E.g. for DataStream you have
Java:
org.apache.flink.streaming.api.datastream.DataStream
Scala:
org.apache.flink.streaming.api.scala.DataStream
In this case in scala it is recommended to import:
org.apache.flink.streaming.api.scala._
If that does not cover your question could you post a concrete example when a java version of a map function is used?
Related
As mentioned in the node version installation instructions, is Python runtime needed to use Tensorflow.js Node? I can install whatever is required but not sure if our production servers have it.
It depends of what you want to do.
If you already have a tensorflow model written in python that you would like to deploy for inference in nodejs,
you can use the tensorflow.js converter. In this case you will need a python runtime
since the version 1.3 of tfjs-node, it is possible to load directly the savedModel in js (only possible in nodejs) using loadSavedModel.
But If you want to write your complete pipeline in js, you don't need to have python installed.
Is it possible to use PyFlink with python machine learning libraries such as LightGBM for a streaming application? Is there any good example for this?
There is no complete example but you can take a loot at Getting Started with Flink Python and then take a look at how Python UDFs can be used: UDFs in the Table API.
In Fink source, there are flink-stream-java and flink-stream-scala modules. Why do we need two modules for flink streaming?
https://github.com/apache/flink/tree/master/flink-streaming-java
https://github.com/apache/flink/tree/master/flink-streaming-scala
Both flink-stream-java and flink-stream-scala provide a similar API to manage Flink Streams ; you only have to use one of them, depending on your language.
Please note that whatever your choice, some dependencies like flink-runtime and flink-clients depend on a version of scala (2.11 or 2.12), because Flink is based on a framework written in scala, Akka.
There is an ongoing effort to remove scala dependency from a higher level API, flink-table (FLINK-11063).
flink-stream-java is the implement of java api for stream. flink-stream-scala is the implement of scala api for stream. So you can find DataStream.java in flink-stream-java, and DataStream.scala in flink-stream-scala.
These two modules will accomplish the same function, but different developers receive different languages, and personal task scala is more suitable for operator description in languages such as big data, flink spark, etc.
Say there we have the package encoding/json. Can I just create a package mypackage and embed all the functions (at least the public functions) into my package without to copy them by hand and basically do calls back to the actual json package? I'm developing a cross platform (Google app engine / native ) solution and I would find a such solution quite useful.
No.
It sounds like you want some sort of package inheritance, this is not a supported feature.
JGraphT has a GraphMLExporter that enables to export a graph to a GraphML file.
There doesn't seem to be a GraphMLImporter.
Is there a simple way to generate a graph in JGraphT from a .grphml file?
Starting from version 1.0.0 JGraphT has both an importer and an exporter for GraphML. The classes are named GraphMLImporter and GraphMLExporter inside the jgrapht-ext package.
Starting from version 1.1 all importers and exporters are located in the jgrapht-io package. Additionally, a simpler version called SimpleGraphMLImporter which provides less functionality in favor of greater speed is also available in latest versions.
A demo is also available, see GraphMLDemo.
Check out jgrapht-sna project. Beside implementations of SNA algorithms, it has GraphMLImporter class.