SnappyData + Zeppelin + Kafka streaming - error while creating streaming table - apache-zeppelin

Question

I'm trying to create SnappyData streaming table using Zeppelin.
I have issue with stream table definition on argument 'rowConverter'
Zeppelin notebook is divided to a few paragraphs:
Paragraph 1:
import org.apache.spark.sql.Row
import org.apache.spark.sql.streaming.{SchemaDStream, StreamToRowsConverter}
class RowsConverter extends StreamToRowsConverter with Serializable {
override def toRows(message: Any): Seq[Row] = {
val log = message.asInstanceOf[String]
val fields = log.split(",")
val rows = Seq(Row.fromSeq(Seq(new java.sql.Timestamp(fields(0).toLong),
fields(1),
fields(2),
fields(3),
fields(4),
fields(5).toDouble,
fields(6)
)))
rows
}
}
Paragraph 2:
snsc.sql(
"CREATE STREAM TABLE adImpressionStream if not exists ("sensor_id string, metric
metric string) using kafka_stream
options (storagelevel 'MEMORY_AND_DISK_SER_2',
rowConverter 'RowsConverter',
zkQuorum 'localhost:2181',
groupId 'streamConsumer', topics 'test'");"
)
First paragraph returns error:
error: not found: type StreamToRowsConverter
class RowsConverter extends StreamToRowsConverter with Serializable {
^
<console>:13: error: not found: type Row
override def toRows(message: Any): Seq[Row] = {
^
<console>:16: error: not found: value Row
val rows = Seq(Row.fromSeq(Seq(new java.sql.Timestamp(fields(0).toLong),
Second paragraph:
java.lang.RuntimeException: Failed to load class : java.lang.ClassNotFoundException: RowsConverter
I have been trying to use default code from git:
snsc.sql("create stream table streamTable (userId string, clickStreamLog string) " +
"using kafka_stream options (" +
"storagelevel 'MEMORY_AND_DISK_SER_2', " +
" rowConverter 'io.snappydata.app.streaming.KafkaStreamToRowsConverter' ," +
"kafkaParams 'zookeeper.connect->localhost:2181;auto.offset.reset->smallest;group.id->myGroupId', " +
"topics 'test')")
but I have similar error:
java.lang.RuntimeException: Failed to load class : java.lang.ClassNotFoundException: io.snappydata.app.streaming.KafkaStreamToRowsConverter
Could you help me with this issue?
Thank you a lot.

Answer 1

You need to provide your application specific classes in the classpath. Please refer to the step of setting classpath here. Zeppelin will pick up classpath set in your spark-env.sh
https://github.com/SnappyDataInc/snappy-poc#lets-get-this-going

Answer 2

Add the snappydata interpreter to Apache Zeppelin as given here: https://snappydatainc.github.io/snappydata/howto/use_apache_zeppelin_with_snappydata/
This will enable running the Zeppelin in the lead so that the code is run in embedded mode. In particular you need to set the required jars using "-classpath" option in the cluster configuration.