We are using flink-cep as a standalone library for finding out patterns in a list of events.
Given the following list of events:
val patientKey = "patient"
val hrKey = "hr"
// Event
val p1e1 = Event("hr", mapOf(patientKey to 1, hr to 1))
val p1e2 = Event("hr", mapOf(patientKey to 1, hr to 2))
val p2e1 = Event("hr", mapOf(patientKey to 2, hr to 1))
val p1e3 = Event("hr", mapOf(patientKey to 1, hr to 3))
val p2e2 = Event("hr", mapOf(patientKey to 2, hr to 2))
val p3e1 = Event("hr", mapOf(patientKey to 3, hr to 1))
val p2e3 = Event("hr", mapOf(patientKey to 2, hr to 3))
val p3e2 = Event("hr", mapOf(patientKey to 3, hr to 2))
val p3e3 = Event("hr", mapOf(patientKey to 3, hr to 3))
We would like to write a pattern which returns as matches:
first match: p1e1, p1e2, p1e3
second match: p2e1, p2e2, p2e3
third match: p3e1, p3e2, p3e3
As such, this seems to be doable running CEP in a flink environment with keyed streams, but how do we do it without keyed streams. We cannot deploy a full flink env as we are running on a constrained device.
We would like to get all the heart rates gathered for a patient within 5 seconds.
Thanks
You can get the effect of keying the stream by instead putting the keyed constraint into the pattern definition. Something like this, if you use SQL:
PATTERN (A B C) WITHIN INTERVAL '5' SECOND
DEFINE
A AS A.hr = 1
B AS B.patientKey = A.patientKey AND B.hr = 2
C AS C.patientKey = B.patientKey AND C.hr = 3
If you don't use SQL, the same logic applies. (For the contiguity, you'll need to specify followedBy rather than next, since you won't have partitioned the stream by patientKey.)
For what it's worth, I can't think of any operational or performance benefit that will come from avoiding keyed streams. (In fact, CEP always uses keyed state, even if you don't explicitly use keyed streams.) The use of keyed streams makes it possible to use a larger Flink cluster and operate in parallel, but doesn't require it.
I am running MS SQL server 2019 on Windows and try to fill - besides others - a 'numberic(5,2)' field using MS ODBC driver version 18 for Linux, but always get an error if I set scale.
The table structure:
CREATE TABLE [dbo].[dboutput_test](
[MD_ID] [int] NOT NULL,
...
[FLD_ASK] [numeric](5, 2) NULL,
...
) ON [PRIMARY]
GO
I am binding the parameter using SQL_NUMERIC_STRUCT:
retcode = SQLBindParameter(inStmt,
inIdx,
SQL_PARAM_INPUT,
SQL_C_NUMERIC,
SQL_NUMERIC,
inParam._num.precision,
inParam._num.scale,
&inParam._num,
0,
&cbNumStr);
But as soon as inParam._num.scale is > 0, I get [[Microsoft][ODBC Driver 18 for SQL Server]Numeric value out of range].
If inParam._num.scale == 0, the value is set but sure with the wrong decimal point.
I even tried the example from MS Retrieve numeric data with SQL_NUMERIC_STRUCT documentation, but with the same result. I also played with the settings of the numeric field, but withot any effect.
PS.:
I found in the meantime, that when I ecplicitely set field descriptions in that way it seems to work:
SQLHDESC hdesc = NULL;
uint64_t prec = inParam._num.precision;
uint64_t scale = inParam._num.scale;
SQLGetStmtAttr(inStmt, SQL_ATTR_APP_PARAM_DESC, &hdesc, 0, NULL);
SQLSetDescField(hdesc, inIdx, SQL_DESC_TYPE, (SQLPOINTER) SQL_C_NUMERIC, 0);
SQLSetDescField(hdesc, inIdx, SQL_DESC_PRECISION, (SQLPOINTER) prec, 0);
SQLSetDescField(hdesc, inIdx, SQL_DESC_SCALE, (SQLPOINTER) scale, 0);
SQLSetDescField(hdesc, inIdx, SQL_DESC_DATA_PTR, (SQLPOINTER) &inParam._num, 0);
Is it really the way it has to be done?
I have a task in which I have to calculate the time taken by each step, for example if we are clicking on a link, how much time is taken to load the page and next step to be executed.
I want fail the test case if time taken is more then say 2 seconds.
I have tried using protractor-perf and it gives me the readings below and these don't help, or I am not able to read anything correctly.
{ Styles: 0,
Javascript: 0,
numAnimationFrames: 4625,
numFramesSentToScreen: 4625,
droppedFrameCount: 417,
meanFrameTime_raf: 18.242160830084256,
framesPerSec_raf: 54.81806729556069,
connectEnd: 1524251882749,
connectStart: 1524251882459,
domComplete: 1524251916054,
domContentLoadedEventEnd: 1524251916053,
domContentLoadedEventStart: 1524251916050,
domInteractive: 1524251916050,
domLoading: 1524251883038,
domainLookupEnd: 1524251882459,
domainLookupStart: 1524251882459,
fetchStart: 1524251882458,
firstPaint: 33600.99983215332,
loadEventEnd: 1524251916055,
loadEventStart: 1524251916054,
navigationStart: 1524251882456,
redirectEnd: 0,
redirectStart: 0,
requestStart: 1524251882749,
responseEnd: 1524251883592,
responseStart: 1524251883032,
secureConnectionStart: 0,
unloadEventEnd: 0,
unloadEventStart: 0,
loadTime: 33597,
domReadyTime: 4,
readyStart: 2,
redirectTime: 0,
appcacheTime: 1,
unloadEventTime: 0,
domainLookupTime: 0,
connectTime: 290,
requestTime: 843,
initDomTreeTime: 32458,
loadEventTime: 1 }
I have also tried using log-timestamp and I can print the timestamp to log but cannot get the difference to use it in a variable and fail the test case.
I get the output like this in the log:
[2018-04-20T19:19:13.325Z] Start
[2018-04-20T19:19:14.046Z] Step1
[2018-04-20T19:19:47.667Z] Step2
[2018-04-20T19:19:50.304Z] Step3
[2018-04-20T19:19:52.111Z] Step4
[2018-04-20T19:19:57.344Z] Step5
[2018-04-20T19:19:59.029Z] Step6
I would really appreciate your help guys, this has wasted a lot my time.
I am trying to use zeppelin to plot a realtime graph. I have doing a sentiment analysis on tweets on per minute . I am able to query statically and plot a graph. But i would like this to be done dynamically. I am new to zeppelin and do not have much knowledge about angularJS. What should be the correct approach to this problem?
val final_score=uni_join.map{case((year,month,day,hour,minutes),(tweet_count,sentiment))=>(year, month, day, hour, minutes(sentiment/tweet_count).ceil)}
final_score.saveToCassandra("twitter", "score",writeConf = WriteConf(ttl = TTLOption.constant(1000)))
final_score.foreachRDD(score => {
val rowRDD =score.map{case(year,month,day,hour,minutes,sentiment) =>(year,month,day,hour,minutes,sentiment) }
val tempDF = sqlContext.createDataFrame(rowRDD)
z.angularBindGlobal("stream", parsed) //to bind parsed to stream.
tempDF.registerTempTable("realTimeTable")
})
Doing a query on the above table , i am able to get the graph. But i would like to dynamically update the graph every minute in order to keep in sync with the sentiment score .
Thanks prior.
[update] the angular part for the zeppelin notebook are as follows:
%angular
<div id="graph" style="height: 100%; width: 100%">
<canvas id="myChart" width="400" height="400"></canvas>
<div id="legendDiv"></div>
</div>
<script>
function initMap() {
var colorList = ["#fde577", "#ff6c40", "#c72a40", "#520833", "#a88399"]
var el = angular.element($('#stream'));
console.log("El is "+el) //returns el as object
angular.element(el).ready(function() {
console.log('Hello')
window.locationWatcher =el.$scope.$watch('stream', function(new, old){
console.log('changed');}, true)})
</script>
But running this code keeps returning the following error.
vendor.js:29 jQuery.Deferred exception: Cannot read property '$watch' of undefined TypeError: Cannot read property '$watch' of undefined
The spark version that i am using is 1.6
And Zeppelin is 0.6
spark-highcharts since version 0.6.3 support Spark Structured Streaming.
For a structuredDataFrame after aggregation, with the following code in one Zeppelin paragraph. The OutputMode can be either append or complete depends how the structureDataFrame is aggregated.
import com.knockdata.spark.highcharts._
import com.knockdata.spark.highcharts.model._
val query = highcharts(
structuredDataFrame.seriesCol("country")
.series("x" -> "year", "y" -> "stockpile")
.orderBy(col("year")), z, "append")
And the following code in the next paragraph. The chart in this paragraph will be updated when there are new data coming to the structureDataFrame.
StreamingChart(z)
Run following code to stop update the chart.
query.stop()
Here is the example generate structureDataFrame.
spark.conf.set("spark.sql.streaming.checkpointLocation","/usr/zeppelin/checkpoint")
case class NuclearStockpile(country: String, stockpile: Int, year: Int)
val USA = Seq(0, 0, 0, 0, 0, 6, 11, 32, 110, 235, 369, 640,
1005, 1436, 2063, 3057, 4618, 6444, 9822, 15468, 20434, 24126,
27387, 29459, 31056, 31982, 32040, 31233, 29224, 27342, 26662,
26956, 27912, 28999, 28965, 27826, 25579, 25722, 24826, 24605,
24304, 23464, 23708, 24099, 24357, 24237, 24401, 24344, 23586,
22380, 21004, 17287, 14747, 13076, 12555, 12144, 11009, 10950,
10871, 10824, 10577, 10527, 10475, 10421, 10358, 10295, 10104).
zip(1940 to 2006).map(p => NuclearStockpile("USA", p._1, p._2))
val USSR = Seq(0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5, 25, 50, 120, 150, 200, 426, 660, 869, 1060, 1605, 2471, 3322,
4238, 5221, 6129, 7089, 8339, 9399, 10538, 11643, 13092, 14478,
15915, 17385, 19055, 21205, 23044, 25393, 27935, 30062, 32049,
33952, 35804, 37431, 39197, 45000, 43000, 41000, 39000, 37000,
35000, 33000, 31000, 29000, 27000, 25000, 24000, 23000, 22000,
21000, 20000, 19000, 18000, 18000, 17000, 16000).
zip(1940 to 2006).map(p => NuclearStockpile("USSR/Russia", p._1, p._2))
input.addData(USA.take(30) ++ USSR.take(30))
val structureDataFrame = input.toDF
And the following code can be simulate to update the chart. The chart will be updated when the following code run.
input.addData(USA.drop(30) ++ USSR.drop(30))
NOTE: The example using Zeppelin 0.6.2 and Spark 2.0
NOTE: Please check the highcharts license for commercial usage
I am very new to programming in Vizard, but I am a pretty strong .js programmer. I have an art gallery and I want a man to walk from picture to picture. He needs to wait for a few seconds at each picture.
So I have a number of walking sequences and I'm trying to use the 'ontimer' function to call the next walk sequence and also add a few seconds of delay.
It works perfectly the first time it is called, in dostuff(), but doesn't work at all in dostuff2(). I assume I am using 'ontimer' incorrectly, could anyone explain where I am going wrong?
Any help or advice would be hugely appreciated!
walkOne = vizact.walkto(4, -0.5, 4)
turnOne = vizact.turn(60)
walking_sequence = vizact.sequence( [walkOne, turnOne])
walkTwo = vizact.walkto(5.350, -0.5, -2)
turnTwo = vizact.turn(60)
walking_sequenceTwo = vizact.sequence( [walkTwo, turnTwo])
def dostuff():
male.addAction(walking_sequence)
vizact.ontimer(10,dostuff2)
def dostuff2():
male.addAction(walking_sequenceTwo)
print(vizact.ontimer)
vizact.ontimer(20,dostuff)
Cracked it!! Got rid of the ontimer completely and used waittimer instead, seems to work ok.
walkOne = vizact.walkto(4, -0.5, 4)
turnOne = vizact.turn(60)
walking_sequence = vizact.sequence(walkOne, turnOne, vizact.waittime(10))
walkTwo = vizact.walkto(5.350, -0.5, -2)
turnTwo = vizact.turn(60)
walking_sequenceTwo = vizact.sequence(walkTwo, turnTwo, vizact.waittime(10))
def dostuff():
male.addAction(walking_sequence)
dostuff2()
def dostuff2():
male.addAction(walking_sequenceTwo)
dostuff3()