I am trying to loop through plots. Each "station" is a pandas dataframe has a single water year of data (oct 1 to Spet 29). The data is being read in with this code:
sh_784_2020 = pd.read_csv("sh_784_WY2020.csv", parse_dates=['Date'])
sh_784_2020.columns = ["Index", "Date", "Temp_C","Precip_mm","SnowDepth_cm","SWE_mm","SM2","SM8","SM20"]
My plots loop through but the x-axis always starts at the year 2000 through the current date displayed but my data is from 2006-2020. Is there a way to have the xlim adjust automatically for the date range in the data frame? Or is there a way to create this plot in matyplotlib and not seaborn?
for station in stations:
station['Density'] = station['SWE_mm']/(station['SnowDepth_cm']*10)*100
station['Density range'] = pd.cut( station['Density'], [-np.inf, 25, 30, 35, 40, np.inf])
Date = station.loc[:, 'Date'].values
SWE_mm = station.loc[:, 'SWE_mm'].values
Density = station.loc[:, 'Density'].values
sns.scatterplot(station['Date'], station['SWE_mm'], hue='Density range', data= station, edgecolor = 'none', palette=['grey', 'green', 'gold', 'orange', 'crimson'], alpha= 1)
plt.xlim ()
plt.show()
Plot example 1
Plot example 2
If you upgrade to seaborn 0.11 you should find that the default autoscaling works better, but you can get a good result without upgrading by creating the Axes object before plotting and setting the units, e.g. something like
ax = plt.figure().subplots()
ax.xaxis.update_units(station["Date"])
OBJECTIVE:
DISPLAY A LIST OF:
MEMBER ID/ MEMBER NAME/ DATE VISITED/ MEMBER'S GRADE
This is the SQL statement I want to code in CakePHP 4:
SELECT visits.member_id, members.mbr_name, visits.created, batches.grade_id
FROM visits
LEFT JOIN members ON members.mbrid = visits.member_id
LEFT JOIN batches ON batches.member_id = members.mbrid
WHERE batches.batch = '2020' AND (batches.sub_batch = '0' OR batches.sub_batch = '1')
ORDER BY visits.created DESC
As you can see, the second LEFT JOIN clause is only related to members, instead of the main table visits.
This is the way I code it in CakePHP 4:
$this->Visits->find()
->contain('Members.Batches', function (Query $q) {
return $q
->select('member_id', 'batch', 'sub_batch', 'grade_id')
->where(['Batches.batch' => '2020', 'Batches.sub_batch' => '0']);
})
->contain('Members')
->order(['Visits.created' => 'DESC'])
);
In the Debug mode, I can see two(2) SQL generated. Not what I expected:
SELECT
Visits.id AS Visits__id,
Visits.member_id AS Visits__member_id,
Visits.vst_datetime AS Visits__vst_datetime,
Visits.location_id AS Visits__location_id,
Visits.created AS Visits__created,
Visits.modified AS Visits__modified,
Members.id AS Members__id,
Members.mbrid AS Members__mbrid,
Members.mbr_name AS Members__mbr_name,
Members.mbr_type AS Members__mbr_type,
Members.created AS Members__created,
Members.modified AS Members__modified
FROM
visits Visits
LEFT JOIN members Members ON Members.mbrid = (Visits.member_id)
ORDER BY
Visits.created DESC
LIMIT
20 OFFSET 0
SELECT
Batches.member_id AS Batches__member_id
FROM
batches Batches
WHERE
(
Batches.member_id in (
187, 1471, 1470, 1463, 250, 350, 1000,
501, 300, 255, 254, 253, 3, 303, 215,
305, 202, 201
)
AND Batches.batch = '2020'
AND Batches.sub_batch = '0'
)
Here's the code that gives me the sql that I need:
$visits = $this->Paginator->paginate($this->Visits->find()
->select(['Members.mbrid', 'Members.mbr_name', 'Visits.created', 'Batches.grade_id'])
->leftJoinWith('Members')
->leftJoinWith('Members.Batches')
->where(['Batches.batch' => '2020', 'Batches.sub_batch' => '0'])
->order(['Visits.created' => 'DESC'])
);
As suggested by #ndm by using leftJoinWith.
I am trying to use zeppelin to plot a realtime graph. I have doing a sentiment analysis on tweets on per minute . I am able to query statically and plot a graph. But i would like this to be done dynamically. I am new to zeppelin and do not have much knowledge about angularJS. What should be the correct approach to this problem?
val final_score=uni_join.map{case((year,month,day,hour,minutes),(tweet_count,sentiment))=>(year, month, day, hour, minutes(sentiment/tweet_count).ceil)}
final_score.saveToCassandra("twitter", "score",writeConf = WriteConf(ttl = TTLOption.constant(1000)))
final_score.foreachRDD(score => {
val rowRDD =score.map{case(year,month,day,hour,minutes,sentiment) =>(year,month,day,hour,minutes,sentiment) }
val tempDF = sqlContext.createDataFrame(rowRDD)
z.angularBindGlobal("stream", parsed) //to bind parsed to stream.
tempDF.registerTempTable("realTimeTable")
})
Doing a query on the above table , i am able to get the graph. But i would like to dynamically update the graph every minute in order to keep in sync with the sentiment score .
Thanks prior.
[update] the angular part for the zeppelin notebook are as follows:
%angular
<div id="graph" style="height: 100%; width: 100%">
<canvas id="myChart" width="400" height="400"></canvas>
<div id="legendDiv"></div>
</div>
<script>
function initMap() {
var colorList = ["#fde577", "#ff6c40", "#c72a40", "#520833", "#a88399"]
var el = angular.element($('#stream'));
console.log("El is "+el) //returns el as object
angular.element(el).ready(function() {
console.log('Hello')
window.locationWatcher =el.$scope.$watch('stream', function(new, old){
console.log('changed');}, true)})
</script>
But running this code keeps returning the following error.
vendor.js:29 jQuery.Deferred exception: Cannot read property '$watch' of undefined TypeError: Cannot read property '$watch' of undefined
The spark version that i am using is 1.6
And Zeppelin is 0.6
spark-highcharts since version 0.6.3 support Spark Structured Streaming.
For a structuredDataFrame after aggregation, with the following code in one Zeppelin paragraph. The OutputMode can be either append or complete depends how the structureDataFrame is aggregated.
import com.knockdata.spark.highcharts._
import com.knockdata.spark.highcharts.model._
val query = highcharts(
structuredDataFrame.seriesCol("country")
.series("x" -> "year", "y" -> "stockpile")
.orderBy(col("year")), z, "append")
And the following code in the next paragraph. The chart in this paragraph will be updated when there are new data coming to the structureDataFrame.
StreamingChart(z)
Run following code to stop update the chart.
query.stop()
Here is the example generate structureDataFrame.
spark.conf.set("spark.sql.streaming.checkpointLocation","/usr/zeppelin/checkpoint")
case class NuclearStockpile(country: String, stockpile: Int, year: Int)
val USA = Seq(0, 0, 0, 0, 0, 6, 11, 32, 110, 235, 369, 640,
1005, 1436, 2063, 3057, 4618, 6444, 9822, 15468, 20434, 24126,
27387, 29459, 31056, 31982, 32040, 31233, 29224, 27342, 26662,
26956, 27912, 28999, 28965, 27826, 25579, 25722, 24826, 24605,
24304, 23464, 23708, 24099, 24357, 24237, 24401, 24344, 23586,
22380, 21004, 17287, 14747, 13076, 12555, 12144, 11009, 10950,
10871, 10824, 10577, 10527, 10475, 10421, 10358, 10295, 10104).
zip(1940 to 2006).map(p => NuclearStockpile("USA", p._1, p._2))
val USSR = Seq(0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
5, 25, 50, 120, 150, 200, 426, 660, 869, 1060, 1605, 2471, 3322,
4238, 5221, 6129, 7089, 8339, 9399, 10538, 11643, 13092, 14478,
15915, 17385, 19055, 21205, 23044, 25393, 27935, 30062, 32049,
33952, 35804, 37431, 39197, 45000, 43000, 41000, 39000, 37000,
35000, 33000, 31000, 29000, 27000, 25000, 24000, 23000, 22000,
21000, 20000, 19000, 18000, 18000, 17000, 16000).
zip(1940 to 2006).map(p => NuclearStockpile("USSR/Russia", p._1, p._2))
input.addData(USA.take(30) ++ USSR.take(30))
val structureDataFrame = input.toDF
And the following code can be simulate to update the chart. The chart will be updated when the following code run.
input.addData(USA.drop(30) ++ USSR.drop(30))
NOTE: The example using Zeppelin 0.6.2 and Spark 2.0
NOTE: Please check the highcharts license for commercial usage
I'm using cal-heatmap for displaying a user activity for a month. My issue is that the colour change is not showing properly. My "init" function is given below.
When I provide data with integer values which have difference of 2 or 3 (eg: 8, 12, 3, 7 etc [pls note that I'm giving data as JSON]), I can't see any significant difference in colour for the blocks (screenshot is added in http://i.stack.imgur.com/xspWR.jpg - numbers given in top indicates the data corresponding to that cell).
init({
start: new Date(newDate.getFullYear(), month, 1),
cellRadius: 35,
cellSize: 58,
itemSelector: "#heatmap_busiestDays",
domain: "month", //hour|day|week|month|year
subDomain: "x_day",
subDomainTextFormat: "%b %d",
range: 1,
domainGutter: 10,
previousSelector: "#cal-heatmap-previous",
nextSelector: "#cal-heatmap-next",
displayLegend: false,
data: busiestDayHeatMap,
legendColors: {
min: "#A2F37B",
max: "#26911F",
empty: "white"
}
});
Am I missing anything in settings? Any help will be greatly appreciated. Thanks in advance.