I am new to the GCP world. I have to check whether my batchSetting for publishing message to pub sub really work or not. This is the batch setting:
private BatchingSettings getBatchingSettings() {
long requestBytesThreshold = 10000L;
long messageCountBatchSize = 100L;
Duration publishDelayThreshold = Duration.ofMillis(2000);
BatchingSettings batchingSettings = BatchingSettings.newBuilder()
.setElementCountThreshold(messageCountBatchSize)
.setRequestByteThreshold(requestBytesThreshold)
.setDelayThreshold(publishDelayThreshold)
.build();
return batchingSettings;
}
I have to check whether pub sub publishes the message in batch of 100 or not.
Is there any way to check how many messages really published by per batch?
As it is explained in the documentation you can monitor Pub/Sub in Cloud monitoring. When you follow the link you will go to the Cloud Monitoring on your project.
In Metrics explorer its possible to create a metric of flowing configuration:
Recourse type: Cloud Pub/Sub Topic
Metric: Publish message operations
Group by: topic_id
Aggergator: sum
Minimum alignment period: 1 minutes
In "SHOW ADVANCED OPTIONS" set:
Aligner: sum
If you search such chart in some dashboard you can check the count of published massages there. Now just submit separate testing batch and wait for peak on the chart. When you hoover on the chart line you will see the number of massages in particular time period. Sometimes it will be decided into more parts, but in such small batch like 100 it should be no more than 2. So its enough to add 2 numbers.
Of course you can create more sophisticated metrics. This is just example.
Related
I have an activity diagram created in plantUML:
#startuml
|#LightBlue|ILC Running Continously|
start
note
ILC grabs data from
VOLTTRON message bus
for HVAC system
and electric meter
end note
repeat :Calculate averaged power;
repeat
repeat while (Is Averaged Power above "Demand Limit"?) is (no)
->yes;
repeat: Calculated needed demand reduction;
:Calculate AHP Wieghts;
:Curtail selected loads;
note
Typical to VOLTTRON edge
device on interacting
with building systems via
protocol driver framework
end note
:Wait in Minutes "Control Time";
backward: Curtail more;
note
Devices already curtailed
saved in ILC agent memory
end note
repeat while (Is "Demand Limit" goal met?) is (no)
->yes;
backward: Manage Demand;
#enduml
Is it possible to send someone a link of the entire configuration which is NOT just the output picture link which is in the snip below but the entire configuration for someone else to modify?!
I don't know whether or not it is documented anywhere but from experience / trial error and observing the URLs with the plantuml webserver one can see that the URLs are:
uml: http://www.plantuml.com/plantuml/uml/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000
png: http://www.plantuml.com/plantuml/png/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000
svg: http://www.plantuml.com/plantuml/svg/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000
ASCII art: http://www.plantuml.com/plantuml/txt/SyfFKj2rKt3CoKnELR1Io4ZDoSa70000
in other words the part uml / png / svg /txt changes according to the outcome.
Also when one tries a none existing part it is reverted back to uml
Any chance I could get a tip for proper way to build an agent that could do read multiple points from multiple devices on a BACnet system? I am viewing the actuator agent code trying learn how to make the proper rpc call.
So going through the agent development procedure with the agent creation wizard.
In the init I have this just hard coded at the moment:
def __init__(self, **kwargs):
super(Setteroccvav, self).__init__(**kwargs)
_log.debug("vip_identity: " + self.core.identity)
self.default_config = {}
self.agent_id = "dr_event_setpoint_adj_agent"
self.topic = "slipstream_internal/slipstream_hq/"
self.jci_zonetemp_string = "/ZN-T"
So the BACnet system in the building has JCI VAV boxes all with the same zone temperature sensor point self.jci_zonetemp_string and self.topic is how I pulled them into volttron/config store through BACnet discovery processes.
In my actuate point function (copied from CSV driver example) am I at all close for how to make the rpc call named reads using the get_multiple_points? Hoping to scrape the zone temperature sensor readings on BACnet device ID's 6,7,8,9,10 which are all the same VAV box controller with the same points/BAS program running.
def actuate_point(self):
"""
Request that the Actuator set a point on the CSV device
"""
# Create a start and end timestep to serve as the times we reserve to communicate with the CSV Device
_now = get_aware_utc_now()
str_now = format_timestamp(_now)
_end = _now + td(seconds=10)
str_end = format_timestamp(_end)
# Wrap the timestamps and device topic (used by the Actuator to identify the device) into an actuator request
schedule_request = [[self.ahu_topic, str_now, str_end]]
# Use a remote procedure call to ask the actuator to schedule us some time on the device
result = self.vip.rpc.call(
'platform.actuator', 'request_new_schedule', self.agent_id, 'my_test', 'HIGH', schedule_request).get(
timeout=4)
_log.info(f'*** [INFO] *** - SCHEDULED TIME ON ACTUATOR From "actuate_point" method sucess')
reads = publish_agent.vip.rpc.call(
'platform.actuator',
'get_multiple_points',
self.agent_id,
[(('self.topic'+'6', self.jci_zonetemp_string)),
(('self.topic'+'7', self.jci_zonetemp_string)),
(('self.topic'+'8', self.jci_zonetemp_string)),
(('self.topic'+'9', self.jci_zonetemp_string)),
(('self.topic'+'10', self.jci_zonetemp_string))]).get(timeout=10)
Any tips before I break something on the live system greatly appreciated :)
The basic form of an RPC call to the actuator is as follows:
# use the agent's VIP connection to make an RPC call to the actuator agent
result = self.vip.rpc.call('platform.actuator', <RPC exported function>, <args>).get(timeout=<seconds>)
Because we're working with devices, we need to know which devices we're interested in, and what their topics are. We also need to know which points on the devices that we're interested in.
device_map = {
'device1': '201201',
'device2': '201202',
'device3': '201203',
'device4': '201204',
}
building_topic = 'campus/building'
all_device_points = ['point1', 'point2', 'point3']
Getting points with the actuator requires a list of point topics, or device/point topic pairs.
# we only need one of the following:
point topics = []
for device in device_map.values():
for point in all_device_points:
point_topics.append('/'.join([building_topic, device, point]))
device_point_pairs = []
for device in device_map.values():
for point in all_device_points:
device_point_pairs.append(('/'.join([building_topic, device]),point,))
Now we send our RPC request to the actuator:
# can use instead device_point_pairs
point_results = self.vip.rpc.call('platform.actuator', 'get_multiple_points', point_topics).get(timeout=3)
maybe it's just my interpretation of your question, but it seems a little open-ended - so I shall respond in a similar vein - general (& I'll try to keep it short).
First, you need the list of info for targeting each device in-turn; i.e. it might consist of just a IP(v4) address (for the physical device) & the (logical) device's BOIN (BACnet Object Instance Number) - or if the request is been routed/forwarded on by/via a BACnet router/BACnet gateway then maybe also the DNET # & the DADR too.
Then you probably want - for each device/one at a time, to retrieve the first/0-element value of the device's Object-List property - in order to get the number of objects it contains, to allow you to know how many objects are available (including the logical device/device-type object) - that you need to retrieve from it/iterate over; NOTE: in the real world, as much as it's common for the device-type object to be the first one in the list, there's no guarantee it will always be the case.
As much as the BACnet standard started allowing for the retrieval of the Property-List (property) from each & every object, most equipment don't as-yet support it, so you might need your own idea of what properties (/at least the ones of interest to you) that each different object-type supports; probably at the very-very least know which ones/object-types support the Present-Value property & which ones don't.
One ideal would be to have the following mocked facets - as fakes for testing purposes instead of testing against a live/important device (- or at least consider testing against a noddy BACnet enabled Raspberry PI or the harware-based like):
a mock for your BACnet service
a mock for the BACnet communication stack
a mock for your device as a whole (- if you can't write your own one, then maybe even start with the YABE 'Room Control Simulator' as a starting point)
Hope this helps (in some way).
I have implement the CEP Pattern in Flink which is working as expected connecting to local Kafka broker. But when i connecting to cluster based cloud kafka setup, the Flink CEP is not triggering.
final StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();
//saves checkpoint
env.getCheckpointConfig().enableExternalizedCheckpoints(
CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime);
I am using AscendingTimestampExtractor,
consumer.assignTimestampsAndWatermarks(
new AscendingTimestampExtractor<ObjectNode>() {
#Override
public long extractAscendingTimestamp(ObjectNode objectNode) {
long timestamp;
Instant instant = Instant.parse(objectNode.get("value").get("timestamp").asText());
timestamp = instant.toEpochMilli();
return timestamp;
}
});
And also i am getting Warn Message that,
AscendingTimestampExtractor:140 - Timestamp monotony violated: 1594017872227 < 1594017873133
And Also i tried using AssignerWithPeriodicWatermarks and AssignerWithPunctuatedWatermarks none of one is working
I have attached Flink console screenshot where Watermark is not assigning.
Updated flink console screenshot
Could Anyone Help?
CEP must first sort the input stream(s), which it does based on the watermarking. So
the problem could be with watermarking, but you haven't shown us enough to debug the cause. One common issue is having an idle source, which can prevent the watermarks from advancing.
But there are other possible causes. To debug the situation, I suggest you look at some metrics, either in the Flink Web UI or in a metrics system if you have one connected. To begin, check if records are flowing, by looking at numRecordsIn, numRecordsOut, or numRecordsInPerSecond and numRecordsOutPerSecond at different stages of your pipeline.
If there are events, then look at currentOutputWatermark throughout the different tasks of your job to see if event time is advancing.
Update:
It appears you may be calling assignTimestampsAndWatermarks on the Kafka consumer, which will result in per-partition watermarking. In that case, if you have an idle partition, that partition won't produce any watermarks, and that will hold back the overall watermark. Try calling assignTimestampsAndWatermarks on the DataStream produced by the source instead, to see if that fixes things. (Of course, without per-partition watermarking, you won't be able to use an AscendingTimestampExtractor, since the stream won't be in order.)
My app runs on App Engine Standard and the Go runtime.
I have this trace for my recent request:
There is a big gap between the "urlfetch" span and the "datastore_v3" span, because my app processes some CPU-bound computation for ~1000ms.
I would love to programmatically add my computation as a custom span into the Trace view, and get something like this:
Is there a way to do this in my app written in go? (source here)
It appears it might be possible. From Setting Up Stackdriver Trace for Go:
Alpha
This is an alpha release of the OpenCensus package for Go. These
libraries might be changed in backward-incompatible ways and are not
recommended for production use. They are not subject to any SLA or
deprecation policy.
Stackdriver Trace can be used by Go applications using the
OpenCensus package for Go.
Stackdriver Trace's Go support is provided by OpenCensus, a set
of tracing and application metrics instrumentation libraries that work
with multiple backends. The latest details about OpenCensus for Go,
along with additional documentation and examples, can be found on its
GitHub page.
Support is enabled by default in the flexible environment, however the docs make no mention about the standard environment (if that's your case I'd say just give it a try). From App Engine:
On Google App Engine flexible environment, the Stackdriver Trace API
access scope is enabled by default, and the OpenCensus client
library can be used without needing to provide credentials or a
project ID.
An application code sample is provided on the same page.
I could make it work with the new AppEngine runtime for Go 1.11 (currently in beta) and OpenCensus with the Stackdriver exporter.
In order to attach my custom span to the main Trace of the request, I use this utility func:
// Start a new span "With Remote Parent"
func startSpanfWRT(r *http.Request, msg string, args ...interface{}) (c2 context.Context, endSpan func()) {
caption := fmt.Sprintf(msg, args...)
c := r.Context()
spanContext, ok := (&propagation.HTTPFormat{}).SpanContextFromRequest(r)
if !ok {
return c, func() {}
}
var span *trace.Span
c2, span = trace.StartSpanWithRemoteParent(c, caption, spanContext)
endSpan = func() {
span.End()
}
return c2, endSpan
}
Note that it requires the *http.Request as argument (a context.Context wouldn't be enough here).
Here is the sample app source code.
As a span needs to be started and then later stopped, the start func returns an "end" callback, and a new Context as well.
It is fine to call startSpanfWRT multiple times, and they may overlap. It requires to pass the *http.Request around, which is not super-convenient (usually we only pass Contexts around).
However, after a call to startSpanfWRT, you may add children spans conveniently, just paying attention to the respective Contexts:
c2, childSpan := trace.StartSpan(c, caption)
One of the more common complaints I have read about the AppEngine database (for Java) is that it is extremely slow when it come to "cold start time". What does this mean? And is it something I should be worried about?
It is something you should be worried about.
Google App Engine spins up a new JVM to service requests when your app has not had any requests for a certain time period. Getting a handle on the datastore from "cold" - i.e. for the first time in a JVM - can take a considerable amount of time, as much as 5+ seconds.
After you have a handle on the datastore (normally an instance of PersistenceManager), everything is fine (for the life of the JVM!).
EDIT:
Spinning up a fresh JVM in GAE-Java is also slow. Read http://code.google.com/appengine/docs/java/datastore/overview.html and you will see that they use a Singleton class for the availability of a PersistenceManagerFactory, as they describe the operation of instantiating one as "expensive".
You could test it out for yourself. Create a brand new application on GAE-Java that merely returns "Hello World!" and you will find that the first request to the application takes a number of seconds.
Add a request for the PersistenceManagerFactory and you will find that the first request takes a few seconds more.
EDIT EDIT:
I have now created this test for your viewing pleasure:
http://stackoverflowanswers.appspot.com/helloworld
You will either instantly see "Hello, world 0" or "Hello, world xxxx" where xxxx is a count in MS of how long it took to get a handle on the datastore. I think that the complexity and number of indexes in the datastore may have an impact on how long it takes to get a handle on the datastore, as it is quicker in this app than in some of my other apps.
PMF is an exact copy of the one provided in the app engine docs.
#SuppressWarnings("serial")
public class HelloWorldServlet extends HttpServlet {
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
long a = System.currentTimeMillis();
PersistenceManager p = PMF.get().getPersistenceManager();
long b = System.currentTimeMillis() - a;
resp.setContentType("text/plain");
resp.getWriter().println("Hello, world "+b);
}
}
EDIT EDIT EDIT:
I changed my code so that it instantiates a PersistenceManagerFactory with each request and now it throws 500 server errors, and in the logs:
javax.jdo.JDOFatalUserException:
Application code attempted to create a
PersistenceManagerFactory named
transactions-optional, but one with
this name already exists! Instances
of PersistenceManagerFactory are
extremely slow to create and it is
usually not necessary to create one
with a given name more than once.
Instead, create a singleton and share
it throughout your code. If you
really do need to create a duplicate
PersistenceManagerFactory (such as for
a unittest suite), set the
appengine.orm.disable.duplicate.pmf.exception
system property to avoid this error.
I don't think I need to provide any more proof that getting a handle on the datastore in app engine is SLOW.