How does one return a PubsubMessage through protoRPC ?
Pub/sub message example,
.{
"data": string,
"attributes": {
string: string,
...
},
"messageId": string,
"publishTime": string
}
As first step you have to use Cloud Pub/Sub Client Libraries (Python) to subscribe to data in Cloud Pub/Sub, you create a subscription based on the topic, and subscribe to that.
You have to define a callback that process each received message. {1}
You can check the Hello World of Google Protocol RPC to pass the received message in the callback to the RPC service. {2}
{1}: https://google-cloud-python.readthedocs.io/en/latest/pubsub/
{2}: https://cloud.google.com/appengine/docs/standard/python/tools/protorpc/
Related
When using paho-mqtt5:test more than once with same clientId then it throw exception Client not connected but if i will use different clientId for each to and from then it will work fine
2021-10-05 19:25:28,650 ERROR [org.apa.cam.pro.err.DefaultErrorHandler] (Camel (camel-1) thread #0 - timer://test) Failed delivery for (MessageId: 871E4623819E4FB-000000000000001B on ExchangeId: 871E4623819E4FB-000000000000001B). Exhausted after delivery attempt: 1 caught: Client is not connected (32104)
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route1 ] [route1 ] [from[timer://test?period=1000] ] [ 0]
...
[route1 ] [to1 ] [paho:test ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
: Client is not connected (32104)
at org.eclipse.paho.mqttv5.client.internal.ExceptionHelper.createMqttException(ExceptionHelper.java:32)
at org.eclipse.paho.mqttv5.client.internal.ClientComms.sendNoWait(ClientComms.java:231)
at org.eclipse.paho.mqttv5.client.MqttAsyncClient.publish(MqttAsyncClient.java:1530)
at org.eclipse.paho.mqttv5.client.MqttClient.publish(MqttClient.java:564)
at org.apache.camel.component.paho.mqtt5.PahoMqtt5Producer.process(PahoMqtt5Producer.java:55)
at org.apache.camel.support.AsyncProcessorConverterHelper$ProcessorToAsyncProcessorBridge.process(AsyncProcessorConverterHelper.java:66)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:172)
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:463)
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:179)
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:64)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:184)
at org.apache.camel.impl.engine.CamelInternalProcessor.process(CamelInternalProcessor.java:398)
at org.apache.camel.component.timer.TimerConsumer.sendTimerExchange(TimerConsumer.java:210)
at org.apache.camel.component.timer.TimerConsumer$1.run(TimerConsumer.java:76)
at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
at java.base/java.util.TimerThread.run(Timer.java:506)
Here is my code which is throwing exception
#ApplicationScoped
class TestRouter : RouteBuilder() {
override fun configure() {
val mqtt5Component = PahoMqtt5Component()
mqtt5Component.configuration = PahoMqtt5Configuration().apply {
brokerUrl = "tcp://192.168.99.101:1883"
clientId = "paho123"
isCleanStart = true
}
context.addComponent("paho-mqtt5", mqtt5Component)
from("timer:test?period=1000").setBody(constant("Testing timer2")).to("paho-mqtt5:test")
from("paho-mqtt5:test").process { e ->
val body = (e.`in`?.body as? ByteArray)?.let { String(it) }
println("test body 1 => $body")
}
}
}
#William, this is expected behavior
The message broker uses the client id to differentiate between clients so it can perform housekeeping for a client connection that is no longer used
In addition, a client may have a "Last Will and Testament" that the broker keeps track of
It is acceptable to append a random number to the end of your current 'clientId' since it is likely no one but you will care about this
If you have access to the individuals login, you could use that as well but you would still want to make each session unique in case they run multiple sessions
Maybe I don't understand what your problem is
Each client must have a unique Id
What are you observing that makes you think that it is creating multiple connections for a single client?
Is there a chance you are opening multiple windows and each is generating a different clientId?
This is a good way to diagnose issues by monitoring what the server is seeing
My paho-mqtt client (Javascript) is connecting as "webclient" and I append a randome number (webclient173) to identify this client
To troubleshoot, I would suggest you close all connections on the client and monitor the log of the MQTT process
When the monitor is in place, open a connection from a client that currently has no connections
This is an example connection to my Mosquitto log file
$ tail -f /var/log/mosquitto/mosquitto.log
1635169943: No will message specified.
1635169943: Sending CONNACK to webclient173 (0, 0)
1635169943: Received SUBSCRIBE from webclient173
1635169943: testtopic (QoS 0)
1635169943: Sending SUBACK to webclient173
1635170003: Received PINGREQ from webclient173
1635170003: Sending PINGRESP to webclient173
1635170003: Received PINGREQ from webclient173
1635170003: Sending PINGRESP to webclient173
What does your log show?
I'm testing pubsub "pull" subscriber on Cloud Run using just listener part of this sample java code (SubscribeAsyncExample...reworked slightly to fit in my SpringBoot app):
https://cloud.google.com/pubsub/docs/quickstart-client-libraries#java_1
It fails to startup during deploy...but while it's trying to start, it does pull items from the pubsub queue. Originally, I had an HTTP "push" receiver (a #RestController) on a different pubsub topic and that worked fine. Any suggestions? I'm new to Cloud Run. Thanks.
Deploying...
Creating Revision... Cloud Run error: Container failed to start. Failed to start and then listen on the port defined
by the PORT environment variable. Logs for this revision might contain more information....failed
Deployment failed
In logs:
2020-08-11 18:43:22.688 INFO 1 --- [ main] o.s.web.context.ContextLoader : Root WebApplicationContext: initialization completed in 4606 ms
2020-08-11T18:43:25.287759Z Listening for messages on projects/ce-cxmo-dev/subscriptions/AndySubscriptionPull:
2020-08-11T18:43:25.351650801Z Container Sandbox: Unsupported syscall setsockopt(0x18,0x29,0x31,0x3eca02dfd974,0x4,0x28). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
2020-08-11T18:43:25.351770555Z Container Sandbox: Unsupported syscall setsockopt(0x18,0x29,0x12,0x3eca02dfd97c,0x4,0x28). It is very likely that you can safely ignore this message and that this is not the cause of any error you might be troubleshooting. Please, refer to https://gvisor.dev/c/linux/amd64/setsockopt for more information.
2020-08-11 18:43:25.680 WARN 1 --- [ault-executor-0] i.g.n.s.i.n.u.internal.MacAddressUtil : Failed to find a usable hardware address from the network interfaces; using random bytes: ae:2c:fb:e7:92:9c:2b:24
2020-08-11T18:45:36.282714Z Id: 1421389098497572
2020-08-11T18:45:36.282763Z Data: We be pub-sub'n in pull mode2!!
Nothing else after this and the app stops running.
#Component
public class AndyTopicPullRecv {
public AndyTopicPullRecv()
{
subscribeAsyncExample("ce-cxmo-dev", "AndySubscriptionPull");
}
public static void subscribeAsyncExample(String projectId, String subscriptionId) {
ProjectSubscriptionName subscriptionName =
ProjectSubscriptionName.of(projectId, subscriptionId);
// Instantiate an asynchronous message receiver.
MessageReceiver receiver =
(PubsubMessage message, AckReplyConsumer consumer) -> {
// Handle incoming message, then ack the received message.
System.out.println("Id: " + message.getMessageId());
System.out.println("Data: " + message.getData().toStringUtf8());
consumer.ack();
};
Subscriber subscriber = null;
try {
subscriber = Subscriber.newBuilder(subscriptionName, receiver).build();
// Start the subscriber.
subscriber.startAsync().awaitRunning();
System.out.printf("Listening for messages on %s:\n", subscriptionName.toString());
// Allow the subscriber to run for 30s unless an unrecoverable error occurs.
// subscriber.awaitTerminated(30, TimeUnit.SECONDS);
subscriber.awaitTerminated();
System.out.printf("Async subscribe terminated on %s:\n", subscriptionName.toString());
// } catch (TimeoutException timeoutException) {
} catch (Exception e) {
// Shut down the subscriber after 30s. Stop receiving messages.
subscriber.stopAsync();
System.out.printf("Async subscriber exception: " + e);
}
}
}
Kolban question is very important!! With the shared code, I would like to say "No". The Cloud Run contract is clear:
Your service must answer to HTTP request. Out of request, you pay nothing and no CPU is dedicated to your instance (the instance is like a daemon when no request is processing)
Your service must be stateless (not your case here, I won't take time on this)
If you want to pull your PubSub subscription, create an endpoint in your code with a Rest controller. While you are processing this request, run your pull mechanism and process messages.
This endpoint can be called by Cloud Scheduler regularly to keep the process up.
Be careful, you have a max request processing timeout at 15 minutes (today, subject to change in a near future). So, you can't run your process more than 15 minutes. Make it resilient to fail and set your scheduler to call your service every 15 minutes
I am facing the below issue with Azure Data Factory using Logic App.
I am using the Azure Data Factory pipeline for migration and Logic App for sending "Success & Failure" notification to the technical team.
Now success is working fine as the message is hardcoded, but failure is not as the Logic App web activity is not able to parse data factory pipeline error.
Here is the input that is going to Logic App web activity
Input
{
"url": "https://xxxxxxxxxxxxxxxxx",
"method": "POST",
"headers": {},
"body": "{\n \"title\": \"PIPELINE RUN FAILED\",\n \"message\":\"Operation on target Migration Validation failed: Execution fail against sql server. Sql error number: 50000. Error Message: The DELETE statement conflicted with the REFERENCE constraint \"FK_cmclientapprovedproducts_cmlinkclientchannel\". The conflict occurred in database \"Core7\", table \"dbo.cmClientApprovedProducts\", column 'linkclientchannelid'.\",\n \"color\": \"Red\",\n \"dataFactoryName\": \"LFC-TO-MCP-ADF\",\n \"pipelineName\": \"LFC TO MCP MIGRATION\",\n \"pipelineRunId\": \"f4f84365-58f0-4da1-aa00-64c3a4daa9e1\",\n \"time\": \"2020-07-31T22:44:01.6477435Z\"\n}"
}
Here is the error logic app is throwing
failures
{
"errorCode": "2108",
"message": "{\"error\":{\"code\":\"InvalidRequestContent\",\"message\":\"The request content is not valid and could not be deserialized: 'After parsing a value an unexpected character was encountered: F. Path 'message', line 3, position 202.'.\"}}",
"failureType": "UserError",
"target": "Send Failed Notification",
"details": []
}
I have tried various options, like set variable and convert by using various existing methods (string, json, replace etc), but no luck
e.g #string(activity('LOS migration').Error.Message)
Struggling almost all day this...please suggest if anyone faced a similar issue...
Below is the data flow activity
now it is working...
Pasting body content into the body text field box WITHOUT clicking on 'Add Dynamic Content' in web activity calling Logic App.
For the failure case, pass the error output use #{activity('LOS migration').error.message.
For sending email, it doesn't know if it's going to send a failure or success email. We have to adapt the body so the activity can use parameters, which we'll define later:
{
"DataFactoryName": "#{pipeline().DataFactory}",
"PipelineName": "#{pipeline().Pipeline}",
"Subject": "#{pipeline().parameters.Subject}",
"ErrorMessage": "#{pipeline().parameters.ErrorMessage}",
"EmailTo": "#pipeline().parameters.EmailTo"
}
We can reference this variables in the body by using the following format: #pipeline().parameters.parametername. For more details, you could refer to this article.
If you want to use the direct error message of the data factory activity as an input to the logic app email expression, you could try.
"ErrorMessage": "#{string(replace(activity('activity_name').Error.Message, '"',''''))}"
Replace 'activity_name' with your failing activity name.
In Google documentation, they explained how to use "statusReport" trait for query intent when there is an error or exception occurred for a device. I'm facing issue while using it for success status without any exception. I tried sending the response with simple status as SUCCESS, Google Home is saying the response "Sorry Unable to reach device".
The response which I was sending:
{ "requestId": "ff36a3cc-ec34-11e6-b1a0-64510650abcf", "payload": { "devices": { "123": { "online": true, "status": "SUCCESS" } } } }
can anyone help me to solve this issue?
After looking at the code, it is clear that you’re not sending the correct & complete response of the StatusReport trait. Your response contains the online attribute but missing the currentStatusReport attribute (it is required as it defines the current error statuses of the associated device IDs). For more information, visit https://developers.google.com/assistant/smarthome/traits/statusreport?hl=en
So I have a rest API that lives at https://foo.bar/api which returns either an empty json list [] or a list that contains 1 or more items:
[
{
"#class": "foo.bar.java.MyObject",
"name": "Joe Bloggs"
},
{
"#class": "foo.bar.java.MyObject",
"name": "Fred Flinstone"
}
]
Now I am trying to have camel take in this data from my endpoint and hand each object within the list to a processor. I've tried the following:
fromF("timer://foo-poll?fixedRate=true&delay=5s&period=%d&bridgeErrorHandler=true", pollRate)
.toF("https4://%s/%s", host, requestPath)
.log("Received: ${body}")
.split()
.jsonpath("$")
.log("Split: ${body}")
.process(barProccessor);
As well as various attempts to unmarshal the data using .unmarshal(new ListJacksonDataFormat(MyObject.class)) or .unmarshal().json(JsonLibrary.Jackson, List.class)
where nothing has worked.
Using the larger code block above, there are no errors nor is the "Split: ${body}" log message printers out.
Unmarshalling using either methods described above throws this regardless of how many items are returned from the API:
com.fasterxml.jackson.databind.exc.MismatchedInputException: No
content to map due to end-of-input
When there
Okay, managed to figure this out if anyone else is facing a similar issue. The working route builder:
fromF("timer://foo-poll?fixedRate=true&delay=5s&period=%d&bridgeErrorHandler=true", pollRate)
.toF("https4://%s/%s", host, requestPath)
.log("Received: ${body}")
.streamCaching("true")
.unmarshal(new ListJacksonDataFormat(MyObject.class))
.split()
.jsonpath("$")
.log("Split: ${body}")
.process(barProccessor);
I have enabled stream caching and unmarshalled the list using Jackson.