I am using the java library to subscribe to a subscription from my code. Using sbt:
"com.google.cloud" % "google-cloud-pubsub" % "0.24.0-beta"
I followed this guide to write a subscriber: https://cloud.google.com/pubsub/docs/pull
val projectId = "test-topic"
val subscriptionId = "test-sub"
def main(args: Array[String]): Unit = {
val subscriptionName = SubscriptionName.create(projectId, subscriptionId)
val subscriber = Subscriber.defaultBuilder(subscriptionName, new PastEventMessageReceiver()).build()
subscriber.startAsync()
System.in.read()
}
class PastEventMessageReceiver extends MessageReceiver {
override def receiveMessage(message: PubsubMessage, consumer: AckReplyConsumer): Unit = {
println(message)
consumer.ack()
}
It works great, I am able to pull published messaged but I keep seeing this error in my log multiple times per minute.
com.google.cloud.pubsub.v1.StreamingSubscriberConnection$1 onFailure
WARNING: Terminated streaming with exception
io.grpc.StatusRuntimeException: UNAVAILABLE: The service was unable to fulfill your request. Please try again. [code=8a75]
at io.grpc.Status.asRuntimeException(Status.java:526)
at io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onClose(ClientCalls.java:385)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:422)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:61)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:504)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:425)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:536)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:102)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
When I first run the app there is a small delay (around 1-2 mins) where I don't see that error, after the delay I see it multiple times per minute. My application seems to still able to pull messages tough.
This message is an internal error in the Google Cloud Pub/Sub library that happens when there is a disconnection or retryable error in a request sent to the Pub/Sub server. The client library should seamlessly recreate connections and retry requests on these errors. In version 0.26.0-beta and later of the client library, these errors should no longer print out in the log unless you have the log level set to FINE. In general, messages should still continue to be sent to your MessageReceiver after this error occurs. Any error that is not retryable by the client library itself, e.g., when a subscription is not found, is propagated back to the caller.
Related
I've some flink jobs which uses kafka as source and sink and I want to add tracing to it, so that any message consumed/produced from/to Kafka is well traced, for that I'm using kafka interceptors to intercepts messages and log trace, span and parent traceId, for that I'm using
opentracing-kafka-client(v0.1.11) in conjunction with brave-opentracing(v0.35.1), the reason why I'm using custom interceptors because I need to log messages in a specified format.
After configuring interceptors they are getting invoked and it uses tracing information (from headers) coming from upstream system and logs it but when it comes to producing message again to kafka then tracing context is lost for instance consider below scenario
1) Message put on Kafka by some rest service
2) Message consumed by flink job and interceptors kicks in and uses tracing information from header and logs it
3) After processing message is produced by flink job to Kafka
It works well until step #2 but when it comes to producing message then tracing information from previous step is not used because it does not have any headers information and hence it produces entirely new trace.
I'm registering tracer as below :-
public class MyTracer {
private static final Tracer INSTANCE = BraveTracer.create(Tracing.newBuilder().build());
public static void registerTracer() {
GlobalTracer.registerIfAbsent(INSTANCE);
}
public static Tracer getTracer() {
return INSTANCE;
}
}
And I'm using TracingConsumerInterceptor and TracingProducerInterceptor from opentracing kafka.
I am trying to use the alpakka kinesis connector to send messages to a Kinesis Stream but I have no success with it. I tried the code below but nothing in my stream.
implicit val sys = ActorSystem()
implicit val mat = ActorMaterializer()
implicit val kinesisAsync: AmazonKinesisAsync = AmazonKinesisAsyncClientBuilder.defaultClient()
val debug = Flow[PutRecordsRequestEntry].map { reqEntry =>
println(reqEntry)
reqEntry
}
val entry = new PutRecordsRequestEntry()
.withData(ByteBuffer.wrap("Hello World".getBytes))
.withPartitionKey(Random.nextInt.toString)
Source.tick(1.second, 1.second, entry).to(KinesisSink("myStreamName", KinesisFlowSettings.defaultInstance)).run()
// 2) Source.tick(1.second, 1.second,entry).via(debug).to(KinesisSink("myStreamName", inesisFlowSettings.defaultInstance)).run()
Using a Sink.foreach(println) instead of KinesisSink prints out the PutRecordsRequestEntry every 1 second => EXPECTED
Using KinesisSink, the entry is generated only once.
What Am I doing wrong ?
I am checking my stream with a KinesisSource and reading is working ( tested with another stream)
Also the monitoring dashboard of AWS Kinesis doesnt show any PUT requests.
Note 1: I tried to enable the debug log of alpakka but with no effect
<logger name="akka.stream.alpakka.kinesis" level="DEBUG"/>
in my logback.xml + debug on root level
Some troubleshooting steps to consider below - I hope they help.
I suspect you're likely missing credentials and/or region configuration for your Kinesis client.
Kinesis Firehose
The Kinesis Producer Library (what Alpakka seems to be using) does not work with Kinesis Firehose. If you're trying to write to Firehose this isn't going to work.
Application Logging
You'll probably want to enable logging for the Kinesis Producer Library, not just in Alpakka itself. Relevant documentation is available here:
Configuring the Kinesis Producer Library
Configuration Defaults for Kinesis Producer Library
AWS Side Logging
AWS CloudTrail is automatically enabled out of the box for Kinesis streams, and by default AWS will keep 90 days of CloudTrail logs for you.
https://docs.aws.amazon.com/streams/latest/dev/logging-using-cloudtrail.html
You can use the CloudTrail logs to see the API calls your application is making to Kinesis on your behalf. There's usually a modest delay in requests showing up - but this will let you know if the request is failing due to insufficient IAM permissions or some other issue with your AWS resource configuration.
Check SDK Authentication
The Kinesis client will be using the DefaultAWSCredentialsProviderChain credentials provider to make requests to AWS.
You'll need to make sure you are providing valid AWS credentials with IAM rights to make those requests to Kinesis. If your code is running on AWS, the preferred way of giving your application credentials is using IAM Roles (specified at instance launch time).
You'll also need to specify the AWS Region when building the client in your code. Use your application.properties for configuring this, or if your application is part of a CloudFormation stack that lives in a single region - using the instance metadata service to retrieve the current region when your code is running on AWS.
The problem was an access denied / permission on the action on the stream.
I had to add the akka actor config for logging
akka {
loggers = ["akka.event.slf4j.Slf4jLogger"]
loglevel = "DEBUG"
stdout-loglevel = "DEBUG"
logging-filter = "akka.event.slf4j.Slf4jLoggingFilter"
logger-startup-timeout = "30s"
}
to see debug lines and I actually run in debug and step in each stage.
It required permission "PutRecords" in the IAM role
I am using the Watson Java SDK from github (https://github.com/watson-developer-cloud/java-sdk) to run a test of the speech-to-text service. I am running an example from my workplace, which has a firewall and an http proxy set up.
The test fails because the remote server cannot be reached. The stack trace is as follows:
Exception in thread "main" java.lang.RuntimeException: java.net.UnknownHostException: stream.watsonplatform.net
at com.ibm.watson.developer_cloud.service.WatsonService$1.execute(WatsonService.java:182)
at com.costaisa.app.api.test.ibm.watson.TestIBMWatson.testSpeechToText(TestIBMWatson.java:76)
at com.costaisa.app.api.test.ibm.watson.TestIBMWatson.main(TestIBMWatson.java:37)
Caused by: java.net.UnknownHostException: stream.watsonplatform.net
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at okhttp3.Dns$1.lookup(Dns.java:39)
at okhttp3.internal.http.RouteSelector.resetNextInetSocketAddress(RouteSelector.java:173)
at okhttp3.internal.http.RouteSelector.nextProxy(RouteSelector.java:139)
at okhttp3.internal.http.RouteSelector.next(RouteSelector.java:81)
at okhttp3.internal.http.StreamAllocation.findConnection(StreamAllocation.java:172)
at okhttp3.internal.http.StreamAllocation.findHealthyConnection(StreamAllocation.java:123)
at okhttp3.internal.http.StreamAllocation.newStream(StreamAllocation.java:93)
at okhttp3.internal.http.HttpEngine.connect(HttpEngine.java:296)
at okhttp3.internal.http.HttpEngine.sendRequest(HttpEngine.java:248)
at okhttp3.RealCall.getResponse(RealCall.java:243)
at okhttp3.RealCall$ApplicationInterceptorChain.proceed(RealCall.java:201)
at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:163)
at okhttp3.RealCall.execute(RealCall.java:57)
at com.ibm.watson.developer_cloud.service.WatsonService$1.execute(WatsonService.java:179)
... 2 more
It seems to me that okhttp3 supports HTTP proxies but not through environment variables or system properties.
Am I missing something? Is it at all possible to invoke the Watson speech-to-text service from Java through a proxy?
EDIT
I'm no longer working with Watson (not even the same job!), but it became apparent that what I wanted to do was not possible with the then current implementation. A comment to this question (not an answer) provided the actual answer: you cannot do it without changing the implementation. I'd vote that as an answer, but alas it's only a comment. I hope this is useful for other people looking for an answer.
Juanjo this is what you can do to pass the request through corporate proxy.
Create a class that extends from TextToSpeech (as ExtendedTextToSpeech that I wrote for me.).
Then change the DemoServlet that comes with the watson sample as below
TextToSpeech textService = new com.ibm.cloudoe.samples.ExtendedTextToSpeech();
textService.setUsernameAndPassword("username", "password");
String voice = req.getParameter("voice");
String text = req.getParameter("text");
String format = "audio/ogg; codecs=opus";
in = textService.synthesize(text, new Voice(voice, null, null), format);
If you get javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target, then
login to stream.watsonplatform.net/text-to-speech/api/v1 with your username/password received from your watson account's text-to-speech service
Click on the lock sign in the browser's address bar and export the certificate.
Follow the steps here http://magicmonster.com/kb/prg/java/ssl/pkix_path_building_failed.html
Two questions:
Does Google App Engine send any kind of message to an application just before relocating it to another server?
If so, what is that message?
No it doesnt. It doesnt relocate either, old instances keep running (and eventually stop when idle for long enough) while new ones are spawned.
There are times when App Engine needs to move your instance to a different machine to improve load distribution.
When App Engine needs to turn down a manual scaling instance it first
notifies the instance. There are two ways to receive this
notification. First, the is_shutting_down() method from
google.appengine.api.runtime begins returning true. Second, if you
have registered a shutdown hook, it will be called. It's a good idea
to register a shutdown hook in your start request. After the
notification is issued, existing requests are given 30 seconds to
complete, and new requests immediately return 404.
If an instance is
handling a request, App Engine pauses the request and runs the
shutdown hook. If there is no active request, App Engine sends an
/_ah/stop request, which runs the shutdown hook. The /_ah/stop request
bypasses normal handling logic and cannot be handled by user code; its
sole purpose is to invoke the shutdown hook. If you raise an exception
in your shutdown hook while handling another request, it will bubble
up into the request, where you can catch it.
The following code sample demonstrates a basic shutdown hook:
from google.appengine.api import apiproxy_stub_map
from google.appengine.api import runtime
def my_shutdown_hook():
apiproxy_stub_map.apiproxy.CancelApiCalls()
save_state()
# May want to raise an exception
runtime.set_shutdown_hook(my_shutdown_hook)
Alternatively, the following sample demonstrates how to use the is_shutting_down() method:
while more_work_to_do and not runtime.is_shutting_down():
do_some_work()
save_state()
More details here: https://developers.google.com/appengine/docs/python/modules/#Python_Instance_states
For some reason, I was under the impression that it was just called Timeout, but it doesn't seem to be.
Thanks!
For datastore calls, the exception is google.appengine.ext.db.Timeout. For total (wall clock) duration exceeded, the exception is google.appengine.runtime.DeadlineExceededError. The DeadlineExceeded error is thrown 'soft' once, at which point you have a short time to return a response and exit; if you don't, it's thrown again, uncatchable, and your script is unceremoniously terminated.
The timeout exception handling is explained in the Request Timer section of the docs:
A request handler has a limited amount of time to generate and return a response to a request, typically around 30 seconds. Once the deadline has been reached, the request handler is interrupted.
The Python runtime environment interrupts the request handler by raising a DeadlineExceededError, from the package google.appengine.runtime. If the request handler does not catch this exception, as with all uncaught exceptions, the runtime environment will return an HTTP 500 server error to the client.
The request handler can catch this error to customize the response. The runtime environment gives the request handler a little bit more time (less than a second) after raising the exception to prepare a custom response.
from google.appengine.runtime import DeadlineExceededError
class MainPage(webapp.RequestHandler):
def get(self):
try:
# Do stuff...
except DeadlineExceededError:
self.response.clear()
self.response.set_status(500)
self.response.out.write("This operation could not be completed in time...")
If the handler hasn't returned a response or raised an exception by the second deadline, the handler is terminated and a default error response is returned.
While a request can take as long as 30 seconds to respond, App Engine is optimized for applications with short-lived requests, typically those that take a few hundred milliseconds. An efficient app responds quickly for the majority of requests. An app that doesn't will not scale well with App Engine's infrastructure.
The DataStore has its own TimeOut exception
The google.appengine.ext.db package provides the following exception classes:
[...]
exception Timeout()
Raised when the datastore operation exceeds the maximum amount of time allowed for datastore operations.