The code below successfully establishes a websocket connection.
The websockets server (also akk-http) deliberately closes the connection using Andrew's suggested answer here.
The SinkActor below receives a message of type akka.actor.Status.Failure so I know that the flow of messages from Server to Client has been disrupted.
My question is ... How should my client reestablish the websocket connection? Has source.via(webSocketFlow).to(sink).run() completed?
What is best practice for cleaning up the resources and retrying the websocket connection?
class ConnectionAdminActor extends Actor with ActorLogging {
implicit val system: ActorSystem = context.system
implicit val flowMaterializer = ActorMaterializer()
private val sinkActor = context.system.actorOf(Props[SinkActor], name = "SinkActor")
private val sink = Sink.actorRefWithAck[Message](sinkActor, StartupWithActor(self.path), Ack, Complete)
private val source = Source.actorRef[TextMessage](10, OverflowStrategy.dropHead).mapMaterializedValue {
ref => {
self ! StartupWithActor(ref.path)
ref
}
}
private val webSocketFlow: Flow[Message, Message, Future[WebSocketUpgradeResponse]] =
Http().webSocketClientFlow(WebSocketRequest("ws://localhost:8080"))
source
.via(webSocketFlow)
.to(sink)
.run()
Try the recoverWithRetries combinator (docs here).
This allows you to provide an alternative Source your pipeline will switch to, in case the upstream has failed. In the most simple case, you can just re-use the same Source, which should issue a new connection.
val wsSource = source via webSocketFlow
wsSource
.recoverWithRetries(attempts = -1, {case e: Throwable => wsSource})
.to(sink)
Note that
the attempts = -1 will retry to reconnect indefinetely
the partial function allows for more granular control over which exception can trigger a reconnect
Related
I have Kinesis Analytics application running with Flink and use external Flink Sink object to transfer data with Apache HttpAsyncClients. The below code is working fine for some time.
override def open(): Unit = {
uuid = randomUUID.toString
client = HttpAsyncClients
.custom()
.setKeepAliveStrategy(
DefaultConnectionKeepAliveStrategy.INSTANCE
)
.build()
client.start()
}
override def invoke(event: String): Unit = {
log.info(s"Received Value to sink: ${event.length} Head: ${event.take(20)}")
val request = createRequest(event)
client.execute(
request,
new FutureCallback[SimpleHttpResponse]() {
override def completed(response: SimpleHttpResponse): Unit = {
val status = StatusLine(response)
if (status.isError) {
log.error(s"Request: $request Status: $status Body: ${response.getBodyText}")
}
}
override def failed(ex: Exception): Unit = {
log.error(s"Request: $request Ex: $ex")
}
override def cancelled(): Unit = {
log.error(s"Request: $request Cancelled")
}
}
)
}
private def createRequest(event: String): SimpleHttpRequest = {
SimpleRequestBuilder
.post()
.setHttpHost(new HttpHost(URIScheme.HTTPS.id, host))
.setPath(path)
.setBody(event, ContentType.APPLICATION_JSON)
.build()
}
override def close(): Unit = {
client.close(CloseMode.GRACEFUL)
}
}
I have around 80k requests every minute and this app works fine for several days, before starting to raise DeadlineTimeoutException. Initially I thought it's related to the throughput, which my server can accept, but I don't see application errors on the backend. Instead, all requests just fail after some time.
I don't really know what is happening, because if socket connection is not closed properly and there is actual timeout I should see some requests dropping off. Instead, at the event of failure I stop to receive any traffic and zero requests successful.
I believe something has to be added for the configuration of the HTTP client, and sockets should not clog. From the documentation it seems like it shouldn't happen, socket connection and leased connection should be terminated. But at this point I have no idea what has to be changed.
I'm trying to migrate a personal project from Akka Classic to Akka Typed and I'm having some troubles with messageAdapters.
Here below the code to reproduce the behavior that is causing me some troubles:
object Example extends App {
val system = ActorSystem(MyActor(), "system")
system ! MyActor.Msg("Foo") // 3 chars lenght
system ! MyActor.Msg("FooBarOsdAsd") // 12 chars lenght
object MyActor {
sealed trait Command
case class Msg(payload: String) extends Command
case class Question(payload: String, replyTo: ActorRef[Response]) extends Command
case class WrappedResponseOne(response: Response) extends Command
case class WrappedResponseTwo(response: Response) extends Command
sealed trait Response
case object Yes extends Response
case object No extends Response
def apply(): Behavior[Command] =
Behaviors.setup { context =>
new MyActor(context).stateless()
}
}
// I need to have a private class that implement a method to expose the wanted behavior
private class MyActor(context: ActorContext[MyActor.Command]) {
// This is the adapter I want to use
private val wrappedOne: ActorRef[MyActor.Response] =
context.messageAdapter[MyActor.Response](MyActor.WrappedResponseOne)
// This variable is unused
private val wrappedTwo: ActorRef[MyActor.Response] =
context.messageAdapter[MyActor.Response](MyActor.WrappedResponseTwo)
def stateless(): Behaviors.Receive[MyActor.Command] = Behaviors.receiveMessage[MyActor.Command] {
case MyActor.Msg(payload) =>
context.log.debug(s"$payload")
context.self ! MyActor.Question(payload, wrappedOne)
Behaviors.same
case MyActor.Question(payload, replyTo) =>
if (payload.length > 10) replyTo ! MyActor.No else replyTo ! MyActor.Yes
Behaviors.same
case MyActor.WrappedResponseOne(response) =>
response match {
case MyActor.Yes =>
context.log.info("YES")
Behaviors.same
case MyActor.No =>
context.log.info("NO")
Behaviors.same
}
case msg =>
context.log.debug(s"$msg")
Behaviors.unhandled
}
}
}
Here I'm defining two messageAdapters: wrappedOne and wrappedTwo. When the actor got a MyActor.Msg I want that the responses will be handled by wrappedOne (leaving wrappedTwo completely unused). I can't get why my code let wrappedTwo manage the responses. Here below is the output from the console.
SLF4J: See also http://www.slf4j.org/codes.html#substituteLogger
22:57:38.879 [system-akka.actor.default-dispatcher-3] DEBUG exlude.ActorAdapter$MyActor - Foo
22:57:38.881 [system-akka.actor.default-dispatcher-3] DEBUG exlude.ActorAdapter$MyActor - FooBarOsdAsd
22:57:38.890 [system-akka.actor.default-dispatcher-3] DEBUG exlude.ActorAdapter$MyActor - WrappedResponseTwo(Yes)
22:57:38.891 [system-akka.actor.default-dispatcher-3] DEBUG exlude.ActorAdapter$MyActor - WrappedResponseTwo(No)
22:57:38.904 [system-akka.actor.default-dispatcher-3] INFO akka.actor.LocalActorRef - Message [exlude.ActorAdapter$MyActor$WrappedResponseTwo] to Actor[akka://system/user] was unhandled. [1] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
22:57:38.907 [system-akka.actor.default-dispatcher-3] INFO akka.actor.LocalActorRef - Message [exlude.ActorAdapter$MyActor$WrappedResponseTwo] to Actor[akka://system/user] was unhandled. [2] dead letters encountered. This logging can be turned off or adjusted with configuration settings 'akka.log-dead-letters' and 'akka.log-dead-letters-during-shutdown'.
Is there something wrong with how I'm defining the adapters?
That behavior is by design: only one message adapter for a given incoming message type can be registered at a time. This prevents unbounded growth in the number of adapters if (e.g.) the adapter is being registered in response to messages. The last adapter wins.
See docs and note that if A is a subclass of B, adapters for A and B can both be registered, but since the adapters are tested last-registered-first, if the B adapter is registered after the A adapter, it will shadow the A adapter.
For situations where there's at-most-one response per request, context.ask may be a better fit.
If you really need multiple adaptations of the same message types, spawning a child actor for each adaptation is probably the way to go; having to do this may be a sign that your messaging protocols could use a rethink.
I am looking for an explanation for the behavior I see with the following code. When the
conn.handleWith is commented out, the TCP client connection that I make with netcat, connects, and in a couple of seconds shows disconnected by peer (i.e. the server disconnected the connection). When the conn.handleWith is present in the code I see no disconnection. I initially though it had to do with the idletimeout set up for the server, but that wasn't the case.
So why does the server disconnect the client when there is no flow to handle the connection?
package com.example;
import java.util.concurrent.CompletionStage;
import akka.Done;
import akka.NotUsed;
import akka.actor.typed.ActorSystem;
import akka.actor.typed.javadsl.Behaviors;
import akka.stream.javadsl.Sink;
import akka.stream.javadsl.Source;
import akka.stream.javadsl.Tcp;
import akka.stream.javadsl.Tcp.IncomingConnection;
import akka.stream.javadsl.Tcp.ServerBinding;
public class SimpleStream00 {
public static void main(String[] args) throws InterruptedException {
ActorSystem actorSystem = ActorSystem.create(Behaviors.empty(), "actorSystem");
final Sink<IncomingConnection, CompletionStage<Done>> handler = Sink.foreach(conn -> {
System.out.println("Client connected from: " + conn.remoteAddress());
// conn.handleWith(Flow.of(ByteString.class), actorSystem);
// Server does not drop the connection when previous line is uncommented
});
Source<IncomingConnection, CompletionStage<ServerBinding>> source = Tcp.get(actorSystem).bind("127.0.0.1",
8888); // .idleTimeout(Duration.ofSeconds(60));
CompletionStage<ServerBinding> bindingFuture = source.to(handler).run(actorSystem);
bindingFuture.handle((binding, throwable) -> {
if (binding != null) {
System.out.println("Server started, listening on: " + binding.localAddress());
} else {
System.err.println("Server could not bind to : " + throwable.getMessage());
actorSystem.terminate();
}
return NotUsed.getInstance();
});
}
}
A general principle in Akka Streams is that if there's no demand, the stream should consume as few resources as possible. Since without handleWith, your stream never signals demand for the ByteStrings from the connection, Akka's TCP layer disconnects the connection to save resources.
The last few weeks we have been experiencing this error message while using the Azure Search SDK (1.1.1 - 1.1.2) and performing searches.
We consume the Search SDK from internal APIs (deployed as Azure Web Apps) that scale up-down based on traffic (so there could be more than 1 instance of the APIs doing the searches).
Our API queries 5 different indexes and maintains an in-memory copy of the SearchIndexClient object that corresponds to each index, a very simple implementation would look like:
public class AzureSearchService
{
private readonly SearchServiceClient _serviceClient;
private Dictionary<string, SearchIndexClient> _clientDictionary;
public AzureSearchService()
{
_serviceClient = new SearchServiceClient("myservicename", new SearchCredentials("myservicekey"));
_clientDictionary = new Dictionary<string, SearchIndexClient>();
}
public SearchIndexClient GetClient(string indexName)
{
try
{
if (!_clientDictionary.ContainsKey(indexName))
{
_clientDictionary.Add(indexName, _serviceClient.Indexes.GetClient(indexName));
}
return _clientDictionary[indexName];
}
catch
{
return null;
}
}
public async Task<SearchResults> SearchIndex(SearchIndexClient client, string text)
{
var parameters = new SearchParameters();
parameters.Top = 10;
parameters.IncludeTotalResultCount = true;
var response = await client.Documents.SearchWithHttpMessagesAsync(text, parameters, null, null);
return response.Body;
}
}
And the API would invoke the service by:
public class SearchController : ApiController
{
private readonly AzureSearchService service;
public SearchController()
{
service = new AzureSearchService();
}
public async Task<HttpResponseMessage> Post(string indexName, [FromBody] string text)
{
var indexClient = service.GetClient(indexName);
var results = await service.SearchIndex(indexClient, text);
return Request.CreateResponse(HttpStatusCode.OK, results, Configuration.Formatters.JsonFormatter);
}
}
We are using SearchWithHttpMessagesAsync due to a requirement to receive custom HTTP headers instead of the SearchAsync method.
This way we avoid opening/closing the client under traffic bursts. Before using this memory cache (and wrapping each client on a using clause) we would get port exhaustion alerts on Azure App Services.
Is this a good pattern? Could we be receiving this error because of the multiple instances running in parallel?
In case it is needed, the stack trace shows:
System.Net.Http.HttpRequestException: Only one usage of each socket address (protocol/network address/port) is normally permitted service.ip.address.hidden:443
[SocketException:Only one usage of each socket address (protocol/network address/port)is normally permitted service.ip.address.hidden:443]
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,Socket s4,Socket s6,Socket& socket,IPAddress& address,ConnectSocketState state,IAsyncResult asyncResult,Exception& exception)
[WebException:Unable to connect to the remote server]
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult,TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
EDIT: We are also receiving this error A connection attempt failed because the connected party did not properly respond after a period of time:
System.Net.Http.HttpRequestException: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond service.ip.address.hidden:443
[SocketException:A connection attempt failed because the connected party did not properly respond after a period of time,or established connection failed because connected host has failed to respond service.ip.address.hidden:443]
at System.Net.Sockets.Socket.EndConnect(IAsyncResult asyncResult)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure,Socket s4,Socket s6,Socket& socket,IPAddress& address,ConnectSocketState state,IAsyncResult asyncResult,Exception& exception)
[WebException:Unable to connect to the remote server]
at System.Net.HttpWebRequest.EndGetRequestStream(IAsyncResult asyncResult,TransportContext& context)
at System.Net.Http.HttpClientHandler.GetRequestStreamCallback(IAsyncResult ar)
As implemented in the code in your question, the cache will not prevent port exhaustion. This is because you're instantiating it as a field of the ApiController, which is created once per request. If you want to avoid port exhaustion, the cache must be shared across all requests. To make it concurrency-safe, you should use something like ConcurrentDictionary instead of Dictionary.
The "connection attempt failed" error is likely unrelated.
I have a policy file server up and running. For a while I was getting the AccessDenied because the policy file was not set properly. Now I no longer receive that error, so I know that's not the issue. I have a simple server running that simple loops on accepting client connections from any address. I also wrote a simple client, so I know the server works. In Silverlight I set my args and then call ConnectAsync. It return immedately on localhost (makes sense) and when I check the event args LastOperation is Connect and SocketError is Success. However, when I check my socket, it is not connected at all. Any ideas..? Been banging my head against a wall for hours over this.
A few other things I've tried. I moved the servers off my local box onto another server. Still didn't work. I did a packet capture and noticed that it is receiving the Poilcy File, but after that, there is no packet sent out by the browser to even attempt to connect to the other server.
public void Connect(string ip)
{
SocketAsyncEventArgs saea = new SocketAsyncEventArgs();
Socket socket = new Socket(AddressFamily.InterNetwork, SocketType.Stream, ProtocolType.Tcp);
var endpoint = new IPEndPoint(IPAddress.Parse(ip), 4502);
saea.UserToken = socket;
saea.RemoteEndPoint = endpoint;
saea.Completed += new EventHandler<SocketAsyncEventArgs>(AsyncEventComplete);
var completedSync = socket.ConnectAsync(saea);
if (completedSync)
{
AsyncEventComplete(null, saea);
}
Result = ip;
}
void AsyncEventComplete(object sender, SocketAsyncEventArgs e)
{
switch (e.LastOperation)
{
case SocketAsyncOperation.Connect:
MessageBox.Show("CONNECTED");
break;
case SocketAsyncOperation.Receive:
MessageBox.Show("DATA RECEIEVED");
// do stuff
break;
}
}
I think you should use e.SocketError and not e.LastOperation
You could also use e.ConnectSocket (in Silverlight only)
You should also add a "not" in this condition : if ( ! completedSync )