After reading from this documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/rest_api.html, I know that I can send a POST Request to start a Flink job from a savepoint.
The problem is that: This REST interface is hosted in JobManager, which is only alive if there is a job already running there (I run this locally in a JVM). So this is a contradiction because if I want to start a job, there is no job already running right?
Anyone have any clues?
Flink's monitoring API web server and the web dashboard web server are currently the same and run together at the same port. This means that these instructions for how to start up the web UI from inside an IDE should do the job.
Related
I am trying to follow the flink cdc example according to https://github.com/ververica/flink-sql-CDC . I have followed it all the way up to the end and can see the data getting to elastic search and kibana can see the index being created. But I cannot get the web ui on the flink job manager to work? I just end up with localhost didn’t send any data. . The REST API to the same port works just fine
Any ideas on what may have gone wrong?
I've just checked it, but for me the Web UI does come up on port 8081. Are you sure you've tried to access http://localhost:8081/ ? If you still can't access it, it's most likely that you have another application already running on that port or your firewall is preventing you from accessing it.
I know cloud run and appengine are different services.
I need connect via ssh to an appengine or cloud run instance to execute some process manually.
The reason to use one of these services is they charge only when I use it, not 24x7 hours
Some way to do that?
Thanks
Short answer: you can't.
In fact, these services are designed to answer to HTTP request, and only when an HTTP request is processed you pay for the service. If you log into an instance in SSH, will you pay for the HTTP request? If you run a process on the instances, will you pay for the HTTP request?
Of course not. But the cost isn't the main reason. Cloud Run and App Engine can create and destroy instances as they wish, according with the traffic or something else. It's useless to log into an instance and to run a process and few seconds/minutes after the instance is deleted and a new one created, you will lost all what you do.
If you use these services, you must accept that the servers are managed by Google, that you can only deploy a service and use it through HTTP. It's not a traditional VM instance, it's "serverless".
After saying that, if you want to explore the runtime configuration, you can use a HTTP reverse shell. But, at the end, it's not very useful...
Context
I code using codeanywhere, because I had multiple places with desktop computers to work and don't want to load a laptop
Actually I had vps's as enviroments, like my projects are long time, don't need to rebuild or change the enviroment in years
The need:
I run some times per month shell commands like test nodejs scripts, before to move them to serverless (cloud run)
The old-approach:
try to run these scripts on a working enviroment connecting via ssh
The moderm developer way:
use codeanywhere containers as code storage and testing + create a gitlab ci/cd to deploy automatically on google cloud run instances
I'm using the AWS Kinesis Data Analytics managed version of Flink to run an IOT service. While ideally I try my best to never restart an application sometimes breaking code changes happen and I have to start the Flink app with allowNonRestoredState or in a worst case scenario I have to clear the state completely and run without state. Is there a parameter that I can check within my Flink application to determine if I have restarted the application without any state?
Why do this at all?
I'm essentially trying to get around the cold start problem I see where in cases of restarting the application without state I notice that the outputs from the Flink app are wrong which I believe is due to the fact the app can only replay the information present in my Kinesis stream. If I knew these records were influenced by the restart then I could mark them invalid before sending them out my app.
Edit:
Checkingpointing is enabled
Using a RocksDB Backend
I am trying to create a web application with Apache Flink as a backend. Here flink need to talk to application layer(generally UI), so that the results from flink can be sent to frontend.
In Short i am looking for flink counterparts of projects like:
Spark Job Server https://github.com/spark-jobserver/spark-jobserver
Mist https://github.com/Hydrospheredata/mist
The (deceptively titled) Monitoring Rest API detailed here seems like the way to go. Even though it says "Monitoring" it actually can upload, start, and stop jobs. In fact, based on the documentation it seems like it should do everything the web UI does.
I have and MVC5 web-app running on Azure. Primarily this is a website but I also have a CRON job running (triggered from an external source) which calls a URL (a GET Request) to carry out some house keeping.
This task is asynchronous can take up to and sometimes over the default timeout on Azure of 230 seconds. I'm regularly getting 500 errors due to this.
Now, I've done a bit of reading and this looks like it's something to do with the Azure Load Balancer settings. I've also found some threads relating to being able to alter that timeout in various contexts but I'm wondering if anyone has experience in altering the Azure Load Balancer timeout in the context of a Web App? Is it a straightforward process?
but I'm wondering if anyone has experience in altering the Azure Load Balancer timeout in the context of a Web App?
You could configure the Azure Load Balancer settings in virtual machines and cloud services.
So, if you want to do it, I suggest you could deploy web app to Virtual Machine or migrate web app to cloud services.
For more detail, you could refer to the link.
If possible, you could try to optimize your query or execution code.
This is a little bit of an old question but I ran into it while trying to figure out what was going on with my app so I figured I would post an answer here in case anyone else does the same.
An Azure Load Balancer (created via ANY means) which mostly comes along with an external IP Address — at least when created via Kubernetes / AKS / Helm — has the 4 min, 240 second idle connection timeout that you referred to.
That being said there are two different types of Public IP Addresses — Basic (which is almost always the default that you probably created) and Standard. More information on the docs: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-standard-overview
You can modify the idle timeout see the following doc: https://learn.microsoft.com/en-us/azure/load-balancer/load-balancer-tcp-idle-timeout
That being said changing the timeout may or may not solve your problem. In our case we had a Rails app that was connection to a database outside of azure. As soon as you add a Load Balancer with a Public IP all traffic will EXIT that public IP and be bound by the 4 minute idle timeout.
That's fine except for that Rails anticipates that a connection to a Database is not going to be cut frequently — which in this case happens ALL the time.
We ended up implementing a connection pooling service that sat between our Rails app and our real database (called PGbouncer — specific to Postgres DBs). That service monitored a connection and re-connected when the timer was nearing the Azure LB timeout.
It took a little while to implement but in our case it works flawlessly. You can see some more details over here: What Azure Kubernetes (AKS) 'Time-out' happens to disconnect connections in/out of a Pod in my Cluster?
The longest timeout you can set for a Public IP / Load Balancer is 30 minutes. If you have a connection that you would like to utilize that runs idle longer than that — then you may be out of luck. As of now 30 mins is the max.