I'm using the AWS Kinesis Data Analytics managed version of Flink to run an IOT service. While ideally I try my best to never restart an application sometimes breaking code changes happen and I have to start the Flink app with allowNonRestoredState or in a worst case scenario I have to clear the state completely and run without state. Is there a parameter that I can check within my Flink application to determine if I have restarted the application without any state?
Why do this at all?
I'm essentially trying to get around the cold start problem I see where in cases of restarting the application without state I notice that the outputs from the Flink app are wrong which I believe is due to the fact the app can only replay the information present in my Kinesis stream. If I knew these records were influenced by the restart then I could mark them invalid before sending them out my app.
Edit:
Checkingpointing is enabled
Using a RocksDB Backend
Related
I'm running a Google App Engine production server, using basic_scaling as the scale type. Whenever I update the code and deploy it - using gcloud app deploy - the old version of the code is shutdown.
According to the documentation, that's expected:
The shutdown process might be triggered by a variety of planned and unplanned events, such as:
You manually stop an instance.
You deploy an updated version to the service.
...
I understand that it's easier for most developers that way. But in my case, I'd like to keep the old versions running until the idle_timeout limit is reached. Does anyone know if there's a way to avoid the automatic shutdown and let the old versions to shutdown by themselves?
Per Google's documentation, when you deploy your code, the default flag of --stop-previous-version is used. This forces the previous version to be stopped. If you do not want that, you should explicitly use the --no-stop-previous-version in your deploy command (we also have this as a feature on our App, a GUI for GAE; you check or uncheck a checkbox).
Unfortunately, Google does not provide a way for the service to automatically shut down later. You'll have to manually shut it down and start the other version later.
I'm trying to create an Electron application using React that will be able to insert and pull data from a local sqlite3 database. I have tried to set up a Web Worker to handle the database interactions but whenever I create an new database connection, the developer tools window disconnects and nothing happens within the app. I think there must be some error happening somewhere that I am unable to see or catch.
Do I have to use a Web Worker or a similar threading/processing tool? If I need to use a Web Worker, what would cause this issue?
I have tried to include the sqlite3 package in one of my React components (without the Web Worker) but that throws all kinds of errors.
Ended up scrapping the web workers as apparently they don't work at all. Swapped over to doing the inserts and queries in the main process and sending the results back to the render process using Electron's build in IPC.
It's perfectly possible to access/use sqlite from the renderer process.
We have a huge complex app communicating with dozens of USB hardware devices via native drivers and everything is running in a single renderer process. To do this we just ensure that no code makes synchronous calls.
I've not used sqlite myself but node-sqlite3 says its fully asynchronous, so you can use it from the Electron renderer without blocking the UI.
After reading from this documentation: https://ci.apache.org/projects/flink/flink-docs-release-1.2/monitoring/rest_api.html, I know that I can send a POST Request to start a Flink job from a savepoint.
The problem is that: This REST interface is hosted in JobManager, which is only alive if there is a job already running there (I run this locally in a JVM). So this is a contradiction because if I want to start a job, there is no job already running right?
Anyone have any clues?
Flink's monitoring API web server and the web dashboard web server are currently the same and run together at the same port. This means that these instructions for how to start up the web UI from inside an IDE should do the job.
Question:
Is there any way to access Firebase using server-side code without using either manual scaling or the flexible environment?
Context:
I want to achieve the following flow:
app posts pending 'updates' to firebase -> backend picks them up -> backend sends emails -> backend modifies firebase 'updates' to non-pending state
From what I can see, if I want the back end to pick these updates up in real time, I need a long-running thread in the App Engine Flexible Environment. I'm prepared to forego this to avoid the flexible environment's pricing model and beta status.
Given that my choice is therefore the App Engine Standard Environment, it appears that to access Firebase i'm stuck with having to enable manual scaling.
It seems madness to have a resident instance running all the time - when there's no requirement to listen to updates in real time - which sits idle for 95% of the time then isn't available outside of a 9-instance-hour (free) contiguous period.
Can Firebase be somehow accessed from a server-side without 'attaching a listener', such that I can call it from an automatically scaled instance and simply get a snapshot? If not, is there an alternate technical or architectural solution here - or something I'm missing?
This must be a fairly common issue!
Thankyou for your time, much appreciated.
I'd like to disable user access to my app so that I can perform a schema migration. I've looked into a few possibilities and found possible shortcomings:
Disable datastore writes - I'd rather just bring my whole application down so that people do not see any errors, etc. Also, I assume disabling writes will prevent me from performing the migration.
Disable the application - It's not clear to me that this would disable it only for my users, leaving me unable to perform the migration. I am also unsure of the disable/enable turnaround time.
Redirect my domain name to a temporary page - my app would still be accessible on appspot.com
Upload a new version of my app that doesn't respond to requests other than to direct to a "temporarily down" page.
Any suggestions?
Suggestion number 4 seems like probably the best way to do this. Some frameworks have a "maintenance mode" in which all incoming requests would be redirected to a page indicating the site is down due to maintenance. If your framework doesn't support such a mode, you can just upload a new version of your app (maybe call the version maintenance) and switch to that as your new default version. This version could be an empty app in which all incoming requests are turned to a "maintenance page" indicating the site is down for maintenance. Then manually go to the version of your app with the migration code and execute it (http://<version>.<appname>.appspot.com). Switch your apps default version to the new version with the new schema when you're done with the migration.
Explanation of your other ideas
Disabling writes would prevent even you from making writes on the application. I believe this was more meant for migrating from one app to another or other applications of "freezing" the datastore.
Disable the application would bring the app down entirely
Redirecting your domain would inflict a DNS lag on your migration, something that can take 48 hours to fully propagate each way (switching to the temporary page, then switching back to the new version)
As aforementioned, IMHO this would be the best way to do it.
Is it not possible to use both schema's, and use a new version of your app to migrate, which only uses the new schema. In this way you can always fall back to your old version.
By the way. Because the datastore is schema-less, It was always possible for me to change the "schema", without bringing the app down.
Go to GCP > App engine > Settings > Disable application.
You can sure enable your application ay any time.
If you disable this app, a few things will happen:
The app will stop serving
All data and state will be saved
Billing charges will still incur when applicable
Your task queues will continue to run. They can be paused in the Cloud Tasks UI