I have two modules/services and each one has a cron.xml. Only one of these ever seems to run (the most recently deployed in my experience), and the other doesn't fail, but the endpoint is never triggered (never shows up in the logs).
Is there a 1-cron per project limit? What is the best way to manage crons so that modules aren't cross-dependant?
The cron.yaml is an app-level config file, not a service/module one. Meaning when you deploy the one you have in one module it'll overwrite the cron config from the other one.
So you have to create a single cron.yaml file containing job configs for all services/modules. As #GAEfan mentioned you'll also need add target configs for each job. You might also need to add a dispatch.yaml file and maybe re-visit/adjust the request paths so that the cron job-issued requests make it to the right service/module.
Deploying the app-level cron.yaml might not be happening implicitly when deploying the service(s), you may need to deploy it explicitly. From
Uploading cron jobs:
Option 2: Upload only your cron updates
To update just the cron configuration without uploading the rest of
the application, run the following command:
appcfg.py update_cron <app-directory>
Some more or less related Q&As:
Deploying different languages services to the same Application [Google App Engine]
Can a default service/module in a Google App Engine app be a sibling of a non-default one in terms of folder structure?
Use the target: backend-module-name parameter inside a cron job you want to send to a module other than default. Only one cron.yaml needed.
Make sure you update: appcfg.py update app.yaml backend_module.yaml cron.yaml
Related
I have a Quarkus application already deployed on Google Cloud Run.
It depends on MySQL, hence there is an instance started on Cloud SQL.
Next step in my deployment process is to add keycloak. From what I've read the best option seems to be Google App Engine.
The approved answer in this question gave me some good insight of what needs to be done ... mostly.
What I did was:
Locally I made a sub-directory in the main project.
In that directory I added the app.yaml and the Dockerfile (as described here for instance).
There I executed the said two commands: gcloud init and gcloud app deploy.
I had my doubts about this set up and they were backed up by the error I got eventually:
ERROR: (gcloud.app.deploy) INVALID_ARGUMENT: The first service (module) you upload to a new application must be the 'default' service (module). Please upload a version of the 'default' service (module) before uploading a version for the 'morph-keycloak-service' service (module).
I understand my set up breaks the overall structure of the project but I'm not sure how to mix those two application with the right services.
I understand keycloak is a stateful application, hence cannot live on Cloud Run (by the way the intention is for keycloak to use the same database instance shared with the application).
So does any one know a more sensible set up, or what can I move in mine in order to fix it?
In short:
The answer really is in reading the error message (thanks #gaefan) - about the error itself it explains enough. So I just commented out the service: my-keycloak-service line in the app.yaml (thus leaving gcloud to implicitly mark it as the default one) and the deployment continued.
Eventually keycloak didn't connect to the database but if I don't manage to adjust the configurations that would probably be a subject to a different question.
On the point of project structure and functionality:
First off, thanks #NoCommandLine and #guillaume-blaquiere for your input!
#NoCommandLine the application on Cloud Run is sort of a headless REST API enabled backend. Most of the API calls are secured by keycloack. A next step in the deployment process would be to port an existing UI (React) client on the Firebase hosting (or on another suitable service - I'm still not completely sure which approach is best) and in order for the users to work with this client properly they must make an SSO through keycloak first.
I'm quite new to GCP and the number and variants of the available options are still overwhelming to me - one must get familiar with the nuances but I guess it takes time. So I'm still taking suggestions on how to adjust my project structure to fit better the services stack. Thanks!
I am trying to use task queues in GAE Java 8, but somehow it does not seem to deploy correctly via the queue.xml file. I can also not see the task queues in the Cloud Tasks console (which is where I get redirected from the app engine console).
I get an error java.lang.IllegalStateException: The specified queue is unknown : xxxxx when running the app.
The app runs fine locally. I can see the task queues appearing locally in the admin page.
Does this mean that I cannot deploy task queues via queue.xml anymore?
You should be aware that the queue configuration is not a per-version config (or even a per-service one!), it is a global, per-application config. Or per-project if you want - considering that there can only be one GAE application per GCP project.
This single queue configuration is shared by all versions of all services of your application, so:
if/when services/versions need different queue configs all of them need to be merged into a single file for deployment.
pay attention at deployment not to overwrite/negatively impact existing services/versions
While in some cases the queue.xml file might be deployed automatically when you deploy your application code it is not always the case. The official recommended deployment method is using the deployment command dedicated for the queue configuration, which can be performed independently from deploying application/service code. From Deploying the queue configuration file:
To deploy the queue configuration file without otherwise altering the
currently serving version, use the command:
appcfg.sh update_queues <application directory>
replacing <application directory> with the path to your application
main directory.
Pay extra attention if you have:
other non-java standard environment services in your app - they use the queue.yaml queue configuration file, managing the same content in 2 different files/formats can be tricky
other services managing queue via using Cloud Tasks. See Using Queue Management versus queue.yaml.
I have backuped datastore via cron using cron.yaml like following
- description: My Daily Backup
url: /_ah/datastore_admin/backup.create?name=BackupToCloud&kind=LogTitle&kind=EventLog&filesystem=gs&gs_bucket_name=whitsend
schedule: every 12 hours
target: ah-builtin-python-bundle
But
According to google announcement, datastore-admin will go to "deprecated".
https://cloud.google.com/datastore/docs/console/datastore-backing-up-restoring
How to backup datastore via cron without datastore_admin?
https://cloud.google.com/appengine/articles/scheduled_backups
says only about using gcloud.
Note that just the backup/restore functionality based on the datastore-admin will be deprecated, not the datastore-admin itself.
The deprecation note points to the Managed export and import service as the recommended replacement alternative.
Exports based on this method can also be scheduled, see Scheduling an Export. You'll note in that article that a standard env GAE app with a cron service is exactly what the method is based on.
The article is targeted at those apps using the Datastore outside of GAE. Since you already have a GAE app you can just modify your existing backup cron job handler following the example in the article or, if you want to separate it a bit from your main app, you can add a separate service to your app, dedicated to the backup cron job.
I am new to Google App Engine, I am tring to create a cron job but I can't find where to start.
I create the App Engine => Task Queues => Cron Jobs and now I don't see an option to create one. Probably I missunderstand something very obvious, please help me to figure out what.
You need to create a new file - cron.yaml, describe your cron job in it, and upload it with the rest of the application as described in the documentation on Cron jobs for flexible environment.
Inside your applications directory, where your app.yaml is, you can create a file called cron.yaml to specify your cron jobs.
I am writing a wrapper around the Github Issues API to allow managers in my company to set up daily reminder emails to be sent to their devs. I want this to be configurable through an admin console, and give them the flexibility of setting up reminders at any time of day and any number of times a day.
The main App Engine cron system is configured statically through the cron.yaml file and cannot be changed by user action. Looking at the documentation it appears like I can only do this by reimplementing an entire cron infrastructure on top of the basic App Engine cron. Am I missing something? Is there anything like this that is already available elsewhere?
You are correct, you cannot setup programmatically the cron configuration.
You can configure a single cron which triggers a customized functions. This functions can read the configured crons (like datastore entities) and the launch different task based on your needs