I found Flink's application deployment is opinionated and inflexible. For example:
Some dependencies need to be marked as provided since they are bundled within the Docker image or the Flink's base image.
If I want to change log4j layout type, I need to change Flink client's local file.
Environment variables and k8s secrets need to be passed in via the deployment command's options, which can get very long.
How do I build a Flink application to run on application mode without using Flink as base image? Ideally, an application where all dependencies are managed in 1 place, log configuration is baked into the app and k8s deployment yaml where env vars and secrets can be specified and the base image provides only JDK/JRE.
Related
My react frontend SPA has some environment variables I would like to pass at deploy time using the Static Web App Deploy task in Azure ADO.
If I'm doing this in a pipeline, it's simple to add directly to the YAML https://learn.microsoft.com/en-us/azure/developer/javascript/how-to/with-authentication/static-web-app-with-api/deploy-static-web-app-to-azure#add-react-client-environment-variables-to-workflow-configuration-file
However, within the "Releases", I don't see where I can add these env variables since I can't directly edit the YAML file.
My questions are:
Is this not possible? Is this limitation due to the fact the Deploy Static Web App is technically still in "Preview"?
Is it possible to write a custom Task/YAML code to run during the release? I would prefer to keep the CD in the Releases section so I can keep approvals/rollback functionality, rather than just make a multi-stage pipeline
Is there a better or more correct way to pass these environment variables? They are not secrets, just for example the API URL. Passing as env variables would allow me to break up deployment into dev/prod and pass different values for each environment.
Thank you!
I have created a Flink Native Kubernetes (1.14.2) cluster which is successful. I am trying to update the logback configuration for which I am using the configmap exposed by Flink Native Kubernetes. Flink Native Kubernetes is creating this configmap during the start of the cluster and deleting it when the cluster is stopped and this behavior is as per the official documentation.
I updated the logback configmap which is also successful and this process even updates the actual logback files (conf folder) in the job manager and task manager. But Flink is not loading (hot reloading) this logback configuration.
Also I want to make sure that the logback configmap configuration is persisted even during cluster restarts. But the Flink Native Kubernetes recreates the configmap each time the cluster is started.
What is that I am missing here? How to make the updated logback configuration work?
May be you can refer 'Configuring logback' in https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/advanced/logging/
Solution
To start Flink Native Kubernetes cluster, we will call kubernetes-session.sh. We just need to update the local conf folder before calling kubernetes-session.sh. The following are the files to be updated based on our requirements.
flink-conf.yaml
logger
logback-console.xml (when using logback)
log4j-console.properties (when using log4j)
How does it works
Basically the files from the local conf folder is used to create the configmap in Kubernetes which contains the above mentioned 3 files irrespective of whether you use logback or log4j. The contents from the configmap is then copied to the containers (jobmanager and taskmanager) when the containers are started.
Reference
Thanks to Sharon Xie who gave the solution in the flink mailing list.
To begin with - let us suppose we have a React application. We want to build it and deploy to 3 environments - dev, test and production. As every front-end app it needs to call some APIs. API addresses will vary between the environments. So they should be stored as environment variables.
As every modern, progressive developer we want to use containers. In particular Kubernetes.
We want to build our web application and deploy it on K8S cluster. The container image should be built and kind of sealed for changes, then before deployment to each particular environment the variables should be injected.
But it seems there's one great impossibility here. When it comes to .NET apps for example, when we have .dll compiled, it reads a config file in the runtime. It's not a case with React. After we generate build we have just static files. The variables are changed to static values in the process of building React app. It seems there's no way to update it after that point - or is it?
The way you are trying to solve your problem is not correct.
You don't need to know anything about the addresses of the backend services in your react app. Only the frontend server/gateway that is serving your react app needs to know about the backend services. Any request from the react app should be proxied via the gateway.
See API gateway pattern - https://microservices.io/patterns/apigateway.html
You can use config map to store the API endpoint address and refer it as environment variable in the pod.
If you want to change some values while the pod is running you can mount the config map and any change to it will be synced in the pod
I am using IntelliJ IDE to develop Spring Boot services with Maven and using Google Cloud Tools plugin to deploy to App Engine Flexible. While I use the following (to connect to local) and run the app. locally, it works fine (in application.properties).
spring.datasource.url=jdbc:mysql://localhost:3309/test
However, when I try to deploy to the GAE with the following (in application.properties),
spring.datasource.url=jdbc:mysql://google/test?cloudSqlInstance=[cloud-sql-instance]&socketFactory=com.google.cloud.sql.mysql.SocketFactory
when trying to build the project before uploading to GAE, it throws UnknownHostException: "google".
Questions:
How can I create different configurations for various environments (dev (local) / qa(gae) / production(gae) ) and deploy to those environments with the corresponding environment values?
When doing the build from the IDE, it validates the DB connection string (which points to the cloud sql instance) and throws an exception if it is not reachable (however it will be from the QA / Prod environment if the build is successful). How to resolve this case?
Any help on this would be greatly appreciated.
Thanks in advance.
You need to use Spring Profiles. Please read all the information in the documentation for an extensive explanation.
Briefly:
Spring Profiles provide a way to segregate parts of your application
configuration and make it only available in certain environments
Now, onto the problem at hand. It can be solved by introducing a "local" profile for you development and leaving the "default" profile to be used in production (GAE).
application.properties
# this file is for the "default" profile that will be used when
# no spring.profiles.active is defined. So consider this production config.
spring.datasource.url=jdbc:mysql://google/test?cloudSqlInstance=[cloud-sql-instance]&socketFactory=com.google.cloud.sql.mysql.SocketFactory
application-local.properties
# this file is for the "local" profile that will be used when
# -Dspring.profiles.active=local is specified when running the application.
# So consider this "local" development config
spring.datasource.url=jdbc:mysql://localhost:3309/test
# In this file you can also override any other property defined in application.properties, or add additional ones
Now to run the application while developing all you have to specify in IntelliJ in your run configuration is -Dspring.profiles.active=local under VM options, or if you're using a "Spring Boot" run configuration, you can just add local in the Active Profiles field.
And on GAE, do not specify any profiles at all and the defaults will be used.
Is there a way to update selected files when using the App Engine Flexible env?
I'm facing an issue whenever I do a small change in the app.yaml file: to test it I would need to deploy the whole application which takes ~5mins.
Is there a way to update only the config file? OR is there a way to test these files locally.
Thanks!
The safe/blanket answer would be no as the flex env docker image would need to be updated regardless of how tiny the changes are, see How can I speed up Rails Docker deployments on Google Cloud Platform?
However, there might be something to try (YMMV).
From App Engine Flexible Environment:
You always have root access to Compute Engine VM instances. SSH access to VM instances in the flexible environment is disabled by
default. If you choose, you can enable root access to your app's VM
instances.
So you might be able to login as root on your GAE instance VM and try to manually modify a particular app artifact. Of course, you'd need to locate the artifact first.
Some artifacts might not even be present in the VM image itself (those used exclusively by the GAE infra, queue definitions, for example). But it should be possible to update these artifacts without updating the docker image, since they aren't part of the flex env service itself.
Other artifacts might be read-only and it might not be possible to change them to read-write.
Even if possible, such manual changes would be volatile, they would not survive an instance reload (which would be using the unmodified docker image), which might be required for some changes to take effect.
Lots of "might"s, lots of risks (manual fiddling with the app code could negatively impact its functionality), up to you to determine if a try is really worthy.
Update: it seems this is actually documented and supported, see Accessing Google App Engine Python App code in production