I have a Python script stored in a Compute Engine instance. I also have a web application deployed on the Google App Engine.
What I would like to achieve is let users enter some parameters on the web application interface and have it execute the script in the Compute Engine instance with the entered parameters.
My question is: how can I access the Compute Engine instance from App Engine and execute the script with the parameters that users passed in?
I think there are several factors here to take into account:
Security implications: Having a web site backend accessing a different host to run a command with client-defined parameters upon a request can easily introduce lots of potential exploits that aren't worth dealing with.
Sanitizing the parameter or parameters from the Python script would be a must, which you might be able to do this with shlex.quote().
Running the script in the VM instance via SSH from App Engine:
With the Google Cloud Client Library for Python you may be able to connect to a given GCE instance and run a command by setting up OS Login and granting roles/compute.osLogin for this instance to the Service Account that runs your App Engine app as described in this guide (with this example).
Otherwise, you may try creating a system account for this purpose inside the instance and allow its login in its /etc/ssh/ssh_config and use a new RSA key added to this user's ~/ssh/.authorized_keys file with a generic SSH client library like Paramiko connecting to its external IP address, assuming it has one.
In both cases this is going to introduce very high latencies for the requests as SSH sessions usually take 2+ seconds to get created.
As an acceptable-latency (and potentially more secure) alternative, you might be able to have a simple HTTPS service in the VM (you can probably check for a correct snakeoil certificate in your Python code if needed) and set up a webhook with a long hash-like URL path (and optionally a non-default port) handled by, for instance, a simple PHP script that runs the end script with exec() after passing the parameter variable in its $_POST[] superglobal through excapeshellarg() to (re-)sanitize it.
Related
For using the google pub sub we need key.json file which serves as service account.
For deployed code we can pass the service_account: name#service-name.iam.gserviceaccount.com in app.yaml which works if everything is in google cloud.
As it is not recommend to download the service account file.
Is there a way where we can just pass the key name in local environment file along with individual credential who has access to service account in run time or any other way where we can run google cloud services in local enviornment without downloading the service key file?
Google makes this really hard to find, but gcloud auth application-default login might do what you need.
From that page:
Obtains user access credentials via a web flow and puts them in the
well-known location for Application Default Credentials (ADC).
This command is useful when you are developing code that would
normally use a service account but need to run the code in a local
development environment where it's easier to provide user credentials.
The credentials will apply to all API calls that make use of the
Application Default Credentials client library. Do not set the
GOOGLE_APPLICATION_CREDENTIALS environment variable if you want to use
the credentials generated by this command in your local development.
This command tries to find a quota project from gcloud's context and
write it to ADC so that Google client libraries can use it for billing
and quota. Alternatively, you can use the --client-id-file flag. In
this case, the project owning the client ID will be used for billing
and quota. You can create the client ID file at
https://console.cloud.google.com/apis/credentials.
I know cloud run and appengine are different services.
I need connect via ssh to an appengine or cloud run instance to execute some process manually.
The reason to use one of these services is they charge only when I use it, not 24x7 hours
Some way to do that?
Thanks
Short answer: you can't.
In fact, these services are designed to answer to HTTP request, and only when an HTTP request is processed you pay for the service. If you log into an instance in SSH, will you pay for the HTTP request? If you run a process on the instances, will you pay for the HTTP request?
Of course not. But the cost isn't the main reason. Cloud Run and App Engine can create and destroy instances as they wish, according with the traffic or something else. It's useless to log into an instance and to run a process and few seconds/minutes after the instance is deleted and a new one created, you will lost all what you do.
If you use these services, you must accept that the servers are managed by Google, that you can only deploy a service and use it through HTTP. It's not a traditional VM instance, it's "serverless".
After saying that, if you want to explore the runtime configuration, you can use a HTTP reverse shell. But, at the end, it's not very useful...
Context
I code using codeanywhere, because I had multiple places with desktop computers to work and don't want to load a laptop
Actually I had vps's as enviroments, like my projects are long time, don't need to rebuild or change the enviroment in years
The need:
I run some times per month shell commands like test nodejs scripts, before to move them to serverless (cloud run)
The old-approach:
try to run these scripts on a working enviroment connecting via ssh
The moderm developer way:
use codeanywhere containers as code storage and testing + create a gitlab ci/cd to deploy automatically on google cloud run instances
Our AppEngine app is connecting to a remote service which requires a VPN and also required me to add entries to the hosts file on my local machine in order to connect to their endpoints.
e.g.
10.200.30.150 foo.bar.com
This is working fine when running the app locally, but I can't figure out how to set this up on Google Cloud to work once deployed.
I can't use the IP addresses directly because it errors that the IP is not on the cert's list.
How do I map the host names to the IPs in Google Cloud so that AppEngine can use them?
From the error mentioned in the comment I suspect connecting directly through the IP fails because the certificate doesn't recognize the IP to DNS mapping as valid and therefore the secure connection setup breaks. Based on the requirements of connecting to the API by VPN and tweaking the hosts mapping there are few things you may try.
The simplest approach that may work would be using a Google Compute Engine VM instance, since there you would able to manipulate the etc/hosts file and replicate the local machine setup. This VM could be used either as the main app service or as a proxy from App Engine to the 3rd party API endpoint. To go that route I would suggest taking a look at these two posts which explain how to change the etc/hosts file on GCE (Changing the file once wouldn't work as the VM periodically overrides it, see the posts for cronjob like workaround).
Separately, as your app runs in App Engine flexible environment there is the chance to provide a docker container with the app packaged. It may be possible to set the workaround above in the docker file and have it working in App Engine too.
In Google App Engine, I have 3 services, 1 for front end, 2 for back end.
Is there a way to block http calls to my backend services for accounts not from my company's domain (and the service account of the front end), but allow everyone http access to my front end service?
I know there is the firewall option, but this is restricted to IP addresses, I would prefer user based
If it matters all services are python3
There's currently no option to filter traffic to specific App Engine services within a single application/project:
App Engine Firewall filters by source IP ranges but can only be set for the whole app, not per service.
Identity-Aware Proxy can filter access by user account as you'd prefer but also applies to the whole app. Also, it only supports user account and can't be used with service accounts.
One option you may have would be to split your app in 2 different projects. Keep the front-end in one project open to the world and restrict access to the backend services in your other project via firewall rules.
I have seen the following being used in task queues in GAE. Maybe it would help.
If u were using python 2, in standard environment, i think u could have used login handler element in app.yaml file.
You could have added following lines to your app.yaml file:
handlers:
- url: /.*
script: worker.app
login: admin
This prevents other users from accessing this service.
But the same login handler is not available for python3, according to Google Docs.
Just found following in Google Docs:
If a task performs sensitive operations (such as modifying data), you might want to secure the handler URL to prevent a malicious external user from calling it directly. You can prevent users from accessing task URLs by restricting access to App Engine administrators. Task requests themselves are issued by App Engine and can always target a restricted URL.
You can restrict a URL by adding the login: admin element to the handler configuration in your app.yaml file.
You can also call your backend services through cloud tasks or task queues (both are almost the same i guess), in case this only work for cloud tasks.
Find the code usage here:
https://github.com/GoogleCloudPlatform/python-docs-samples/tree/6f5f3bcb81779679a24e0964a6c57c0c7deabfac/appengine/standard/taskqueue/counter
Find details about handler here.
https://cloud.google.com/appengine/docs/standard/python/config/appref#handlers_element
Find details about Cloud task and queue here:
https://cloud.google.com/appengine/docs/standard/python/taskqueue/push/creating-handlers
I would like to debug my Google App Engine (GAE) app locally but without using localhost. Since my application is made up of microservices, the urls in a production environment would be along the lines of:
https://my-service.myapp.appspot.com/
But code in one service can call another service and that means that the urls are hardcoded. I could of course use a mechanism in code to determine whether the app is running locally or on GAE and use urls that are different although I don't see how a local url would handle the since the only way to run an app locally is to use localhost. Hence:
http://localhost:8080/some-service
Notice that "some-service" maps to a servlet, whereas "my-service" is a name assigned to a service when the app is uploaded. These are really two different things.
The only possible solution I was able to find was to use a reverse proxy which would map one url to a different one. Still, it isn't clear whether the GAE development SDK even supports this.
Personally I chose to detect the local development vs GAE environment and build my inter-services URLs accordingly. I feel it was a well-worthy effort, I've been (re)using it a lot. No reverse proxy or any other additional ops necessary, it just works.
Granted, I'm using Python, so I'm not 100% sure a complete similar Java solution exists. But maybe it can point you in the right direction.
To build the per-service URLs I used modules.get_hostname() (the implementation is presented in Resolve Discovery path on App Engine Module). I believe the Java equivalent would be getInstanceHostname() from com.google.appengine.api.modules.
This method, when executed on the local server, automatically provides the particular port the server listens to for each service.
BTW, all my services for an app are executed by a single development server process, which listens on multiple ports (this is, I guess, how it can provide the modules.get_hostname() info). See Running multiple services using dev_appserver.py on different ports. This is part I'm unsure about: if/how the java local dev server can simultaneously run multiple services. Apparently this used to be supported some time ago (when services were still called modules):
Serving multiple GAE modules from one development server?
GAE modules on development server
This can be accomplished with the following steps:
Create an entry in the hosts file
Run the App Engine Dev server from a Terminal using certain options
Use IntelliJ with Remote debugging to attach the App Engine Dev server.
To edit the hosts file on a Mac, edit the file /etc/hosts and supply the domain that corresponds to your service:. Example:
127.0.0.1 my-service.myapp.com
After you save this, you need to restart your computer for the changes to take place.
Run the App Engine Dev server manually:
dev_appserver.sh --address=0.0.0.0 --jvm_flag=-Xdebug
--jvm_flag=-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=8000
[path_to_exploded_war_directory]
In IntelliJ, create a debug configuration. Use the Remote template to create this configuration. Set the host to the url you set in the hosts file and set the port to 8000.
You can set a breakpoint and run the app in IntelliJ. IntelliJ will attach to the running instance of App Engine Dev server.
Because you are using a port during debugging and no port is actually used when the app is uploaded to the GAE during production, you need to add code that identifies when the app is running locally and when it's running on GAE. This can be done as follows:
private String mServiceUrl = "my-service.my-app.appspot.com";
...
if (SystemProperty.environment.value() != SystemProperty.Environment.Value.Production) {
mServiceUrl += ":8000";
}
See https://cloud.google.com/appengine/docs/standard/java/tools/using-local-server
An improved solution is to avoid including the port altogether and not having to use code to determine whether your app is running locally or on the production server. One way to do this is to use Charles (an application for monitoring and interacting with requests) and use a feature called Remote Mapping which lets you map one url to another. When enabled, you could map something like:
https://my-service.my-app.appspot.com/
to
https://localhost:8080
You would then enable the option to include the original host, so that this gets delivered to the local dev server. As far as your code is concerned it only sees:
https://my-service.my-app.appspot.com/
although the ip address will be 127.0.0.1:8080 when remote mapping is enabled. To use https on local host however does require that you enable ssl certificates for Charles.
For a complete overview on how to setup and debug microservices for a GAE Java app in IntelliJ, see:
https://github.com/JohannBlake/gae-microservices