I recently enabled AppEngine's Datastore Admin. I do not understand the instructions on how to copy my data to another app.
Note: The target application must
enable remote_api and must include
this application’s ID in its
HTTP_X_APPENGINE_INBOUND_APPID list.
WARNING This application’s data is
writable. We can only guarantee a
consistent copy when the data being
copied is read-only. Note: Blobs
(binary data) will not be copied.
To enable the remote_api I included the following in the app.yaml:
builtins:
- remote_api: on
I have no idea what HTTP_X_APPENGINE_INBOUND_APPID is, and a Google search yields no results....any ideas?
The datastore copy feature is currently available only for Python applications.
If you are using GAE for Java you must do the following steps:
Notes: Assuming you are copying from app example1.appspot.com to app example2.appspot.com and your Java WEB-INF app folder is located in /Users/admin/src/main/webapp/WEB-INF
Download Python SDK
https://developers.google.com/appengine/downloads#Google_App_Engine_SDK_for_Python
Run the Python SDK
Create a new file inside your WEB-INF folder: app.yaml
Create a new file inside your WEB-INF folder: appengine_config.py
Deploy your new version (run in terminal): appcfg.py -A example2 update **/Users/admin/src/main/webapp/WEB-INF
Go to your example1 datastore admin view and put: datastore-admin.example2.appspot.com/_ah/remote_api
app.yaml:
application: example2
version: datastore-admin
runtime: python
api_version: 1
builtins:
- remote_api: on
- datastore_admin: on
appengine_config.py:
remoteapi_CUSTOM_ENVIRONMENT_AUTHENTICATION = (
'HTTP_X_APPENGINE_INBOUND_APPID', ['example1'])
If you are using java and already tried this and its not working(i had the same problem) i added:
<servlet>
<display-name>Remote API Servlet</display-name>
<servlet-name>RemoteApiServlet</servlet-name>
<servletclass>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>RemoteApiServlet</servlet-name>
<url-pattern>/remote_api</url-pattern>
</servlet-mapping>
in web.xml and update with this command: appcfg.py -A appid update /yourappfolder/war/WEB-INF
Strange but after i did that it worked.
If you are using GAE for Java you must do the steps in this answer.
GAE now use Python 2.7. So you need to create your app.yaml as below.
application: example2
version: datastore-admin
runtime: python27
api_version: 1
threadsafe: true
builtins:
- remote_api: on
The datastore_admin tool should enable through the admin console instead of enabling the builtin in your app.yaml.
Related
I have created a Cloud Scheduler job with target to App Engine HTTP. The target URL is /admin/task/create-documents
I have an App Engine flexible running with java 8.
It works fine, but now I would like to secure the access to the servlet called by the Cloud Scheduler job (/admin/task/create-documents) to GCP developers only (usually referenced as 'admin'). I assumed the Cloud Scheduler job is considered as 'admin'.
Option 1 - didn't work
I tried to modify the web.xml file as for the standard environment, as mentioned here, but with no success:
The web.xml file:
<?xml version="1.0" encoding="utf-8"?>
<web-app xmlns="http://xmlns.jcp.org/xml/ns/javaee"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://xmlns.jcp.org/xml/ns/javaee
http://xmlns.jcp.org/xml/ns/javaee/web-app_3_1.xsd"
version="3.1">
<welcome-file-list>
<welcome-file>index.html</welcome-file>
</welcome-file-list>
<security-constraint>
<web-resource-collection>
<web-resource-name>admin</web-resource-name>
<url-pattern>/admin/*</url-pattern>
</web-resource-collection>
<auth-constraint>
<role-name>admin</role-name>
</auth-constraint>
</security-constraint>
</web-app>
Option 2 - didn't work
I tried to modify the app.yaml by adding login: admin to the handlers section, as mentioned in the documentation here, but with no success either.
The app.yaml file:
runtime: java
env: flex
threadsafe: true
runtime_config:
jdk: openjdk8
server: jetty9
handlers:
- url: /admin/.*
script: auto
login: admin
- url: /.*
script: auto
secure: always
env_variables:
JETTY_ARGS: -Djava.util.logging.config.file=WEB-INF/logging.properties
network:
instance_tag: no-ip
name: my-network
subnetwork_name: my-subnet
But I also noticed that the login parameter is deprecated, as mentioned here and here. They mention IAM policies but I'm not sure how I should configure them.
How do I secure the Cloud Scheduler job endpoint to 'admin' only?
You can't by configuration. You need to implement the check in your code.
You can add a static value to the URL like https://my-url.appspot.com/admin/task/create-documents?key=my_secret or use OIDC authentication with Cloud Scheduler to provide a JWT token to your endpoint.
In both case, you will have to check if the secret (content on the JWT) is the expected one or not.
I am trying to deploy an asp.net core webapi to google appengine flex. If you don't mention network name in app.yaml then it takes default.
In my case there is no default network. I have to explicitly specify the network name and subnetwork_name in my app.yaml. This project where I am trying to deploy my api in appengine flex is in a shared VPC.This is why I am specifying the complete path, "projects/myorg-npe-232f/global/networks/myorg-shared-network". Please refer the app.yaml below -
runtime: aspnetcore
env: flex
service: default
api_version: 1.1
network:
name: projects/myorg-npe-232f/global/networks/myorg-shared-network
subnetwork_name: myorg-exxc-bbdf-subnet-central
I am getting the below error -
Unable to assign value 'projects/myorg-npe-232f/global/networks/myorg-shared-network' to attribute 'name':
value 'projects/myorg-npe-232f/global/networks/myorg-shared-network' for name does not match expression '^(?:^[a-z]([a-z\d-]{0,61}[a-z\d])?$)$' in app.yaml
As explained in the documentation here:
In a service project, App Engine Flexible resources cannot participate
in Shared VPC.
You may want to consider VPC Peering instead if it fits your use case.
Once the firewall rule and proper permissions are set up, can be deployed either a new or an existing of App Engine flexible ENV service into Shared VPC network, check the parameters in the manifest file:
network:
name: projects/HOST_PROJECT_ID/global/networks/NETWORK_NAME
subnetwork_name: SUBNETWORK_NAME
If you are using AppEngine flex, you can use the same settings.
However, make sure the vpc subnet exists, permissions and the firewall rule setup, I was getting the exact same error when I did not provide subnetwork_name.
Re: https://cloud.google.com/appengine/docs/flexible/dotnet/using-shared-vpc
network:
name: projects/project-id/global/networks/myorg-shared-vpc
session_affinity: true
subnetwork_name: mycompany-develop-us-central1
Another update:
Found this same issue with a co-worker, fixed it by updating the cloud commandline client.
I am running into this error while deploying my app to google app-engine.
Error:
Error 404: --- begin server output ---
This application does not exist (project_id=u'homework-153002'). To create an App Engine application in this project, run "gcloud beta app create" in your console.
--- end server output ---
When I try to create the app using the above command this is what I see:
ERROR: (gcloud.beta.app.create) You do not have permission to access app [homework] (or it may not exist): Operation not allowed
I have created a project in google developers api website with project_name of "homework" and random id. Here is the url for that:
https://console.developers.google.com/apis/library?project=homework-153002
Any pointers on what I might be doing wrong here?
This is my yaml file:
application: homework-153002
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: .*
script: main.app
libraries:
- name: webapp2
version: "2.5.2"
Your app URL indicates your app ID is homework-153002, but the
error message indicates you're connecting to an ap called homework - likely not your own, hence the permission issue.
Simply correcting the app ID in your deployment command and/or your app.yaml file should suffice.
As your comment indicates, for your case (gcloud deployment) the solution was:
gcloud config set project homework-153002
gcloud beta app create
appcfg.py update homework/
I have seen posts in Google Groups and here that mention using xlrd in Google App Engine:
How do you read excel files with xlrd on Appengine
https://groups.google.com/forum/?fromgroups=#!searchin/google-appengine-python/xlrd/google-appengine-python/lMix6vXhvtA/O_ExzkGhsKEJ
but when I try to add it to the app.yaml file, it says "the library xlrd is not supported". Also, in the list of 3rd party libraries I don't see xlrd.
https://developers.google.com/appengine/docs/python/tools/libraries27
But then other people seem to be using it on GAE, so does GAE support it or not? In the past I was able to import numpy successfully. Here is app.yaml. On main.py I have "import xlrd". Thank you very much.
application: uploadsample
version: 1
runtime: python27
api_version: 1
threadsafe: yes
handlers:
- url: /favicon\.ico
static_files: favicon.ico
upload: favicon\.ico
- url: .*
script: main.app
libraries:
- name: webapp2
version: "2.5.1"
- name: xlrd
version: "0.7.3"
You can include any pure python third-party libraries with your app that you want. You don't specify them in app.yaml, that's only for third-party libraries that are included by Google on the servers.
xlrd is not "supported" per se, but it should work.
In the answer to the question Error sending e-mail via SMTP server on App Engine development server there is a nice solution by Blixt:
"dev_appserver.py does not support TLS
which is required by Gmail. You can
enable it by adding a few lines in api/mail_stub.py:"
# After smtp.connect(self._smtp_host, self._smtp_port)
smtp.ehlo()
smtp.starttls()
smtp.ehlo()
But windows Vista would not let me change api/mail_stub.py
Is there another way to send mail from development server. Other options (Sendmail and ISP are not good for me). Thanks!
EDIT
I changed the api/mail_stub.py according to instructions here and I use the following command-line options:
dev_appserver.py
--smtp_host=smtp.gmail.com
--smtp_port=25
--smtp_user=xxxx#gmail.com
--smtp_password=gmail_pw
C:\Users\A\Desktop\repeater # path to root directory
But I get this error from Log Console:
***********************************************************
2010-11-18 10:24:37 Running command: "['C:\\Python26\\pythonw.exe',
'C:\\Program Files(x86)\\Google\\google_appengine\\dev_appserver.py',
'--admin_console_server=',
'--port=8080',
u'dev_appserver.py',
u'--smtp_host=smtp.gmail.com',
u'--smtp_port=25',
u'--smtp_user=xxxx#gmail.com',
u'--smtp_password=gmail_pw
C:\\Users\\A\\Desktop\\repeater',
'C:\\Users\\A\\Desktop\\repeater']"
Runs a development application server for an application.
dev_appserver.py [options] <application root>
Application root must be the path to the application to run in this server.
Must contain a valid app.yaml or app.yml file.
****************************************************
This is the app.yaml:
application: re-peater
version: 1
runtime: python
api_version: 1
handlers:
- url: /favicon.ico
static_files: static/images/favicon.ico
upload: static/images/favicon.ico
- url: /stylesheets
static_dir: stylesheets
- url: /.*
script: repeater.py
All this works without these command line options. Any suggestions why this is not working?
If you really, really need to send real email from the dev_appserver, you should set up your own mail relay on your machine, and point the SDK at that. I'm curious why it's so important to send real email, though - this is the development server, and you shouldn't be using it for anything other than development.
Change the file permissions so that you can change the api/mail_stub.py file.
This is no longer necessary
in /appengine/api/mail_stub.py
if self._allow_tls and smtp.has_extn ('STARTTLS'):
smtp.starttls ()
I am using appengine sdk version 1.9.15.