I am developing a website to make my own portfolio. I wanted to switch to google cloud service. I got my app working and a database up. I am able to connect via ip but if a an other instance get's create the ip has to be whitelisted first.
SQLAlchemly url is what i saw was the anwser. But i want to keep working with peewee if possible.
Can someone explain to me why my code is not connect from the app engine to the cloud sql.
My code
"""Main script for the blog website."""
from flask import Flask
from peewee import *
from livereload import Server, shell
from flask_uploads import UploadSet, IMAGES, configure_uploads
from flask_login import LoginManager, login_user
from jinja2 import Environment
import os
from playhouse.db_url import connect
import pymysql
# db = connect(host='127.0.0.1', port=3306, user='root', password='fakepassword')
# db = pymysql.connect(host='127.0.0.1', port=3306, user='root', password='fakepassword')
try:
db = connect('sqlite:///root:fakepassword#/DATABASE?unix_socket=/cloudsql/austinbakkerblog:us-west1:database')
except Exception:
print('did not connect to database')
# 'mysql+mysqldb://%s:%s#/blog?unix_socket=/cloudsql/%s'
app = Flask(__name__)
app.config.from_pyfile('config.py', silent=False)
DEBUG = app.debug
# db = MySQLDatabase('database')
# db.connect()
# db = connect('mysql://root:fakepassword#127.0.0.1:3306/database')
# db = connect('mysql://root:fakepassword#/DATABASE?unix_socket=/cloudsql/austinbakkerblog:us-west1:database')
# db = connect('mysql://root:fakepassword#/cloudsql/austinbakkerblog:us-west1:database')
# db = connect('mysql://root:fakepassword#cloudsql/austinbakkerblog:us-west1:database')
# print(db.connect())
login_manager = LoginManager()
login_manager.init_app(app)
login_manager.login_view = 'login'
photos = UploadSet('photos', IMAGES)
configure_uploads(app, photos)
import models
from views import *
if __name__ == '__main__':
models.initialize()
print('=' * 100)
app.run()
If anyone is willing to help that would amazing, i spendt the last to day's trying to get it work but no success.
So after a few days of grinding i got it to work. Not sure exactly how but it
app.yaml
works. runtime: python
env: flex
entrypoint: gunicorn -b :$PORT app:app
runtime_config:
python_version: 3
env_variables:
SQL_USER: root
SQL_PASSWORD: test
SQL_DATABASE: database
INSTANCE_CONNECTION_NAME: austinbakkerblog:europe-west2:mydatabase1
beta_settings:
cloud_sql_instances: "austinbakkerblog:europe-west2:mydatabase1"
automatic_scaling:
min_num_instances: 1
max_num_instances: 2
cpu_utilization:
target_utilization: 0.5
app.py
"""Main script for the blog's website."""
from flask import Flask
from peewee import *
from livereload import Server, shell
from flask_uploads import UploadSet, IMAGES, configure_uploads
from flask_login import LoginManager, login_user
from jinja2 import Environment
import os
import pymysql
db = MySQLDatabase(host='127.0.0.1', user='root', password='test', unix_socket='/cloudsql/austinbakkerblog:europe-west2:mydatabase1', database='database')
print(db.connect())
print(db.close())
Rather than using a connection URL, it might be easier to debug if you just setup the database connection directly.
For example,
db = MySQLDatabase('db_name', user='root', password='dbpassword',
host='googlecloud.whatever.com', port=31337)
Can you get pymysql to connect?
Related
I have searched for hours trying to find a decent solution to my problem and I can't seem to find a solution. Basically, I broke up my Flask API into two separate files using blueprints, one that serves as the main Flask application and the other to handle all the authentication routes. I am running into CORS issues... Before I separated them, it worked fine with the flask-cors library. However, when I broke it into separate files using blueprints, it didn't work. I did a bunch of research and there seems be a couple ways to do it, but the best way seems to be solving the problem on the server side (at least in theory because I haven't found a solution to it yet).
app.py
# Third-party libraries
from flask import Flask
from flask_cors import CORS
# Internal imports
from login import login_page
# Flask app setup
app = Flask(__name__)
app.register_blueprint(login_page)
CORS(app)
app.config['CORS_HEADERS'] = 'Content-Type'
if __name__ == "__main__":
app.run(host="localhost", debug=True)
login.py
# Third-party libraries
from flask import Flask, redirect, request, url_for, Blueprint, render_template, abort
from flask_login import (
LoginManager,
current_user,
login_required,
login_user,
logout_user
)
from oauthlib.oauth2 import WebApplicationClient
import requests
from flask_cors import CORS
login_page = Blueprint('login_page', __name__)
CORS(login_page)
# Login route
#login_page.route('/login', methods=['GET'])
def login():
# Find out what URL to hit for Google login
google_provider_config = get_google_provider_config()
authorization_endpoint = google_provider_config["authorization_endpoint"]
# Use library to construct the request for Google login and provide
# scopes that let you retrieve user's profile from Google
request_uri = client.prepare_request_uri(
authorization_endpoint,
redirect_uri = request.base_url + "/callback",
scope = ["openid", "email", "profile"]
)
return redirect(request_uri)
From the documentation:
https://flask-cors.readthedocs.io/en/latest/api.html#using-cors-with-blueprints
Which seems to be what I am doing... But it's not working.
Edit:
React code:
authenticate = async () => {
console.log('authenticate called')
const response = await users.get('/login');
}
I added a test route to see if the way I was handling blue prints was incorrect with the flask-cors library as follows:
#login_page.route('/test')
def test():
print('test')
return 'test'
Sure enough, I was able to see the text print on the console. So this leaves me to believe it is something else preventing me from accessing Google's authentication servers.
Flask application is running on localhost:5000 and React application is running on localhost:3000. The string that gets passed into the redirect method in login is as follows:
https://accounts.google.com/o/oauth2/v2/auth?response_type=code&client_id=None&redirect_uri=http%3A%2F%2Flocalhost%3A5000%2Flogin%2Fcallback&scope=openid+email+profile
However, this was working prior to working with React.
Chrome console error:
Access to XMLHttpRequest at '...' (redirected from 'http://localhost:3000/login') from origin 'http://localhost:3000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
I have also added a proxy to my package.json file as follows:
"proxy": "http://localhost:5000"
Which I got from: https://blog.miguelgrinberg.com/post/how-to-create-a-react--flask-project/page/3
Update:
I have decided to go with the approach of solving this problem using Nginx since it seems to that flask-cors library is not suitable for this problem as no solution seems to be possible.
Nginx.conf
user www-data;
worker_processes 1;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;
events {
worker_connections 1024;
}
http {
upstream react-client {
server react-client;
}
upstream flask-backend {
server flask-backend;
}
server {
listen 80;
server_name localhost;
location / {
# From docker-compose, react-client port 80 is exposed
proxy_pass http://react-client;
}
# location /login/callback
# location /logout
}
server {
listen 3000;
server_name localhost;
location /login {
proxy_pass http://flask-backend;
}
}
}
docker-compose.yml:
version: '3'
services:
reverse-proxy:
image: nginx:1.17.10
container_name: reverse-proxy
depends_on:
- flask-backend
- react-client
volumes:
- ./reverse-proxy/nginx.conf:/etc/nginx/nginx.conf
ports:
- 80:80
react-client:
image: react-client
container_name: react-client
build:
context: ./client
depends_on:
- flask-backend
ports:
- 3000:3000
restart: on-failure
flask-backend:
image: flask-backend
container_name: flask-backend
build:
context: ./api
ports:
- 5000:5000
restart: on-failure
With the Nginx configuration and Docker files, I am able to go to http://localhost:80 which forwards me to the react client. This time, I am not getting the CORS errors, but I do get this message in my inspector, Failed to load resource: The network connection was lost.
My understanding is this:
http://localhost:80 -> http://localhost:3000, then when a button is pressed, it prompts the new route
http://localhost:3000/login -> http://localhost:5000/login
I have two server blocks because the web application is running in port 3000 and Nginx server on port 80.
Instead of CORS(login_page) try CORS(login_page, resources=r'/login')
Edit: To explain why this might work.
The error you're getting indicates that the /login route doesn't have CORS permitted. You're instantiating Flask-CORS correctly so I suspect if you tried delineating the specific route you want CORS to apply to you would fix the issue.
The docs make it seem like CORS(login_page) should automatically open up all routes to CORS but because that's not working trying the above answer might provide better results.
I've been trying to get a webapp to work which uses Server-Sent Events. The app that I've written works on my local machine when using Flask's app.run() method. But when I run it on GAE, I've not been able to make it work.
The webapp uses SSE to publish a message with the current time every so often. The client simply adds it to the HTML of a div.
Flask app
import random
from datetime import datetime
from flask import render_template, Response
from time import sleep
from message_server import app
def event_stream():
while True:
time_now = datetime.now()
message = "New message at time: {0}".format(time_now.strftime("%H:%M:%S"))
yield "event: {0}\ndata: {1}\n\n".format("listen", message)
sleep(random.randint(1, 5))
#app.route('/')
def hello():
return render_template('home.html')
#app.route('/stream')
def stream():
return Response(event_stream(), mimetype="text/event-stream")
Javascript in home.html
var source = new EventSource("/stream");
source.onmessage = function(event) {
document.getElementById("messages").innerHTML += event.data + "<br>";
};
source.addEventListener("listen", function(event) {
document.getElementById("messages").innerHTML += event.data + "<br>";
}, false);
GAE app.yaml
runtime: python
env: flex
entrypoint: gunicorn -b :$PORT --worker-class gevent --threads 10 message_server:app
runtime_config:
python_version: 3
manual_scaling:
instances: 1
resources:
cpu: 1
memory_gb: 0.5
disk_size_gb: 10
My directory structure is as following:
app.yaml
/message_server
__init__.py
sse.py
/templates
home.html
message_server is the package that contains the flask app object.
I am using Firefox 67 to test my app.
In the networking tab of Firefox developer console, I see a GET request made to /stream, but no response received even after a minute.
In the GAE logs, I am seeing "GET /stream" 499.
How do I figure out what's wrong?
I found the answer while browsing Google App Engine's documentation - on this page: https://cloud.google.com/appengine/docs/flexible/python/how-requests-are-handled
Essentially, you want the following header in the HTTP response for SSE to work:
X-Accel-Buffering: no
This disables buffering that is enabled by default. I tested it and SSE is working as expected for me.
I am following the docs on how to backup datastore using AppEngine.
I am performing a gcloud app deploy cron.yaml command on a GCE VM that is meant to update a cronjob in AppEngine. the GCE VM and AppEngine cron are in the same project, and I have granted AppEngine admin to the GCE VM via a default Service Account. When I run this on my local machine, it updates fine. However on the GCE instance, thats where issues arise
Here are the files
app.yaml
runtime: python27
api_version: 1
threadsafe: true
service: cloud-datastore-admin
libraries:
- name: webapp2
version: "latest"
handlers:
- url: /cloud-datastore-export
script: cloud_datastore_admin.app
login: admin
cron.yaml
cron:
- description: "Daily Cloud Datastore Export"
url: /cloud-datastore-export?namespace_id=&output_url_prefix=gs://<my-project-id>-bucket
target: cloud-datastore-admin
schedule: every 24 hours
cloud_datastore_export.yaml
import datetime
import httplib
import json
import logging
import webapp2
from google.appengine.api import app_identity
from google.appengine.api import urlfetch
class Export(webapp2.RequestHandler):
def get(self):
access_token, _ = app_identity.get_access_token(
'https://www.googleapis.com/auth/datastore')
app_id = app_identity.get_application_id()
timestamp = datetime.datetime.now().strftime('%Y%m%d-%H%M%S')
output_url_prefix = self.request.get('output_url_prefix')
assert output_url_prefix and output_url_prefix.startswith('gs://')
if '/' not in output_url_prefix[5:]:
# Only a bucket name has been provided - no prefix or trailing slash
output_url_prefix += '/' + timestamp
else:
output_url_prefix += timestamp
entity_filter = {
'kinds': self.request.get_all('kind'),
'namespace_ids': self.request.get_all('namespace_id')
}
request = {
'project_id': app_id,
'output_url_prefix': output_url_prefix,
'entity_filter': entity_filter
}
headers = {
'Content-Type': 'application/json',
'Authorization': 'Bearer ' + access_token
}
url = 'https://datastore.googleapis.com/v1/projects/%s:export' % app_id
try:
result = urlfetch.fetch(
url=url,
payload=json.dumps(request),
method=urlfetch.POST,
deadline=60,
headers=headers)
if result.status_code == httplib.OK:
logging.info(result.content)
elif result.status_code >= 500:
logging.error(result.content)
else:
logging.warning(result.content)
self.response.status_int = result.status_code
except urlfetch.Error:
logging.exception('Failed to initiate export.')
self.response.status_int = httplib.INTERNAL_SERVER_ERROR
app = webapp2.WSGIApplication(
[
('/cloud-datastore-export', Export),
], debug=True)
The Error I'm getting is
Configurations to update:
descriptor: [/usr/local/sbin/pluto/<my-project-id>/datastore/cron.yaml]
type: [cron jobs]
target project: [<my-project-id>]
Do you want to continue (Y/n)?
Updating config [cron]...
failed.
ERROR: (gcloud.app.deploy) Server responded with code [403]:
Forbidden Unexpected HTTP status 403.
You do not have permission to modify this app (app_id=u'e~<my-project-id>').
I have checked other posts related to this, however they seem to deal with an old version/deployment of appengine
Service Accounts!
From Deploying using IAM roles:
To grant a user account the ability to deploy to App Engine:
Click Add member to add the user account to the project and then select all of the roles for that account by using the dropdown menu:
Required roles to allow an account to deploy to App Engine:
a. Set the one of the following roles:
Use the App Engine > App Engine Deployer role to allow the account to deploy a version of an app.
To also allow the dos.yaml or dispatch.yaml files to be deployed with an app, use the App Engine > App Engine Admin role
instead.
The user account now has adequate permission to use the Admin API to deploy apps.
b. To allow use of App Engine tooling to deploy apps, you must also give the user account the Storage > Storage Admin role
so that the tooling has permission to upload to Cloud Storage.
Optional. Give the user account the following roles to grant permission for uploading additional configuration files:
Cloud Scheduler > Cloud Scheduler Admin role: Permissions for uploading cron.yaml files.
Potentially of interest:
Deployments with predefined roles
Predefined roles comparison matrix
Okay after some tinkering. I added the project editor role to the service account linked to the GCE instance running my server. I am not fully aware if this is the role with least priviledge to enable this to work.
I am writing a new application using App Engine and, as the docs suggest to not use Blobstore API, I'm using the Google Cloud Storage client (GCS). All is good but I want to be able to return "signed urls" to clients so they can get the GCS resources without passing through the application. I believe that is what signet urls are for.
But how to test that? I can sucessfully test GCS calls from the client, but I have no idea how to test the client's HTTP calls using urlfetch.
Below is a full test case that illustrates my issue:
import base64
import mimetypes
import urllib
import urllib2
from datetime import datetime, timedelta
import time
from google.appengine.api import app_identity
from google.appengine.datastore import datastore_stub_util
from google.appengine.ext import testbed
from google.appengine.ext import ndb
import unittest
import cloudstorage
# IS THIS RIGHT ?
GCS_API_ACCESS_ENDPOINT = 'http://localhost:8000/_ah/gcs'
def sign_url(bucket_object, expires_after_seconds=60):
""" cloudstorage signed url to download cloudstorage object without login
Docs : https://cloud.google.com/storage/docs/access-control?hl=bg#Signed-URLs
API : https://cloud.google.com/storage/docs/reference-methods?hl=bg#getobject
"""
# source: https://github.com/voscausa/appengine-gcs-signed-url/blob/05b8a93e2777679d40af62cc5ffce933216e6a85/sign_url.py
method = 'GET'
gcs_filename = urllib.quote(bucket_object)
content_md5, content_type = None, None
# expiration : number of seconds since epoch
expiration_dt = datetime.utcnow() + timedelta(seconds=expires_after_seconds)
expiration = int(time.mktime(expiration_dt.timetuple()))
# Generate the string to sign.
signature_string = '\n'.join([
method,
content_md5 or '',
content_type or '',
str(expiration),
gcs_filename])
signature_bytes = app_identity.sign_blob(signature_string)[1]
google_access_id = app_identity.get_service_account_name()
# Set the right query parameters. we use a gae service account for the id
query_params = {'GoogleAccessId': google_access_id,
'Expires': str(expiration),
'Signature': base64.b64encode(signature_bytes)}
# Return the built URL.
result = '{endpoint}{resource}?{querystring}'.format(endpoint=GCS_API_ACCESS_ENDPOINT,
resource=gcs_filename,
querystring=urllib.urlencode(query_params))
return result
FILE_DATA = "This is file contents."
MIME = "text/plain"
class TestGCS(unittest.TestCase):
def setUp(self):
self.testbed = testbed.Testbed()
self.testbed.activate()
self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)
self.testbed.init_datastore_v3_stub(consistency_policy=self.policy)
self.testbed.init_app_identity_stub()
self.testbed.init_memcache_stub()
self.testbed.init_urlfetch_stub()
self.testbed.init_blobstore_stub()
ndb.get_context().clear_cache()
def tearDown(self):
self.testbed.deactivate()
def test_gcs_works(self):
with cloudstorage.open('/mybucket/test.txt', 'w', content_type=MIME) as f:
f.write(FILE_DATA)
with cloudstorage.open('/mybucket/test.txt', 'r') as f:
data = f.read()
print(data)
self.assertEqual(data, FILE_DATA)
def test_signurl(self):
url = sign_url('/mybucket/test.txt')
# FIXME: Not yet working as we have no idea on how to access local GCS during the test.
result = urllib2.urlopen(url)
self.assertEqual(200, result.code)
self.assertEqual(FILE_DATA, result.read())
You can test GCS and service_accounts in your SDK, but you do not have a local appengine GCS service when you use a signed url.
But you can test your local app with service accounts and google cloud services.
Service accounts make it very easy to authorize appengine requests to other Google APIs and services.
To use a service account in the appengine SDK, you have to add two undocumented options when you run the development server:
--appidentity_email_address=<SERVICE_ACCOUNT_EMAIL_ADDRESS>
--appidentity_private_key_path=<PEM_KEY_PATH>
More info in this request for documentation issue
You can create or find the service account in the developers console permissions section of your appengine cloud project.
And you can create and download a p12 key for the service account.
Use OpenSSL to convert this p12 key in a RSA pem key.
I used this OpenSSL installer for Windows.
To create the pem key file in Windows use:
openssl pkcs12 -in <P12_KEY_PATH> -nocerts -nodes -passin pass:notasecret | openssl rsa -out <PEM_KEY_PATH>
Now you can use your cloud app service accounts in the development server and use app_identity to sign and authorize requests.
I'm trying to use the App Engine datastore in my application outside of the bounds of one of the request handler pages (like main.py or the other files you can specify in app.yaml). Is this possible? When I try to run it, it says my database model does not have a method "put" associated with it, but it has no problems when I run it from a script connected to CGI and request handling (like main.py or any other .py files I declare similarly in app.yaml). Do you know what I can do to add CGI to those pages without adding request handling? Thanks!
Edit my code:
users.py file:
from google.appengine.ext import db, webapp
class User(db.Model):
email = db.EmailProperty()
password = db.StringProperty()
main.py file
from user import *
class CreateHandler(webapp.RequestHandler):
def get(self):
u = User()
u.email = "email#email.com"
u.password = "mypass"
u.put()
It gives me this error:
File "........./main.py", line 75, in get
u.put()
AttributeError: User instance has no attribute 'put'
Yes, you can access the datastore from other scripts. You don't have to add request handling to them, that can stay in your main script. Eg., you can do something like this:
app.yaml:
- url: /.*
script: main.py
main.py:
from SectionHandlers import * # This imports classes from SectionHandlers.py
application = webapp.WSGIApplication([
("/section1/.*", Section1Handler), # Map requests to handlers
("/section2/.*", Section2Handler),
], debug=True)
SectionHandlers.py:
from google.appengine.ext import db, webapp
class Section1Handler(BlogHandler):
def get(self):
# Code using 'db' here