I am building a web crawler app on Google App Engine. I want to use a post method to pass the variable to Flask. Then, the received variable became the input of my web crawler app. However, Flask only accept single variable from post. If I add another variable in the funciton, Flask would crash.
I have limited knowledge in Flask and Google app engine. I struggled with the problem for several days and your help will be highly appreciated.
Failed function
#server-side function that does not work,with 2 variable passed
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
url = request_json["url"]
#baseconnect.Baseconnect(url=url,filename=filename).run()
return filename,url
#The function to post on client side
import requests
req = requests.Session()
data = req.post('https://project.appspot.com/bac',json={"filename":"yuan","url":"https:...f5"})
print(data.text)
#output:
Internal server eror 500
Succeeded function
#server-side function that works,with 1 variable passed
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
#url = request_json["url"]
#baseconnect.Baseconnect(url=url,filename=filename).run()
return filename
#The function to post on client side
import requests
req = requests.Session()
data = req.post('https://project.appspot.com/bac',json={"filename":"yuan"})
print(data.text)
#output:
yuan
Flask seems only accept single variable. What is the problem....
The problem you have here is that Flask only returns Response object, and Flask will consider return filename, url a shortcut of return Response, status or header.
In this case, url becomes the http status code or header, which is obviously not right.
You need flask.jsonify() to return the proper format of so called 'multiple variables'.
Something like this: (only the important part)
# In server-side code
from flask import jsonify
#app.route('/bac',methods=['GET', 'POST'])
def bac():
request_json = request.get_json()
filename = request_json["filename"]
url = request_json["url"]
# Do your logic here
return jsonify({ filename_returned: filename, url_returned: url })
# client-side
import requests
req = requests.Session()
json_data = req.post('https://project.appspot.com/bac',json={"filename":"yuan", "url": "http:xxxxxxx"})
real_data = json.loads(json_data)
# real_data should be the result you want
Related
I'm trying to create a function that let me select a file from input on the frontend, then pass the file to the backend FastAPI, and eventually upload the file to Azure Blob storage. My Frontend code is below:
//front end
async function handleSubmit(){
const formdata = new FormData();
formdata.append(
"file",
file[0],
)
const headers={'Content-Type': file[0].type}
await axios.post("/uploadfile",formdata,headers)
.then(function (response) {
console.log(response)
});
}
Backend FastAPI - two methods I tried
//Backend FastAPI
#app.post("/uploadfile") //currently using
async def create_upload_file(file: UploadFile):
name = file.filename
type = file.content_type
return uploadtoazure(file,name,type)
#app.post("/files") //another method I tried
async def create_file(file: bytes= File()):
await uploadtoazure(file)
And the uploadtoazure() function
//Backend uploadtoazure() function
async def uploadtoazure(f,n,t):
connect_str = ""//removed from sample code
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "notes"
file = f.read()
local_file_name = n
cnt_settings = ContentSettings(content_type=t)
blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)
blob_client.upload_blob(file,cnt_settings)
The error kept coming at me and when I tried another method another new error prevents me from moving forward.
Some problem I'm aware:
blob_client.upload_blob() only accept some types of files, in which the file type I passed in from API isn't one of it.
I used the sample code in https://fastapi.tiangolo.com/tutorial/request-files/In which I am not quite sure of the characteristic of the class UploadFile and File().
When I use the method which the file passed to API is file: bytes= File(), the file type seems able to upload to the blob storage, however it does not have a content type or suffix, hence I think finding a way to pass in the content_type of the file could be another solution, but it was harder than I thought.
I hope there's enough information, I desperately need someone to clear my confusion. Thank you very much.
You were pretty close, but there are some improvements. For example, you are using synchronous and asynchronous code together. This leads to confusing issues, such as not awaiting f.read() while that is an async method. Also, you don't have to pass content type to the azure storage blob client, just leave that to Azure.
I would recommend to use the async version of the azure-storage-blob package.
I re-wrote it a little. The below will run as-is and is a full working example.
from fastapi import FastAPI, HTTPException, UploadFile
from azure.storage.blob.aio import BlobServiceClient
app = FastAPI()
#app.post("/uploadfile")
async def create_upload_file(file: UploadFile):
name = file.filename
type = file.content_type
return await uploadtoazure(file,name,type)
async def uploadtoazure(file: UploadFile,file_name: str,file_type:str):
connect_str = "HERE_YUOUR_CONNECTION_STRING"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "stackoverflow"
async with blob_service_client:
container_client = blob_service_client.get_container_client(container_name)
try:
blob_client = container_client.get_blob_client(file_name)
f = await file.read()
await blob_client.upload_blob(f)
except Exception as e:
print(e)
return HTTPException(401, "Something went terribly wrong..")
return "{'did_it_work':'yeah it did!'}"
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000, )
Hope this clarifies some bits!
So I want to take my mongo db and use it in react, its small so it won't overpower react. My Flask looks like this
import subprocess
from flask import Flask, request, jsonify, json
from flask_cors import CORS, cross_origin
import pymongo
disaster = ""
app = Flask(__name__)
CORS(app, support_credentials=True)
client = pymongo.MongoClient("NOT IMPORTANT FOR GITHUB")
db = client["twitterdb"]
col = db["tweets"]
our_mongo_database_compressed = col.find({},{'user.created_at':1, 'user.location':1,'_id':0})
def request_tweets(disaster):
print(disaster)
#subprocess.call("./../../../backend/get_tweets", disaster)
#app.route('/refresh_data', methods=['GET', 'POST'])
##cross_origin(supports_credentials=True)
def refresh_data():
disaster = request.get_json()
request_tweets(disaster)
x = 0
y = []
for datas in our_mongo_database_compressed:
y.append(datas)
if(x > 100):
break
x+=1
#print(y)
return str(y)
and my react function looks like
this.setState({
disaster: event.target.value
})
axios.post('http://localhost:5000/refresh_data', [this.state.disaster])
.then(function(response){
console.log(JSON.parse(response.data));
})
}
I keep getting a " Unexpected token ' in JSON at position 2" and I just want the data to be sent to react
So I figured it out and I want anyone who has this problem in the future to be able to see this.
for datas in our_mongo_database_compressed:
y.append(datas)
This creates an array of dictionaries. so y[0]["user"]["location"] would be how you'd get an element from this array.
With that in mind we need to change this array to a JSON String which is a data type that you use to transfer between flask and react so you'd return
return json.dumps(y)
Now you might think this means that you got a string in React when you write
JSON.parse(response.data)
NO. That would be too easy, you have a response object that secretly a string. So you'd need to change the response object into a string with JSON stringify
JSON.parse(JSON.stringify(response.data))
Now you have your json in react
I'm following the docs and yet it appears the requests are still being made synchronously.
https://cloud.google.com/appengine/docs/standard/python/issue-requests
Here is my code:
rpcs = []
for url in urls:
rpc = urlfetch.create_rpc()
urlfetch.make_fetch_call(rpc, url)
rpcs.append(rpc)
result = []
for rpc in rpcs:
result.append(rpc.get_result().content)
return result
I did some profiling and compared using requests.get and they both take exactly the same amount of time.
The urls i'm fetching are from different sites so I'm sure that I don't have concurrent limitations on the server side.
Running on GAE Standard, Python 2.7
I got it working but for some reason only with callbacks. Also It only works on production and not on local env. :D. Here is the working code:
from google.appengine.api import urlfetch
import functools
class ClassName(object):
responses = []
def fetch_concurrent_callback(self, rpc):
response = rpc.get_result()
json_response = json.loads(response.content)
self.responses.append(json_response)
def fetch_concurrent(self, urls):
rpcs = []
for url in urls:
rpc = urlfetch.create_rpc()
rpc.callback = functools.partial(self.fetch_concurrent_callback, rpc)
urlfetch.make_fetch_call(rpc, url)
rpcs.append(rpc)
for rpc in rpcs:
rpc.wait()
return self.responses
How can I post message as a bot(async) in new hangouts chat without using the Google App Engine. I have gone through the examples, but all of them use App Engine for authentication, but i need to authenticate it without using the same.
Here is a code sample that connects to a chat using an http request and a webhook from Google Hangout Chat with a Python script. Webhooks are the only alternative to using a service account. More info here: https://developers.google.com/hangouts/chat/how-tos/webhooks
`from httplib2 import Http
from json import dumps
#
# Hangouts Chat incoming webhook quickstart
#
def main():
url = '<webhook url here>'
bot_message = {
'text' : 'text go here'}
message_headers = { 'Content-Type': 'application/json; charset=UTF-8'}
http_obj = Http()
response = http_obj.request(
uri=url,
method='POST',
headers=message_headers,
body=dumps(bot_message),
)
print(response)
if __name__ == '__main__':
main()
`
If your bot implementation is with google app script try to do it with google service account and as indicated here an example of async message
// Example bot for Hangouts Chat that demonstrates bot-initiated messages
// by spamming the user every minute.
//
// This bot makes use of the Apps Script OAuth2 library at:
// https://github.com/googlesamples/apps-script-oauth2
//
// Follow the instructions there to add the library to your script.
// When added to a space, we store the space's ID in ScriptProperties.
function onAddToSpace(e) {
PropertiesService.getScriptProperties()
.setProperty(e.space.name, '');
return {
'text': 'Hi! I\'ll post a message here every minute. ' +
'Please remove me after testing or I\'ll keep spamming you!'
};
}
// When removed from a space, we remove the space's ID from ScriptProperties.
function onRemoveFromSpace(e) {
PropertiesService.getScriptProperties()
.deleteProperty(e.space.name);
}
// Add a trigger that invokes this function every minute via the
// "Edit > Current Project's Triggers" menu. When it runs, it will
// post in each space the bot was added to.
function onTrigger() {
var spaceIds = PropertiesService.getScriptProperties()
.getKeys();
var message = { 'text': 'Hi! It\'s now ' + (new Date()) };
for (var i = 0; i < spaceIds.length; ++i) {
postMessage(spaceIds[i], message);
}
}
var SCOPE = 'https://www.googleapis.com/auth/chat.bot';
// The values below are copied from the JSON file downloaded upon
// service account creation.
var SERVICE_ACCOUNT_PRIVATE_KEY = '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n';
var SERVICE_ACCOUNT_EMAIL = 'service-account#project-id.iam.gserviceaccount.com';
// Posts a message into the given space ID via the API, using
// service account authentication.
function postMessage(spaceId, message) {
var service = OAuth2.createService('chat')
.setTokenUrl('https://accounts.google.com/o/oauth2/token')
.setPrivateKey(SERVICE_ACCOUNT_PRIVATE_KEY)
.setClientId(SERVICE_ACCOUNT_EMAIL)
.setPropertyStore(PropertiesService.getUserProperties())
.setScope(SCOPE);
if (!service.hasAccess()) {
Logger.log('Authentication error: %s', service.getLastError());
return;
}
var url = 'https://chat.googleapis.com/v1/' + spaceId + '/messages';
UrlFetchApp.fetch(url, {
method: 'post',
headers: { 'Authorization': 'Bearer ' + service.getAccessToken() },
contentType: 'application/json',
payload: JSON.stringify(message),
});
}
You need to perform some below steps.
Create a service-account in console.developers.google.com and download the private key in JSON format
Use below modules if you code in python.
from oauth2client.service_account import ServiceAccountCredentials
from googleapiclient.discovery import build, build_from_document
from httplib2 import Http
Below snippet will post the message to user via chat.google.
scopes = ['https://www.googleapis.com/auth/chat.bot']
credentials = ServiceAccountCredentials.from_json_keyfile_name('/path/to/json',
scopes)
http = Http()
credentials.authorize(http)
chat = build('chat', 'v1', http=http)
resp = chat.spaces().messages().create(
parent=space,
body={'text': 'HELLO WORLD'}).execute()
You would require a space name where you can post the code. You will get the same from hangout chat response.
It’s possible to do so using JavaScript, python, (possibly more). You can check out examples here: https://github.com/gsuitedevs/hangouts-chat-samples/tree/master/node/basic-cloud-functions-bot
If you’re using cards and JavaScript I would encourage you to checkout my library here: https://github.com/BaReinhard/hangouts-card-helper
I’m also in the process of creating another example for JavaScript that is more async focused that should provide and example that’s a bit easier to reason about the code. Will link when the PR is pushed.
Edit:
I realize that you mentioned REST api. The above answer is more useful for a specific bot that can be accessed #mentions. However, if you can provide us with a bit more information I can better fix my answer to answer your question.
I'd like to avoid using Webapp from GAE, so i use this code to upload an image to the Blobstore (code snippet from : http://flask.pocoo.org/mailinglist/archive/2011/1/8/app-engine-blobstore/#7fd7aa9a5c82a6d2bf78ccd25084ac3b)
#app.route("/upload", methods=['POST'])
def upload():
if request.method == 'POST':
f = request.files['file']
header = f.headers['Content-Type']
parsed_header = parse_options_header(header)
blob_key = parsed_header[1]['blob-key']
return blob_key
It returns what it seems to be indeed a Blobkey, wich is something like this :
2I9oX6J0U5nBCVw8kEndpw==
I then try to display the recently stored Blob image with this code :
#app.route("/testimgdisplay")
def test_img_display():
response = make_response(db.get("2I9oX6J0U5nBCVw8kEndpw=="))
response.headers['Content-Type'] = 'image/png'
return response
Sadly this part doesn't work, I got the following error :
BadKeyError: Invalid string key 2I9oX6J0U5nBCVw8kEndpw==
Do you guys have faced this error before ? It seems the Blobkey is well-formatted, and I can't find a clue.
There was a simple mistake on the call for getting the Blob, I wrote:
db.get("2I9oX6J0U5nBCVw8kEndpw==")
and the right call was instead:
blobstore.get("2I9oX6J0U5nBCVw8kEndpw==")
For those looking for a complete Upload/Serving image via GAE Blobstore and Flask without using Webapp, here is the complete code:
Render the template for the upload form:
#app.route("/upload")
def upload():
uploadUri = blobstore.create_upload_url('/submit')
return render_template('upload.html', uploadUri=uploadUri)
Place your uploadUri in the form path (html):
<form action="{{ uploadUri }}" method="POST" enctype="multipart/form-data">
Here is the function to handle the upload of the image (I return the blob_key for practical reasons, replace it with your template):
#app.route("/submit", methods=['POST'])
def submit():
if request.method == 'POST':
f = request.files['file']
header = f.headers['Content-Type']
parsed_header = parse_options_header(header)
blob_key = parsed_header[1]['blob-key']
return blob_key
Now say you serve your images with a path like this:
/img/imagefilename
Then your image serving function is :
#app.route("/img/<bkey>")
def img(bkey):
blob_info = blobstore.get(bkey)
response = make_response(blob_info.open().read())
response.headers['Content-Type'] = blob_info.content_type
return response
Finally, anywhere you need to display an image in a template, you simply put the code:
<img src="/img/{{ bkey }} />
I don't think Flask is any better or worse than Webapp in serving up Blobstore images, since they both use the Blobstore API for Serving a Blob.
What you're calling a Blobkey is just a string, which needs to be converted into a key (called resource here):
from google.appengine.ext import blobstore
from google.appengine.ext.webapp import blobstore_handlers
class ServeHandler(blobstore_handlers.BlobstoreDownloadHandler):
def get(self, resource):
resource = str(urllib.unquote(resource))
blob_info = blobstore.BlobInfo.get(resource)
self.send_blob(blob_info)