I'm trying to create a function that let me select a file from input on the frontend, then pass the file to the backend FastAPI, and eventually upload the file to Azure Blob storage. My Frontend code is below:
//front end
async function handleSubmit(){
const formdata = new FormData();
formdata.append(
"file",
file[0],
)
const headers={'Content-Type': file[0].type}
await axios.post("/uploadfile",formdata,headers)
.then(function (response) {
console.log(response)
});
}
Backend FastAPI - two methods I tried
//Backend FastAPI
#app.post("/uploadfile") //currently using
async def create_upload_file(file: UploadFile):
name = file.filename
type = file.content_type
return uploadtoazure(file,name,type)
#app.post("/files") //another method I tried
async def create_file(file: bytes= File()):
await uploadtoazure(file)
And the uploadtoazure() function
//Backend uploadtoazure() function
async def uploadtoazure(f,n,t):
connect_str = ""//removed from sample code
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "notes"
file = f.read()
local_file_name = n
cnt_settings = ContentSettings(content_type=t)
blob_client = blob_service_client.get_blob_client(container=container_name, blob=local_file_name)
blob_client.upload_blob(file,cnt_settings)
The error kept coming at me and when I tried another method another new error prevents me from moving forward.
Some problem I'm aware:
blob_client.upload_blob() only accept some types of files, in which the file type I passed in from API isn't one of it.
I used the sample code in https://fastapi.tiangolo.com/tutorial/request-files/In which I am not quite sure of the characteristic of the class UploadFile and File().
When I use the method which the file passed to API is file: bytes= File(), the file type seems able to upload to the blob storage, however it does not have a content type or suffix, hence I think finding a way to pass in the content_type of the file could be another solution, but it was harder than I thought.
I hope there's enough information, I desperately need someone to clear my confusion. Thank you very much.
You were pretty close, but there are some improvements. For example, you are using synchronous and asynchronous code together. This leads to confusing issues, such as not awaiting f.read() while that is an async method. Also, you don't have to pass content type to the azure storage blob client, just leave that to Azure.
I would recommend to use the async version of the azure-storage-blob package.
I re-wrote it a little. The below will run as-is and is a full working example.
from fastapi import FastAPI, HTTPException, UploadFile
from azure.storage.blob.aio import BlobServiceClient
app = FastAPI()
#app.post("/uploadfile")
async def create_upload_file(file: UploadFile):
name = file.filename
type = file.content_type
return await uploadtoazure(file,name,type)
async def uploadtoazure(file: UploadFile,file_name: str,file_type:str):
connect_str = "HERE_YUOUR_CONNECTION_STRING"
blob_service_client = BlobServiceClient.from_connection_string(connect_str)
container_name = "stackoverflow"
async with blob_service_client:
container_client = blob_service_client.get_container_client(container_name)
try:
blob_client = container_client.get_blob_client(file_name)
f = await file.read()
await blob_client.upload_blob(f)
except Exception as e:
print(e)
return HTTPException(401, "Something went terribly wrong..")
return "{'did_it_work':'yeah it did!'}"
if __name__ == "__main__":
import uvicorn
uvicorn.run(app, host="0.0.0.0", port=8000, )
Hope this clarifies some bits!
Related
I have an issue to send the contents of a .mat file to my frontend. My end goal is to allow clients to download the content of this .mat file at the click of a button so that they end up with the same file in their possession. I use Next.js + Django Rest Framework.
My first try was as follow:
class Download(APIView):
def get(self, request):
with open('file_path.mat', 'rb') as FID:
fileInstance = FID.read()
return Response(
fileInstance,
status=200,
content_type="application/octet-stream",
)
If I print out the fileInstance element I get some binary results:
z\xe1\xfe\xc6\xc6\xd2\x1e_\xda~\xda|\xbf\xb6\x10_\x84\xb5~\xfe\x98\x1e\xdc\x0f\x1a\xee\xe7Y\x9e\xb5\xf5\x83\x9cS\xb3\xb5\xd4\xb7~XK\xaa\xe3\x9c\xed\x07v\xf59Kbn(\x91\x0e\xdb\xbb\xe8\xf5\xc3\xaa\x94Q\x9euQ\x1fx\x08\xf7\x15\x17\xac\xf4\x82\x19\x8e\xc9...
But I can't send it back to my frontend because of a
"UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9c in position 137: invalid start byte"
This error is always the same regardless of which .mat file I try to send in my response.
Next I tried to use the scipy.io.loadmat() method. In this case, fileInstance gives me a much more readable dictionary object, but I still can't get it to transfer to the frontend because of the presence of NaN in my dict:
ValueError: Out of range float values are not JSON compliant
Finally, some suggested to use h5py to send back the data as such:
with h5py.File('file_path.mat', 'r') as fileInstance:
print(fileInstance)
But in that case the error I get is
Unable to open file (file signature not found)
I know my files are not corrupted because I can open them in Matlab with no problem.
With all this trouble I'm wondering if I'm using the right approach to this problem. I could technically send the dictionary obtained through 'scipy.io.loadmat()' as a str element instead of binary, but I'll have to figure out a way to convert this text back to binary inside a Javascript function. Would anybody have some ideas as to how I should proceed?
The problem was in my frontend after all. Still, here's the correct way to go about it:
class Download(APIView):
parser_classes = [FormParser, MultiPartParser]
def get(self, request):
try:
file_path = "xyz.mat"
response = FileResponse(file_path.open("rb"), content_type="application/octet-stream")
response["Content-Disposition"] = f"attachment; filename=file_name"
return response
except Exception as e:
return Response(status=500)
This should send to the frontend the right file in the right format. No need to worry about encoding and such.
Meanwhile, on the frontend you should receive the file as follows:
onClick={() => {
const url = '/url_to_your_api/';
axios({ method: 'get', url: url, responseType: 'blob' })
.then((response) => {
const { data } = response;
const fileName = 'file_name';
const blob = new Blob([data], { type: 'application/octet-stream' });
const href = URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = href;
link.download = fileName + '.mat';
document.body.appendChild(link);
link.click();
document.body.removeChild(link);
URL.revokeObjectURL(href);
})
.catch((response) => {
console.error(response);
});
}}
Long story short, the part I was missing was to specify to receive the data as blob inside the 'onClick()' function. By default, responseType from Axios is set to Json/String. For that reason, my file was modified at reception and would not be usable in matlab afterwards. If you face a similar problem in the future, try to use the 'shasum' BASH function to observe the hashed value of the file. It is with the help of that function that I could deduce that my API function would return the correct value and that therefore the problem was happenign on the frontend.
The front end enables people to upload their photos, so i was sending the base64 to the server and working with it initially, but there are problems with firewall which blocks the request which contains base64. As an alternative solution I was trying to upload the image to azure blob get the file name and then send that to the server for processing where I generate a sas token for the blob validation and processing.
This works perfectly fine when I work locally and the front end connection works with #azure/storage-blob
and uploadBrowserData() when I send the arrayBuffer as the param
export const uploadSelfieToBlob = async arrayBuffer => {
try {
const blobURL = `https://${accountName}.blob.core.windows.net${sasString}`;
const blobServiceClient = new BlobServiceClient(blobURL, anonymousCredential);
const containerClient = blobServiceClient.getContainerClient(containerName);
let randomString = Math.random().toString(36).substring(7);
const blobName = `${randomString}_${new Date().getTime()}.jpg`;
const blockBlobClient = containerClient.getBlockBlobClient(blobName);
const uploadBlobResponse = await blockBlobClient.uploadBrowserData(arrayBuffer);
return { blobName, blobId: uploadBlobResponse.requestId };
} catch (error) {
console.log('error when uploading to blob', error);
throw new Error('Error Uploading the selfie to blob');
}
};
When I deploy this is not working, the front is deployed in the EastUs2 location and the local development location is different.
I thought the sasString generated for anonymous access had the timezone option so I generated 2 different one's one for local and one for hosted server with the same location selected.
Failed to send request to https://xxxx.blob.core.windows.net/contanainer-name/26pcie_1582087489288.jpg?sv=2019-02-02&ss=b&srt=c&sp=rwdlac&se=2023-09-11T07:57:29Z&st=2020-02-18T00:57:29Z&spr=https&sig=9IWhXo5i%2B951%2F8%2BTDqIY5MRXbumQasOnY4%2Bju%2BqF3gw%3D
What am I missing any lead would be helpful thanks
First, as mentioned in the comments there was an issue with the CORS Settings because of which you're getting the initial error.
AuthorizationResourceTypeMismatchThis
request is not authorized to perform this operation using this
resource type. RequestId:7ec96c83-101e-0001-4ef1-e63864000000
Time:2020-02-19T06:57:31.2867563Z
I looked up this error code here and then closely looked at your SAS URL.
One thing I noticed in your SAS URL is that you have set the signed resource type (srt) as c (container) and trying to upload the blob. If you look at the description of the kind of operations you can do using srt=c here, you will notice that blob related operations are not supported.
In order to perform blob related operations (like blob upload), you would need to set signed resource type value to o (for object).
Please regenerate your SAS Token and include signed resource type as object (you can also include container and/or service in there as well) and then your request should work. So essentially your srt in your SAS URL should be something like srt=o or srt=co or srt=sco.
I couldn't notice anything wrong with the code you mentioned about, but I have been using a different method to upload files to Azure Blog Storage using React, the method is exactly the same as in this blog article which works perfectly for me.
https://medium.com/#stuarttottle/upload-to-azure-blob-storage-with-react-34f37805fdfc
I’m using next.js to build static HTML webpages.
One of my webpages needs data from a third-party API, which I’d like to fetch at build time and bake into the resulting HTML.
I don’t want this call to ever happen on the client, because:
CORS prevents the request from succeeding anyway
I would have to expose an API key on the client (no thank you)
I thought getInitialProps was the answer, because the fetched data is indeed baked in during the build/export process, but when I navigate away from the page and return from it, getInitialProps gets triggered on the client, breaking everything.
My current code in getInitialProps is something like:
static async getInitialProps(){
// Get Behance posts
const behanceEndpoint = `https://www.behance.net/v2/users/${process.env.BEHANCE_USERNAME}/projects?api_key=${process.env.BEHANCE_API_KEY}`
const behanceRes = await fetch(behanceEndpoint)
let behancePosts = await behanceRes.json()
// Return only the required number of posts
return {
behancePosts: behancePosts
}
}
Any advice or best practice on how to handle this? I know Gatsby.js does it out of the box.
one possibility would be, if you just want to execute this once on the server to check if the req parameter is present in getInitialProps: (Documentation)
req - HTTP request object (server only).
One dirty approach:
static async getInitialProps({ req }){
if (req) {
// only executed on server
// Get Behance posts
const behanceEndpoint = `https://www.behance.net/v2/users/${process.env.BEHANCE_USERNAME}/projects?api_key=${process.env.BEHANCE_API_KEY}`
const behanceRes = await fetch(behanceEndpoint)
let behancePosts = await behanceRes.json()
// Return only the required number of posts
return {
behancePosts: behancePosts
}
} else {
// client context
}
Hope this helps a little bit.
How can I post message as a bot(async) in new hangouts chat without using the Google App Engine. I have gone through the examples, but all of them use App Engine for authentication, but i need to authenticate it without using the same.
Here is a code sample that connects to a chat using an http request and a webhook from Google Hangout Chat with a Python script. Webhooks are the only alternative to using a service account. More info here: https://developers.google.com/hangouts/chat/how-tos/webhooks
`from httplib2 import Http
from json import dumps
#
# Hangouts Chat incoming webhook quickstart
#
def main():
url = '<webhook url here>'
bot_message = {
'text' : 'text go here'}
message_headers = { 'Content-Type': 'application/json; charset=UTF-8'}
http_obj = Http()
response = http_obj.request(
uri=url,
method='POST',
headers=message_headers,
body=dumps(bot_message),
)
print(response)
if __name__ == '__main__':
main()
`
If your bot implementation is with google app script try to do it with google service account and as indicated here an example of async message
// Example bot for Hangouts Chat that demonstrates bot-initiated messages
// by spamming the user every minute.
//
// This bot makes use of the Apps Script OAuth2 library at:
// https://github.com/googlesamples/apps-script-oauth2
//
// Follow the instructions there to add the library to your script.
// When added to a space, we store the space's ID in ScriptProperties.
function onAddToSpace(e) {
PropertiesService.getScriptProperties()
.setProperty(e.space.name, '');
return {
'text': 'Hi! I\'ll post a message here every minute. ' +
'Please remove me after testing or I\'ll keep spamming you!'
};
}
// When removed from a space, we remove the space's ID from ScriptProperties.
function onRemoveFromSpace(e) {
PropertiesService.getScriptProperties()
.deleteProperty(e.space.name);
}
// Add a trigger that invokes this function every minute via the
// "Edit > Current Project's Triggers" menu. When it runs, it will
// post in each space the bot was added to.
function onTrigger() {
var spaceIds = PropertiesService.getScriptProperties()
.getKeys();
var message = { 'text': 'Hi! It\'s now ' + (new Date()) };
for (var i = 0; i < spaceIds.length; ++i) {
postMessage(spaceIds[i], message);
}
}
var SCOPE = 'https://www.googleapis.com/auth/chat.bot';
// The values below are copied from the JSON file downloaded upon
// service account creation.
var SERVICE_ACCOUNT_PRIVATE_KEY = '-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n';
var SERVICE_ACCOUNT_EMAIL = 'service-account#project-id.iam.gserviceaccount.com';
// Posts a message into the given space ID via the API, using
// service account authentication.
function postMessage(spaceId, message) {
var service = OAuth2.createService('chat')
.setTokenUrl('https://accounts.google.com/o/oauth2/token')
.setPrivateKey(SERVICE_ACCOUNT_PRIVATE_KEY)
.setClientId(SERVICE_ACCOUNT_EMAIL)
.setPropertyStore(PropertiesService.getUserProperties())
.setScope(SCOPE);
if (!service.hasAccess()) {
Logger.log('Authentication error: %s', service.getLastError());
return;
}
var url = 'https://chat.googleapis.com/v1/' + spaceId + '/messages';
UrlFetchApp.fetch(url, {
method: 'post',
headers: { 'Authorization': 'Bearer ' + service.getAccessToken() },
contentType: 'application/json',
payload: JSON.stringify(message),
});
}
You need to perform some below steps.
Create a service-account in console.developers.google.com and download the private key in JSON format
Use below modules if you code in python.
from oauth2client.service_account import ServiceAccountCredentials
from googleapiclient.discovery import build, build_from_document
from httplib2 import Http
Below snippet will post the message to user via chat.google.
scopes = ['https://www.googleapis.com/auth/chat.bot']
credentials = ServiceAccountCredentials.from_json_keyfile_name('/path/to/json',
scopes)
http = Http()
credentials.authorize(http)
chat = build('chat', 'v1', http=http)
resp = chat.spaces().messages().create(
parent=space,
body={'text': 'HELLO WORLD'}).execute()
You would require a space name where you can post the code. You will get the same from hangout chat response.
It’s possible to do so using JavaScript, python, (possibly more). You can check out examples here: https://github.com/gsuitedevs/hangouts-chat-samples/tree/master/node/basic-cloud-functions-bot
If you’re using cards and JavaScript I would encourage you to checkout my library here: https://github.com/BaReinhard/hangouts-card-helper
I’m also in the process of creating another example for JavaScript that is more async focused that should provide and example that’s a bit easier to reason about the code. Will link when the PR is pushed.
Edit:
I realize that you mentioned REST api. The above answer is more useful for a specific bot that can be accessed #mentions. However, if you can provide us with a bit more information I can better fix my answer to answer your question.
I want to download file that can be in any format viz. pdf, jpeg, png, xlsx, csv etc. The download API on backend using pyramid framework is sending FileResponse as below:
def delivery_item_download_view(request, *args, **kw):
context = request.context
item_row = context.item_row
if item_row and item_row["deleted_at"] is None:
print(request.upload_dir+'/'+item_row["file_name"]+'.'+item_row["file_extension"])
response = FileResponse(
request.upload_dir+'/'+item_row["file_name"]+'.'+item_row["file_extension"],
request=request,
)
response.headers["attachment"] = item_row["name"];
return response
This, when executed using POSTMAN works as expected giving file as output. However,when tried implementing same using ReactJS, it's not working as expected. My client-code is as below:
onDownloadItem= (item) => {
console.log("item id is:", item.item_id)
var apiBaseUrl = "https://dev.incodax.com/api/deliveries_items/"+ item.item_id+ "/download";
fetch(apiBaseUrl, {
method: "GET",
}).then((res) => {
fileDownload(res,item.file_name)
console.log(res)
})
}
This fileDownload function simply downloading file but with no content inside. In downloaded file I could see something like:
[object Response]
I am getting 200 response from server. So I dont't think there is any issue with server side code. How can I handle it on client?
Thanks in advance
Will it suit you if you just redirected user to link to file? Browser will automatically handle it and download it.
The issue is in your fileDownload function which you do not post here. It's not clear what the first parameter is supposed to be but likely it is not the response object. Likely you at least need to pull the body out of the response and save that! The response body can be converted to a buffer object which could work (again it depends on what fileDownload is expecting):
res.arrayBuffer().then(buffer => {
fileDownload(buffer, item.file_name);
});