Upload file in GAE Go - file

I am trying to upload a file in my GAE app. How do I the upload of a file in Google App Engine using Go and using the r.FormValue()?

You have to go through the Blobstore Go API Overview to get an idea and there is a full example on how could you store & serve user data on Google App Engine using Go.
I would suggest you to do that example in a completely separate application, so you'll be able to experiment with it for a while before trying to integrate it to your already existing one.

I managed to solve my problem by using the middle return param, "other". These code below are inside the upload handler
blobs, other, err := blobstore.ParseUpload(r)
Then assign corresponding formkey
file := blobs["file"]
**name := other["name"]** //name is a form field
**description := other["description"]** //descriptionis a form field
And use it like this in my struct value assignment
newData := data{
Name: **string(name[0])**,
Description: **string(description[0])**,
Image: string(file[0].BlobKey),
}
datastore.Put(c, datastore.NewIncompleteKey(c, "data", nil), &newData )
Not 100% sure this is the right thing but this solves my problem and it is now uploading the image to blobstore and saving other data and blobkey to datastore.
Hope this could help others too.

I have tried the full example from here https://developers.google.com/appengine/docs/go/blobstore/overview, and it worked fine doing the upload in blobstore and serving it.
But inserting extra post values to be saved somewhere in the datastore erases the values of "r.FormValue() "? Please refer to the code below
func handleUpload(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
//tried to put the saving in the datastore here, it saves as expected with correct values but would raised a server error.
blobs, _, err := blobstore.ParseUpload(r)
if err != nil {
serveError(c, w, err)
return
}
file := blobs["file"]
if len(file) == 0 {
c.Errorf("no file uploaded")
http.Redirect(w, r, "/", http.StatusFound)
return
}
// a new row is inserted but no values in column name and description
newData:= data{
Name: r.FormValue("name"), //this is always blank
Description: r.FormValue("description"), //this is always blank
}
datastore.Put(c, datastore.NewIncompleteKey(c, "Data", nil), &newData)
//the image is displayed as expected
http.Redirect(w, r, "/serve/?blobKey="+string(file[0].BlobKey), http.StatusFound)
}
Is it not possible to combine the upload with regular data? How come the values of r.FormValue() seems to disappear except for the file (input file type)? Even if I would have to force upload first before associating the blobkey, as the result of the upload, to other data, it would not be possible since I could not pass any r.FormValue() to the upload handler(which like I said becomes empty, or would raised an error when accessed prior the blobs, _, err := blobstore.ParseUpload(r) statement). I hope someone could help me solve this problem. Thank you!

In addition to using the Blobstore API, you can just use the Request.FormFile() method to get the file upload content. Use net\http package documentation for additional help.
Using the Request directly allows you to skip setting up an blobstore.UploadUrl() before handling the upload POST message.
A simple example would be:
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// Create an App Engine context.
c := appengine.NewContext(r)
// use FormFile()
f, _, err := r.FormFile("file")
if err != nil {
c.Errorf("FormFile error: %v", err)
return
}
defer f.Close()
// do something with the file here
c.Infof("Hey!!! got a file: %v", f)
}

Related

Parse multipart form on Google App Engine

A project I'm working on depends on having a service hosted on Google App Engine parse from SendGrid. The following code is an example of what we're doing:
package sendgrid_failure
import (
"net/http"
"fmt"
"google.golang.org/appengine"
"google.golang.org/appengine/log"
)
func init() {
http.HandleFunc("/sendgrid/parse", sendGridHandler)
}
func sendGridHandler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
err := r.ParseMultipartForm(-1)
if err != nil {
log.Errorf(ctx, "Unable to parse form: %v", err)
}
fmt.Fprint(w, "Test.")
}
When SendGrid POSTs its multipart form, the console shows similar to the following:
2018/01/04 23:44:08 ERROR: Unable to parse form: open /tmp/multipart-445139883: no file writes permitted on App Engine
App Engine doesn't allow you to read/write files, but Golang appears to need it to parse. Is there an App Engine specific library to parse multipart forms, or should we be using a different method from the standard net/http library entirely? We're using the standard go runtime.
The documentation for ParseMultipartForm says:
The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.
The server attempts to write all files to disk because the application passed -1 as maxMemory. Use a value larger than the size of the files you expect to upload.

Trouble creating signed URL in App Engine

I'm trying to create a signed URL to download a file in Google Cloud Storage, from an App Engine app written in Go. There's a nifty signing method in App Engine which I'm using, which [in theory] allows me to avoid putting a private key in my app. However, the URLs don't appear to work: I always get a 403 "SignatureDoesNotMatch" error. Any ideas?
Here's the code:
func createDownloadURL(c context.Context, resource string, validUntil time.Time, bucket string) (string, error) {
serviceAccount, err := appengine.ServiceAccount(c)
if err != nil {
return "", err
}
// Build up the string to sign.
validUntilString := strconv.FormatInt(validUntil.Unix(), 10)
toSign := []string{
"GET", // http verb (required)
"", // content MD5 hash (optional)
"", // content type (optional)
validUntilString, // expiration (required)
resource, // resource (required)
}
// Sign it.
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\n")))
if err != nil {
return "", err
}
signedString := base64.StdEncoding.EncodeToString(signedBytes)
// Build and return the URL.
arguments := url.Values{
"GoogleAccessId": {serviceAccount},
"Expires": {validUntilString},
"Signature": {signedString},
}
return fmt.Sprintf("https://storage.googleapis.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
}
Solved. There were 2 problems with my code.
I forgot to include the bucket name when building up toSign. Fix:
fmt.Sprintf("/%s/%s", bucket, resource), // resource (required)
This returned an AccessDenied error -- progress!
The second mistake was using the XML API storage.googleapis.com instead of the authenticated browser endpoint storage.cloud.google.com. Fix:
return fmt.Sprintf("https://storage.cloud.google.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
This works.
StringToSign may require uninterpreted newlines. Could you give this a try:
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\\n"))) // escaped newline
Writing a function that signs URLs is tricky, since due to the nature of encryption it's very difficult to tell what's wrong when things don't work. You may find it easier to use a library like gcloud-golang, which has a SignedURL method.

google storage BlobKeyForFile error

Im trying to serve images from google storage via google images
In order to do that i need to create blob key.
i tried several ways togenerate the key but got errors
loc := fmt.Sprintf("/gs/%s/%s", BUCKET, s)
applog.Infof(appCtx, "%s", loc)
bkey, err := blobstore.BlobKeyForFile(appCtx, loc)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
opt := &image.ServingURLOptions{}
u, err := image.ServingURL(appCtx, bkey, opt)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
// i tried with file extension to
/gs/bucktname/CXlvJUKiTmo9joe6
OBJECT_NOT_FOUND
gs://bucktname/rUAJOYKQbORzOYvs
"description": "API error 6 (images: INVALID_BLOB_KEY)"
gs:/bucktname/MlY77iFNbBca2KCA
"description": "API error 6 (images: INVALID_BLOB_KEY)"
What is the correct path ?
Dose the images cached and behind cdn ?
I got this error as well (among others) and here are the fixes which worked for me:
When saving the file to Google Cloud Storage you need to make it public like so: writer.ACL = []storage.ACLRule{{Entity: storage.AllUsers, Role: storage.RoleReader}}
The correct path is fmt.Sprintf("/gs/%s/%s", bucketName, objectName) where objectName is the path to the file within the bucket.
You have to make the bucket public. How to do that is described in here.

Why doesn't fmt.Println work in Google app engine

I built a simple web app using google app engine and golang. in code below, I use fmt.Println twice to print out somehting for debugging purpose. I have no problem running the app. everything works except nothing print out on the terminal.
func HomeHandler(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
q := datastore.NewQuery("Post").Ancestor(goblogKey(c)).Order("-CreatedOn").Limit(10)
//posts := make([]entity.Post, 0, 10)
var posts []entity.Post
if _, err := q.GetAll(c, &posts); err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
fmt.Println(string(len(posts)) + "...........")
postList := []dto.Post{}
for _, val := range posts {
newpost := dto.Post{
Post: val,
BodyHTML: template.HTML(val.Body),
}
fmt.Println(val.Title)
postList = append(postList, newpost)
}
page := dto.PageData{Title: "Home", Posts: postList}
templates.ExecuteTemplate(w, "index", page)
}
In the real appengine enviroment you can't se anything output to stdout.
Appengine context give you away to log (that you can check in you appengine admin's page and in console playground).
func HomeHandler(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
c.Debugf("The message: %s", "foo")
...
Read more: https://developers.google.com/appengine/docs/go/reference#Context
standard i/o Or Error is used communicate with app server used by the devleoper. In production system there's no meaning of using standard i/o. In production systems log is used to track the results. In app engine there's some limitations. like fmt, socket etc.
Its always better to use log when testing or running program in remote server.

GAE Golang - log.Print()?

Where can one read logs created by calling function:
log.Print("Message")
The tab "Logs" under Main seems to only display information about what URLs were called, but without any debug information that would be displayed by the application.
As described in the documentation, you should use the Context interface to log instead of log, if you want your logs to show up in the console.
c := appengine.NewContext(r)
c.Infof("Requested URL: %v", r.URL)
If you are using the new App Engine package google.golang.org/appengine, in the README:
Logging methods that were on appengine.Context are now functions in google.golang.org/appengine/log
So you should use
c := appengine.NewContext(r)
log.Infof(c, "Requested URL: %v", r.URL)
The same context object must be passed around in other method calls.
Here is an example:
func handleSign(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
if err := r.ParseForm(); err != nil {
writeError(c, err)
return
}
}
func writeError(c appengine.Context, err os.Error) {
c.Errorf("%v", err)
}

Resources