I'm trying to create a signed URL to download a file in Google Cloud Storage, from an App Engine app written in Go. There's a nifty signing method in App Engine which I'm using, which [in theory] allows me to avoid putting a private key in my app. However, the URLs don't appear to work: I always get a 403 "SignatureDoesNotMatch" error. Any ideas?
Here's the code:
func createDownloadURL(c context.Context, resource string, validUntil time.Time, bucket string) (string, error) {
serviceAccount, err := appengine.ServiceAccount(c)
if err != nil {
return "", err
}
// Build up the string to sign.
validUntilString := strconv.FormatInt(validUntil.Unix(), 10)
toSign := []string{
"GET", // http verb (required)
"", // content MD5 hash (optional)
"", // content type (optional)
validUntilString, // expiration (required)
resource, // resource (required)
}
// Sign it.
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\n")))
if err != nil {
return "", err
}
signedString := base64.StdEncoding.EncodeToString(signedBytes)
// Build and return the URL.
arguments := url.Values{
"GoogleAccessId": {serviceAccount},
"Expires": {validUntilString},
"Signature": {signedString},
}
return fmt.Sprintf("https://storage.googleapis.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
}
Solved. There were 2 problems with my code.
I forgot to include the bucket name when building up toSign. Fix:
fmt.Sprintf("/%s/%s", bucket, resource), // resource (required)
This returned an AccessDenied error -- progress!
The second mistake was using the XML API storage.googleapis.com instead of the authenticated browser endpoint storage.cloud.google.com. Fix:
return fmt.Sprintf("https://storage.cloud.google.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
This works.
StringToSign may require uninterpreted newlines. Could you give this a try:
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\\n"))) // escaped newline
Writing a function that signs URLs is tricky, since due to the nature of encryption it's very difficult to tell what's wrong when things don't work. You may find it easier to use a library like gcloud-golang, which has a SignedURL method.
Related
When i tried to implement push notification in golang App Engine using onesignal enviorment.But iam getting error "http.DefaultTransport and http.DefaultClient are not available in App Engine".This is my code,
func (c *PushNotificationController) CreateNotification() {
client := onesignal.NewClient(nil)
client.AppKey = "MyAppKey"
client.UserKey = "MyUserKey"
notifID := CreateNotifications(client)
log.Println(notifID)
}
func CreateNotifications(client *onesignal.Client) string {
playerID := "SamplePlayerId" // valid
notificationReq := &onesignal.NotificationRequest{
AppID: "MyAppKey",
Contents: map[string]string{"en": "English message"},
IsIOS: true,
IncludePlayerIDs: []string{playerID},
}
if createRes, res, err := client.Notifications.Create(notificationReq){
if err != nil {
log.Fatal(err)
}
return createRes.ID
}
...
}
Use http on appengine, you have to use urlfetch.
https://cloud.google.com/appengine/docs/standard/go/urlfetch/reference
i.e. the package you use doesn't support appengine.
A similar issue has been addressed by Robby Colvin in this blog
It explains how to make a third party package run in such conditions. Hoping this will help.
How would I go about creating a GET HTTP request mid application in GAE? I don't want to have it as a handler function, I simply have a URL that I need to get the response body from.
Use the urlfetch package.
ctx := appengine.NewContext(r) // r is the *http.Request arg to the handler
client := urlfetch.Client(ctx)
resp, err := client.Get("http://example.com")
if err != nil {
// handle the error
}
body := resp.Body // body is an io.Reader containing the response body
Here's a complete example.
I haven't used it so apologies if this doesn't work. According to GAE's docs you probably want to use urlfetch to get a *http.Client something like (N.B. the context package is standard in the just released Go 1.7):
import (
"context" // Go 1.7
// "golang.org/x/net/context" // Go < 1.7
"google.golang.org/appengine/urlfetch"
)
client := urlfetch.Client(context.Background())
resp, err := client.Get("http://example.com/")
What is the GO appropriate way to create a FetchUrl/GetURL function that works from the command line and works from google app engine with its custom way to fetch a url.
I have basic code that fetches and processes some data on a URL. I want to be able to call it from code I use on my desktop, and code deployed to app engine.
Hopefully thats clear, if not please let me know and Ill clarify.
If you have some code which works both on local machine and on AppEngine environment, you have nothing to do.
If you need to do something which should or must be done differently on AppEngine, then you need to detect the environment and write different code for the different environments.
This detection and code selection is easiest done using build constraints. You can put a special comment line in the beginning of your .go file, and it may or may not be compiled and run depending on the environment.
Quoting from The Go Blog: The App Engine SDK and workspaces (GOPATH):
The App Engine SDK introduces a new build constraint term: "appengine". Files that specify
// +build appengine
will be built by the App Engine SDK and ignored by the go tool. Conversely, files that specify
// +build !appengine
are ignored by the App Engine SDK, while the go tool will happily build them.
So for example you can have 2 separate .go files, one for AppEngine and one for local (non-AppEngine) environment. Define the same function in both (with same parameter list), so no matter in which environment the code is built, the function will have one declaration. We will use this signature:
func GetURL(url string, r *http.Request) ([]byte, error)
Note that the 2nd parameter (*http.Request) is only required for the AppEngine (in order to be able to create a Context), so in the implementation for local env it is not used (can even be nil).
An elegant solution can take advantage of the http.Client type which is available in both the standard environment and in AppEngine, and which can be used to do an HTTP GET request. An http.Client value can be acquired differently on AppEngine, but once we have an http.Client value, we can proceed the same way. So we will have a common code that receives an http.Client and can do the rest.
Example implementation can look like this:
url_local.go:
// +build !appengine
package mypackage
import (
"net/http"
)
func GetURL(url string, r *http.Request) ([]byte, error) {
// Local GetURL implementation
return GetClient(url, &http.Client{})
}
url_gae.go:
// +build appengine
package mypackage
import (
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
"net/http"
)
func GetURL(url string, r *http.Request) ([]byte, error) {
// Appengine GetURL implementation
ctx := appengine.NewContext(r)
c := urlfetch.Client(ctx)
return GetClient(url, c)
}
url_common.go:
// No build constraint: this is common code
package mypackage
import (
"net/http"
)
func GetClient(url string, c *http.Client) ([]byte, error) {
// Implementation for both local and AppEngine
resp, err := c.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return body, nil
}
You could get some clues in the golang/appengine project itself.
For instance, its remote_api/client.go provides the client for connecting remotely to a user's production application.
Im trying to serve images from google storage via google images
In order to do that i need to create blob key.
i tried several ways togenerate the key but got errors
loc := fmt.Sprintf("/gs/%s/%s", BUCKET, s)
applog.Infof(appCtx, "%s", loc)
bkey, err := blobstore.BlobKeyForFile(appCtx, loc)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
opt := &image.ServingURLOptions{}
u, err := image.ServingURL(appCtx, bkey, opt)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
// i tried with file extension to
/gs/bucktname/CXlvJUKiTmo9joe6
OBJECT_NOT_FOUND
gs://bucktname/rUAJOYKQbORzOYvs
"description": "API error 6 (images: INVALID_BLOB_KEY)"
gs:/bucktname/MlY77iFNbBca2KCA
"description": "API error 6 (images: INVALID_BLOB_KEY)"
What is the correct path ?
Dose the images cached and behind cdn ?
I got this error as well (among others) and here are the fixes which worked for me:
When saving the file to Google Cloud Storage you need to make it public like so: writer.ACL = []storage.ACLRule{{Entity: storage.AllUsers, Role: storage.RoleReader}}
The correct path is fmt.Sprintf("/gs/%s/%s", bucketName, objectName) where objectName is the path to the file within the bucket.
You have to make the bucket public. How to do that is described in here.
I am trying to upload a file in my GAE app. How do I the upload of a file in Google App Engine using Go and using the r.FormValue()?
You have to go through the Blobstore Go API Overview to get an idea and there is a full example on how could you store & serve user data on Google App Engine using Go.
I would suggest you to do that example in a completely separate application, so you'll be able to experiment with it for a while before trying to integrate it to your already existing one.
I managed to solve my problem by using the middle return param, "other". These code below are inside the upload handler
blobs, other, err := blobstore.ParseUpload(r)
Then assign corresponding formkey
file := blobs["file"]
**name := other["name"]** //name is a form field
**description := other["description"]** //descriptionis a form field
And use it like this in my struct value assignment
newData := data{
Name: **string(name[0])**,
Description: **string(description[0])**,
Image: string(file[0].BlobKey),
}
datastore.Put(c, datastore.NewIncompleteKey(c, "data", nil), &newData )
Not 100% sure this is the right thing but this solves my problem and it is now uploading the image to blobstore and saving other data and blobkey to datastore.
Hope this could help others too.
I have tried the full example from here https://developers.google.com/appengine/docs/go/blobstore/overview, and it worked fine doing the upload in blobstore and serving it.
But inserting extra post values to be saved somewhere in the datastore erases the values of "r.FormValue() "? Please refer to the code below
func handleUpload(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
//tried to put the saving in the datastore here, it saves as expected with correct values but would raised a server error.
blobs, _, err := blobstore.ParseUpload(r)
if err != nil {
serveError(c, w, err)
return
}
file := blobs["file"]
if len(file) == 0 {
c.Errorf("no file uploaded")
http.Redirect(w, r, "/", http.StatusFound)
return
}
// a new row is inserted but no values in column name and description
newData:= data{
Name: r.FormValue("name"), //this is always blank
Description: r.FormValue("description"), //this is always blank
}
datastore.Put(c, datastore.NewIncompleteKey(c, "Data", nil), &newData)
//the image is displayed as expected
http.Redirect(w, r, "/serve/?blobKey="+string(file[0].BlobKey), http.StatusFound)
}
Is it not possible to combine the upload with regular data? How come the values of r.FormValue() seems to disappear except for the file (input file type)? Even if I would have to force upload first before associating the blobkey, as the result of the upload, to other data, it would not be possible since I could not pass any r.FormValue() to the upload handler(which like I said becomes empty, or would raised an error when accessed prior the blobs, _, err := blobstore.ParseUpload(r) statement). I hope someone could help me solve this problem. Thank you!
In addition to using the Blobstore API, you can just use the Request.FormFile() method to get the file upload content. Use net\http package documentation for additional help.
Using the Request directly allows you to skip setting up an blobstore.UploadUrl() before handling the upload POST message.
A simple example would be:
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// Create an App Engine context.
c := appengine.NewContext(r)
// use FormFile()
f, _, err := r.FormFile("file")
if err != nil {
c.Errorf("FormFile error: %v", err)
return
}
defer f.Close()
// do something with the file here
c.Infof("Hey!!! got a file: %v", f)
}