Im trying to serve images from google storage via google images
In order to do that i need to create blob key.
i tried several ways togenerate the key but got errors
loc := fmt.Sprintf("/gs/%s/%s", BUCKET, s)
applog.Infof(appCtx, "%s", loc)
bkey, err := blobstore.BlobKeyForFile(appCtx, loc)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
opt := &image.ServingURLOptions{}
u, err := image.ServingURL(appCtx, bkey, opt)
if err != nil {
gc.JSON(500, model.GenericResponse{500, err.Error()})
return
}
// i tried with file extension to
/gs/bucktname/CXlvJUKiTmo9joe6
OBJECT_NOT_FOUND
gs://bucktname/rUAJOYKQbORzOYvs
"description": "API error 6 (images: INVALID_BLOB_KEY)"
gs:/bucktname/MlY77iFNbBca2KCA
"description": "API error 6 (images: INVALID_BLOB_KEY)"
What is the correct path ?
Dose the images cached and behind cdn ?
I got this error as well (among others) and here are the fixes which worked for me:
When saving the file to Google Cloud Storage you need to make it public like so: writer.ACL = []storage.ACLRule{{Entity: storage.AllUsers, Role: storage.RoleReader}}
The correct path is fmt.Sprintf("/gs/%s/%s", bucketName, objectName) where objectName is the path to the file within the bucket.
You have to make the bucket public. How to do that is described in here.
Related
I am using Golang(gin-gonic framework) for my backend. I have developed below rest API to receive images or files from the frontend applications. I am saving files in this path "/home/user/Desktop/files/"+dt.String()+"_"+filepath.Base(file.Filename). Till now everything fine, I can see the saved files.
Now I am planning to return these file paths to frontend via another rest API (Eg: API: /getImageUrl, Response: https://via.placeholder.com/600/771796) so that users can download files. But I am not sure how to return these file path URLs to frontend. Can somebody guide me on how to develop this rest API?
requests.POST("/upload", func(c *gin.Context) {
// Source
file, err := c.FormFile("file")
if err != nil {
c.String(http.StatusBadRequest, fmt.Sprintf("get form err: %s", err.Error()))
return
}
dt := time.Now()
filename := "/home/user/Desktop/files/"+dt.String()+"_"+filepath.Base(file.Filename)
if err := c.SaveUploadedFile(file, filename); err != nil {
c.String(http.StatusBadRequest, fmt.Sprintf("upload file err: %s", err.Error()))
return
}
c.JSON(200, gin.H{"message": "Files Uploaded Successfully"})
})
First, you should store the image name in the database.
To serve the static assets (image file) in gin you have to define a static route.
router := gin.Default()
router.Static("/image", "./path-to-image-dir")
file path will look something like that
http://myapp.com/image/filename.jpg
but i will recommend you to use a custom header for serving images content using bind-header method that way you will have more control. you can read more about it in the gin documentation.
https://github.com/gin-gonic/gin#bind-header
I'm trying to create a signed URL to download a file in Google Cloud Storage, from an App Engine app written in Go. There's a nifty signing method in App Engine which I'm using, which [in theory] allows me to avoid putting a private key in my app. However, the URLs don't appear to work: I always get a 403 "SignatureDoesNotMatch" error. Any ideas?
Here's the code:
func createDownloadURL(c context.Context, resource string, validUntil time.Time, bucket string) (string, error) {
serviceAccount, err := appengine.ServiceAccount(c)
if err != nil {
return "", err
}
// Build up the string to sign.
validUntilString := strconv.FormatInt(validUntil.Unix(), 10)
toSign := []string{
"GET", // http verb (required)
"", // content MD5 hash (optional)
"", // content type (optional)
validUntilString, // expiration (required)
resource, // resource (required)
}
// Sign it.
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\n")))
if err != nil {
return "", err
}
signedString := base64.StdEncoding.EncodeToString(signedBytes)
// Build and return the URL.
arguments := url.Values{
"GoogleAccessId": {serviceAccount},
"Expires": {validUntilString},
"Signature": {signedString},
}
return fmt.Sprintf("https://storage.googleapis.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
}
Solved. There were 2 problems with my code.
I forgot to include the bucket name when building up toSign. Fix:
fmt.Sprintf("/%s/%s", bucket, resource), // resource (required)
This returned an AccessDenied error -- progress!
The second mistake was using the XML API storage.googleapis.com instead of the authenticated browser endpoint storage.cloud.google.com. Fix:
return fmt.Sprintf("https://storage.cloud.google.com/%s/%s?%s", bucket, resource, arguments.Encode()), nil
This works.
StringToSign may require uninterpreted newlines. Could you give this a try:
_, signedBytes, err := appengine.SignBytes(c, []byte(strings.Join(toSign, "\\n"))) // escaped newline
Writing a function that signs URLs is tricky, since due to the nature of encryption it's very difficult to tell what's wrong when things don't work. You may find it easier to use a library like gcloud-golang, which has a SignedURL method.
New question: Everything I can find on Google suggests that the CloudSQL connector should be working, with the exception of the source code itself (and importing the SDK's cloudsql doesn't work). Is there some place I can find this updated library, if it exists?
---EDIT: I have my answer: it seems the problem is with CloudSQL itself rather than with the driver or something similar.---
I'm trying to access CloudSQL from a Google App Engine Go program.
I've tried both go-sql-driver/mysql and ziutek/mymysql, using Go 1.2.1 and Go 1.4.2. I've tried both the go get version of go-sql-driver and cloned it directly from Github. I've tried both the App Engine installer and the archive.
Every time I try to access a database with my application, the resulting webpage states:
cloudsql: not supported in dev yet
I've seen the other similar question here, tried everything noted there, none of it worked.
The code in question:
import (
_ "github.com/go-sql-driver/mysql"
_ "appengine/cloudsql"
"database/sql"
"net/http"
)
func adminLogin(w http.ResponseWriter, r *http.Request) {
username := formatter(r.FormValue("username"))
password := formatter(r.FormValue("password"))
db, err := sql.Open("mysql",
"root:password#cloudsql(ws-virtual-classroom:database)/logins") // And all the variations on that string I could think of...
defer db.Close()
if err != nil {
log.Print(err)
} else {
rows, err := db.Query("SELECT username FROM admin_logins WHERE username=? AND password=? LIMIT 1", username, password)
defer rows.Close()
if err != nil {
log.Print(err)
} else {
var user string
for rows.Next() {
err = rows.Scan(&user)
if err != nil {
log.Print(err)
} else {
makeCookie(w, r, user, true, true)
}
}
}
}
teachersHome(w, r)
}
The result (this is displayed in my browser after I submit the login form):
the runtime process gave a bad HTTP response: ''
2015/05/17 01:53:06 cloudsql: not supported in dev yet
2015/05/17 01:53:06 http: panic serving 127.0.0.1:56970: runtime error: invalid memory address or nil pointer dereference
goroutine 12 [running]:
net/http.func·011()
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1130 +0xbb
database/sql.(*Rows).Close(0x0, 0x0, 0x0)
/tmp/appengine/go_appengine/goroot/src/database/sql/sql.go:1659 +0x31
main57750.adminLogin(0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
main.go:208 +0x25a
net/http.HandlerFunc.ServeHTTP(0x927c78, 0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1265 +0x41
github.com/gorilla/mux.(*Router).ServeHTTP(0xc20800c730, 0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/home/daniel/go/src/github.com/gorilla/mux/mux.go:98 +0x297
net/http.(*ServeMux).ServeHTTP(0xc20803a6f0, 0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1541 +0x17d
appengine_internal.handleFilteredHTTP(0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/tmp/appengine/go_appengine/goroot/src/appengine_internal/api_dev.go:98 +0x413
net/http.HandlerFunc.ServeHTTP(0x927248, 0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1265 +0x41
net/http.serverHandler.ServeHTTP(0xc208042540, 0x7f76a72ef5d8, 0xc208045860, 0xc2080c4820)
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080457c0)
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
/tmp/appengine/go_appengine/goroot/src/net/http/server.go:1751 +0x35e
This is when running it with goapp serve. If I deploy it I seem to have the same problem--the resulting page is blank rather than showing that text, but the log error messages are the same.
The updated library is here: https://godoc.org/google.golang.org/appengine. It doesn't have CloudSQL though. Perhaps you should just use a regular mysql connection locally:
func dialSQL() (*sql.DB, error) {
if appengine.IsDevAppServer() {
// or sql.Open("mysql", "user-name:password#ip-address-of-google-cloud-sql-instance/dbname")
return sql.Open("mysql", "user:password#/dbname")
}
return sql.Open("mysql", "cloudsql:my-instance*dbname/user/passwd")
}
Using a local database is usually the best option because connecting to a live database can be super dangerous. For example it's all too easy to accidentally run your tests against your production database and drop everything.
Nevertheless, Google has instructions for connecting to CloudSQL here: https://cloud.google.com/sql/docs/introduction. The instructions for 3rd party tools will work for Go too.
I don't think cloudsql does anything? The source just looks like it returns that error no matter what lol
https://code.google.com/p/appengine-go/source/browse/appengine/cloudsql/cloudsql.go
// Dial connects to the named Cloud SQL instance.
func Dial(instance string) (net.Conn, error) {
return connect(instance)
}
https://code.google.com/p/appengine-go/source/browse/appengine/cloudsql/cloudsql_dev.go
func connect(instance string) (net.Conn, error) {
return nil, errors.New("cloudsql: not supported in dev yet")
}
I am trying to upload a file in my GAE app. How do I the upload of a file in Google App Engine using Go and using the r.FormValue()?
You have to go through the Blobstore Go API Overview to get an idea and there is a full example on how could you store & serve user data on Google App Engine using Go.
I would suggest you to do that example in a completely separate application, so you'll be able to experiment with it for a while before trying to integrate it to your already existing one.
I managed to solve my problem by using the middle return param, "other". These code below are inside the upload handler
blobs, other, err := blobstore.ParseUpload(r)
Then assign corresponding formkey
file := blobs["file"]
**name := other["name"]** //name is a form field
**description := other["description"]** //descriptionis a form field
And use it like this in my struct value assignment
newData := data{
Name: **string(name[0])**,
Description: **string(description[0])**,
Image: string(file[0].BlobKey),
}
datastore.Put(c, datastore.NewIncompleteKey(c, "data", nil), &newData )
Not 100% sure this is the right thing but this solves my problem and it is now uploading the image to blobstore and saving other data and blobkey to datastore.
Hope this could help others too.
I have tried the full example from here https://developers.google.com/appengine/docs/go/blobstore/overview, and it worked fine doing the upload in blobstore and serving it.
But inserting extra post values to be saved somewhere in the datastore erases the values of "r.FormValue() "? Please refer to the code below
func handleUpload(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
//tried to put the saving in the datastore here, it saves as expected with correct values but would raised a server error.
blobs, _, err := blobstore.ParseUpload(r)
if err != nil {
serveError(c, w, err)
return
}
file := blobs["file"]
if len(file) == 0 {
c.Errorf("no file uploaded")
http.Redirect(w, r, "/", http.StatusFound)
return
}
// a new row is inserted but no values in column name and description
newData:= data{
Name: r.FormValue("name"), //this is always blank
Description: r.FormValue("description"), //this is always blank
}
datastore.Put(c, datastore.NewIncompleteKey(c, "Data", nil), &newData)
//the image is displayed as expected
http.Redirect(w, r, "/serve/?blobKey="+string(file[0].BlobKey), http.StatusFound)
}
Is it not possible to combine the upload with regular data? How come the values of r.FormValue() seems to disappear except for the file (input file type)? Even if I would have to force upload first before associating the blobkey, as the result of the upload, to other data, it would not be possible since I could not pass any r.FormValue() to the upload handler(which like I said becomes empty, or would raised an error when accessed prior the blobs, _, err := blobstore.ParseUpload(r) statement). I hope someone could help me solve this problem. Thank you!
In addition to using the Blobstore API, you can just use the Request.FormFile() method to get the file upload content. Use net\http package documentation for additional help.
Using the Request directly allows you to skip setting up an blobstore.UploadUrl() before handling the upload POST message.
A simple example would be:
func uploadHandler(w http.ResponseWriter, r *http.Request) {
// Create an App Engine context.
c := appengine.NewContext(r)
// use FormFile()
f, _, err := r.FormFile("file")
if err != nil {
c.Errorf("FormFile error: %v", err)
return
}
defer f.Close()
// do something with the file here
c.Infof("Hey!!! got a file: %v", f)
}
I'm writing an app on the Google App Engine, and I have been able to write code that runs locally and puts data into the local datastore. However, when I deploy the code to the google server, no data is put into the datastore. My primary indicators that nothing is getting put into the datastore are, one, when I go to the Developer Console, I'm told there are no entries, and two, when I run the page that gets data, nothing is returned.
Do I need to define new Kinds in the app.yaml or developer console before they will work on the production datastore?
Here is my put and get methods from the data store. Again, works in the testing environment and does not work in production, everything compiles just fine.
package tweetdata
import (
"net/url"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/log"
"github.com/ChimeraCoder/anaconda"
)
const linkTweetKind string = "LinkTweet"
const tweetKey string = "Tweets"
const tweetKeyID string = "default_tweetstore"
//LinkTweet contains the address extracted from a tweet and the original tweet
type LinkTweet struct {
Address *url.URL
Tweet anaconda.Tweet
}
//StoreTweet is a struct used for storing a tweet in the datastore
type StoreTweet struct {
Address string
Text string
TweetID int64
CreatedTime time.Time
Retweets int
Favorites int
}
//TweetScore is a struct that shows the relative score of an address based on
// it's populatrity
type TweetScore struct {
Address *url.URL
score int
lastActive time.Time
}
//WriteLinkTweet writes a given Tweet to the datastore
func WriteLinkTweet(tweet LinkTweet, c context.Context) error {
log.Infof(c, "Putting Tweet into datastore: %v", tweet.Tweet.Id)
key := datastore.NewIncompleteKey(c, linkTweetKind, getTweetKey(c))
created, _ := tweet.Tweet.CreatedAtTime()
store := &StoreTweet{Address: tweet.Address.String(),
Text: tweet.Tweet.Text,
TweetID: tweet.Tweet.Id,
CreatedTime: created,
Retweets: tweet.Tweet.RetweetCount,
Favorites: tweet.Tweet.FavoriteCount,
}
err := datastore.RunInTransaction(c, func(c context.Context) error {
_, err := datastore.Put(c, key, store)
log.Errorf(c, "Failed to write LinkTweet to datastore. %v", err.Error())
return err
}, nil)
return err
}
//GetAllNewTweets queries the datastore and gets all tweets created since the last
// time given
func GetAllNewTweets(since time.Time, c context.Context) []StoreTweet {
q := datastore.NewQuery(linkTweetKind).Ancestor(getTweetKey(c)).Filter("CreatedTime >=", since)
out := make([]StoreTweet, 0, 15)
q.GetAll(c, &out)
return out
}
// guestbookKey returns the key used for all guestbook entries.
func getTweetKey(c context.Context) *datastore.Key {
// The string "default_guestbook" here could be varied to have multiple guestbooks.
return datastore.NewKey(c, tweetKey, tweetKeyID, 0, nil)
}
You're seeing the (often initially confusing) effect of "eventual consistency". Entities that are written outside of a transaction take a while to be visible to queries.
https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ explains the problem. https://cloud.google.com/appengine/docs/go/datastore/transactions is specific to go.
Are you using datastore:,noindex on any fields in the StoreTweet struct? In order to run any query against any fields, the datastore must first index the fields you'll be querying. So not indexing those fields will cause the query to return nil without any errors. Does your dev server automatically create an index.yaml file in your project after running your queries for the first time? The production datastore uses this file to decide which fields should be indexed, all fields not indexed won't be able to be queried. From the information you've provided, this is all I can think of that would be causing your queries to return empty.
In order to debug this further, I would browse your production datastore and make sure your data is structured properly and the values are set to what you are expecting them to be.