I have written a number of GAE Golang applications and I'm rather unsatisfied with my current logging approach. I can't seem to find an easy way to log line numbers or even which functions a log was created in App Engine. Is there some automatic way of creating such meaningful logs? At the moment I am resulting to just manually numbering my errors, like:
c.Debugf("Backend - Upkeep error 5 - %v", err)
You need to create your own function and use runtime.Caller
Working Example: http://play.golang.org/p/CQghRzJ3x_
func debug(format string, args ...interface{}) {
_, file, line, ok := runtime.Caller(1)
if !ok {
file = "???"
line = 0
}
log.Printf(file+":"+strconv.Itoa(line)+" - "+format, args...)
}
Related
A project I'm working on depends on having a service hosted on Google App Engine parse from SendGrid. The following code is an example of what we're doing:
package sendgrid_failure
import (
"net/http"
"fmt"
"google.golang.org/appengine"
"google.golang.org/appengine/log"
)
func init() {
http.HandleFunc("/sendgrid/parse", sendGridHandler)
}
func sendGridHandler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
err := r.ParseMultipartForm(-1)
if err != nil {
log.Errorf(ctx, "Unable to parse form: %v", err)
}
fmt.Fprint(w, "Test.")
}
When SendGrid POSTs its multipart form, the console shows similar to the following:
2018/01/04 23:44:08 ERROR: Unable to parse form: open /tmp/multipart-445139883: no file writes permitted on App Engine
App Engine doesn't allow you to read/write files, but Golang appears to need it to parse. Is there an App Engine specific library to parse multipart forms, or should we be using a different method from the standard net/http library entirely? We're using the standard go runtime.
The documentation for ParseMultipartForm says:
The whole request body is parsed and up to a total of maxMemory bytes of its file parts are stored in memory, with the remainder stored on disk in temporary files.
The server attempts to write all files to disk because the application passed -1 as maxMemory. Use a value larger than the size of the files you expect to upload.
What is the GO appropriate way to create a FetchUrl/GetURL function that works from the command line and works from google app engine with its custom way to fetch a url.
I have basic code that fetches and processes some data on a URL. I want to be able to call it from code I use on my desktop, and code deployed to app engine.
Hopefully thats clear, if not please let me know and Ill clarify.
If you have some code which works both on local machine and on AppEngine environment, you have nothing to do.
If you need to do something which should or must be done differently on AppEngine, then you need to detect the environment and write different code for the different environments.
This detection and code selection is easiest done using build constraints. You can put a special comment line in the beginning of your .go file, and it may or may not be compiled and run depending on the environment.
Quoting from The Go Blog: The App Engine SDK and workspaces (GOPATH):
The App Engine SDK introduces a new build constraint term: "appengine". Files that specify
// +build appengine
will be built by the App Engine SDK and ignored by the go tool. Conversely, files that specify
// +build !appengine
are ignored by the App Engine SDK, while the go tool will happily build them.
So for example you can have 2 separate .go files, one for AppEngine and one for local (non-AppEngine) environment. Define the same function in both (with same parameter list), so no matter in which environment the code is built, the function will have one declaration. We will use this signature:
func GetURL(url string, r *http.Request) ([]byte, error)
Note that the 2nd parameter (*http.Request) is only required for the AppEngine (in order to be able to create a Context), so in the implementation for local env it is not used (can even be nil).
An elegant solution can take advantage of the http.Client type which is available in both the standard environment and in AppEngine, and which can be used to do an HTTP GET request. An http.Client value can be acquired differently on AppEngine, but once we have an http.Client value, we can proceed the same way. So we will have a common code that receives an http.Client and can do the rest.
Example implementation can look like this:
url_local.go:
// +build !appengine
package mypackage
import (
"net/http"
)
func GetURL(url string, r *http.Request) ([]byte, error) {
// Local GetURL implementation
return GetClient(url, &http.Client{})
}
url_gae.go:
// +build appengine
package mypackage
import (
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
"net/http"
)
func GetURL(url string, r *http.Request) ([]byte, error) {
// Appengine GetURL implementation
ctx := appengine.NewContext(r)
c := urlfetch.Client(ctx)
return GetClient(url, c)
}
url_common.go:
// No build constraint: this is common code
package mypackage
import (
"net/http"
)
func GetClient(url string, c *http.Client) ([]byte, error) {
// Implementation for both local and AppEngine
resp, err := c.Get(url)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
return body, nil
}
You could get some clues in the golang/appengine project itself.
For instance, its remote_api/client.go provides the client for connecting remotely to a user's production application.
I'm wondering if it's possible to log the line/column number when i log an error. The GAE logging lib seems to store only the error message but I think it would be quite valuable to get the exact location of the error/log point like most logging libraries do.
What you request is definitely possible, you just need to be very explicit about it! I recommend Andrew Gerrand's excellent article at http://blog.golang.org/error-handling-and-go for general information on the issue, including notes specific to App Engine.
That article does not specifically address stack traces, but of course you could do those yourself, via http://golang.org/pkg/runtime/#Stack .
Let's be honest and admit that Go -- being by design more of a system-programming language than of an application-focused one -- doesn't do quite as much implicit, automatic hand-holding for you, as more app-oriented languages such as the other App Engine ones -- Java, Python, PHP... [*] but, Go does give you all the tools you need to do just as little, or as much, "supporting infrastructure", as you desire to have in support of your own apps!-)
[*] hey, you don't even get automatically by-default propagating exceptions, as you do for other languages -- nay, you're responsible for catching and dealing with errors yourself, oh the horror...!-)
You can use the runtime package to trace the caller(I assume that's basically what you want). There is no GAE specific issue and it has nothing to do with the error handling.
// The depth specifies how many stack frames above
// lives the source line to be identified in the log message.
func traceCaller(depth int)(string, int){
_, file, line, ok := runtime.Caller(2 + depth)
if !ok {
file = "???"
line = 1
} else {
slash := strings.LastIndex(file, "/")
if slash >= 0 {
file = file[slash+1:]
}
}
return file, line
}
I was trying to run a profiler on some handler code up on appspot and was looking for a way to show the pstats output without writing directly to the response body (which is problematic for handlers responding with structured data such as JSON).
I decided to try writing the output to the response header, and added some js to my html-rendering handlers that could parse the header and console.log() it (turned out to be pretty simple and convenient to work with). For the non-html rendering handlers, I was thinking I might try and build a chrome extension to do something similar, but of course, whenever I'm making an ajax call to such a handler, the calling page (with the js to parse the header) can step in and handle the display.
All this looked good as I was testing on dev_appserver, but then I deployed to appspot and found that my custom header never showed up. I have similar pieces of code in this project that track the elapsed execution time of certain operations, and the custom headers I write that stuff to shows up fine.
I'm assuming there's a limit in place on appspot python runtime process that omits the header if it's greater than a certain length, but I don't see the size published anywhere on the developer site.
If there is in fact a limit, does anyone know what it is?
I cannot find any documentation about this, but the limit seems to be 497 bytes for the entire header line (including the key, the colon, the whitespace after the colon, but excluding the '\r\n' at the end of the line).
Here's a Go test handler:
func Test(c appengine.Context, w http.ResponseWriter, r *http.Request) {
l, err := strconv.ParseInt(r.URL.Query().Get("len"), 10, 64)
if err != nil {
http.Error(w, "", http.StatusBadRequest)
return
}
value := ""
for l != 0 {
value += "X"
l--
}
w.Header().Set("Set-Cookie", value)
w.Header().Set("Test-Header", value)
w.Header().Set("Very-Very-Long-Test-Header", value)
w.Header().Set(value, "Test-Value")
w.WriteHeader(http.StatusNoContent)
}
Setting the len query parameter to 469 is the largest value that does not remove the Very-Very-Long-Test-Header header. 484 is the largest that will keep the Test-Header header, and 485 is the largest that will keep the Xxxxx… header.
They all add up to 497 per line, or if you will, 495 excluding the ": ".
NOTE that this limit does not apply to incoming (request) headers.
UPDATE: The Set-Cookie header has a different limit: 4108 bytes instead of 497.
I'm writing an app on the Google App Engine, and I have been able to write code that runs locally and puts data into the local datastore. However, when I deploy the code to the google server, no data is put into the datastore. My primary indicators that nothing is getting put into the datastore are, one, when I go to the Developer Console, I'm told there are no entries, and two, when I run the page that gets data, nothing is returned.
Do I need to define new Kinds in the app.yaml or developer console before they will work on the production datastore?
Here is my put and get methods from the data store. Again, works in the testing environment and does not work in production, everything compiles just fine.
package tweetdata
import (
"net/url"
"time"
"golang.org/x/net/context"
"google.golang.org/appengine/datastore"
"google.golang.org/appengine/log"
"github.com/ChimeraCoder/anaconda"
)
const linkTweetKind string = "LinkTweet"
const tweetKey string = "Tweets"
const tweetKeyID string = "default_tweetstore"
//LinkTweet contains the address extracted from a tweet and the original tweet
type LinkTweet struct {
Address *url.URL
Tweet anaconda.Tweet
}
//StoreTweet is a struct used for storing a tweet in the datastore
type StoreTweet struct {
Address string
Text string
TweetID int64
CreatedTime time.Time
Retweets int
Favorites int
}
//TweetScore is a struct that shows the relative score of an address based on
// it's populatrity
type TweetScore struct {
Address *url.URL
score int
lastActive time.Time
}
//WriteLinkTweet writes a given Tweet to the datastore
func WriteLinkTweet(tweet LinkTweet, c context.Context) error {
log.Infof(c, "Putting Tweet into datastore: %v", tweet.Tweet.Id)
key := datastore.NewIncompleteKey(c, linkTweetKind, getTweetKey(c))
created, _ := tweet.Tweet.CreatedAtTime()
store := &StoreTweet{Address: tweet.Address.String(),
Text: tweet.Tweet.Text,
TweetID: tweet.Tweet.Id,
CreatedTime: created,
Retweets: tweet.Tweet.RetweetCount,
Favorites: tweet.Tweet.FavoriteCount,
}
err := datastore.RunInTransaction(c, func(c context.Context) error {
_, err := datastore.Put(c, key, store)
log.Errorf(c, "Failed to write LinkTweet to datastore. %v", err.Error())
return err
}, nil)
return err
}
//GetAllNewTweets queries the datastore and gets all tweets created since the last
// time given
func GetAllNewTweets(since time.Time, c context.Context) []StoreTweet {
q := datastore.NewQuery(linkTweetKind).Ancestor(getTweetKey(c)).Filter("CreatedTime >=", since)
out := make([]StoreTweet, 0, 15)
q.GetAll(c, &out)
return out
}
// guestbookKey returns the key used for all guestbook entries.
func getTweetKey(c context.Context) *datastore.Key {
// The string "default_guestbook" here could be varied to have multiple guestbooks.
return datastore.NewKey(c, tweetKey, tweetKeyID, 0, nil)
}
You're seeing the (often initially confusing) effect of "eventual consistency". Entities that are written outside of a transaction take a while to be visible to queries.
https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/ explains the problem. https://cloud.google.com/appengine/docs/go/datastore/transactions is specific to go.
Are you using datastore:,noindex on any fields in the StoreTweet struct? In order to run any query against any fields, the datastore must first index the fields you'll be querying. So not indexing those fields will cause the query to return nil without any errors. Does your dev server automatically create an index.yaml file in your project after running your queries for the first time? The production datastore uses this file to decide which fields should be indexed, all fields not indexed won't be able to be queried. From the information you've provided, this is all I can think of that would be causing your queries to return empty.
In order to debug this further, I would browse your production datastore and make sure your data is structured properly and the values are set to what you are expecting them to be.