According to the documentation I can use log.Errorf(ctx, "whatever: %v", err)
Somehow that isn't working for me. I have ctx declared as an appengine.Context, as in the documentation example, but Go is complaining that I need to pass a net/context instead!? I would assume that AppEngine would ONLY accept an AppEngine context, and nothing else.
AppEngine log expects context.Context created by appengine.NewContext() or it's derivatives.
Related
I cannot get the appengine taskqueue to accept any context I throw at it:
import (
"context"
"google.golang.org/appengine"
"google.golang.org/appengine/taskqueue"
)
/* snip */
ctx:= context.Background()
task := taskqueue.NewPOSTTask("/b/mytask", params)
_, err = taskqueue.Add(ctx, task, "")
if err != nil {
return fmt.Errorf("adding background task with path %s: %v", task.Path, err)
}
I'm calling appengine.Main() in my main.go main func as stated by the go111 migration docs (But this line is missing in go112 migration docs so I'm not sure it's required).
I've tried:
context.Background()
request.Context()
appengine.NewContext(r)
appengine.BackgroundContext()
context.TODO()
All result in error:
not an App Engine context
except for appengine.BackgroundContext() which gets:
service bridge HTTP failed: Post
http://appengine.googleapis.internal:10001/rpc_http: dial tcp
169.254.169.253:10001: i/o timeout
I experienced the same problems when migrating a GAE standard project from go19 to go112 in order to use go modules. In addition I got a lot of "502 bad gateway" messages.
Replacing http.ListenAndServe() in main() with appengine.Main() fixed the context problem. Moving to go111 instead of 112 took care of the other issue. The docs and examples are not very clear on this.
The documentation for migration to 1.12 states:
Use Cloud Tasks to enqueue tasks from Go 1.12 using the cloudtasks package. You can use any App Engine service as the target of an App Engine task.
But the cloudtasks package documentation (as at today’s date) is clearly marked as beta and unstable. So the answer here is probably. This feature is unsupported.
That said, I’m using it in production under go111 without any serious issue that I have noticed so far.
You're seeing internal.flushLog: Flush RPC: service bridge HTTP failed because you have appengine.Main() or other appengine lib calls while trying to run a Go 1.12+ runtime. (My guess is that the older runtime had to call into some Google-internal accounting infrastructure and that's not available for the 1.12 "next gen" systems.)
The solution isn't to downgrade your Go versions -- you're missing a ton of performance and security improvements doing that, and you can't take advantage of the new hardware -- the solution is to remove all the calls to the appengine lib and use GCP's cloud libraries instead (see https://godoc.org/cloud.google.com/go)
I'm using appengine and golang to develop simple RESTful APIs. The code works fine when I start service using goapp serve, and I started writing the unit test functions to test the API endpoints, and I'm struck here with the panic error appengine: NewContext passed an unknown http.Request. I'm getting this error when i run goapp test.
It looks like for some reason, i'm not able to pass the request that I created and pass it to appengine.NewContext()
Below is the snippet of the code..
body := strings.NewReader("")
request, err := http.NewRequest("GET", "endpoint url", body) //inst.NewRequest("GET", goalUrl, body) //
if err != nil {
t.Error(err)
}
t.Log(request)
c := appengine.NewContext(request) // ERROR: appengine: NewContext passed an unknown http.Request
I have created a simple reproducible code. Can you help me with this? or Does anyone have their golang API project on appengine, and have unit test functions to test the endpoints, i'll like to take a look at their code...
Here is the gitlab issue that I posted, which has all the required details of the issue along with examples and detailed error message. Thank for all your help.
Apologies for the delay in response. I was working on the solution myself
Here is how I solved the go appengine unit tests issue: the complete solution code can be found on this github repo branch.
In the *_test.go file, I have used the following.
httptest.NewServer(..) to create a new instance of the test server. also used to capture the base url of the service which is used in preparing the request object.
aetest.NewContext() to create a new context for testing purpose
http.NewRequest(..) to create the new request
gorilla's context.Set(..) to assign key ("Context"), value (context created in above step) to the request created above
httptest.NewRecorder() a new recorder to save the results
http.Handler.ServeHTTP(..) passing the recorded and request. used to make the API request
For each API handler code, instead of directly creating the new context using appengine.NewContext, I have code described in the below sudocode
using the gorilla's context.GetOk(..), check
IF the received request object has the key "Context"
THEN using value of that key as the context
ELSE derive context using appengine.NewContext(r)
I have wrapped the reusable functionality into a separate library called aeunittest and used it in my code.
With this setup, i'm, able to run the goapp test to fire the unit tests. see the solved code for complete details of the solution.
Here is the blogpost that helped me find this solution. Thanks a lot Mark. TESTING GO HTTP HANDLERS IN GOOGLE APP ENGINE WITH MUX AND HIGHER ORDER FUNCTIONS.
NewContext derives a context from in flight HTTP requests, i.e. ones that have been registered with it.
Since you are creating a new request, unknown to the internal appengine package, it's panicking.
That package also provides a RegisterTestRequest function, but it's not mentioned in the main documentation so YMMV. I have no experience with this.
I cannot say exactly since when we have this problem, since our project is still in development and we only periodically deploy to test things.
In our most recent test the PubSub authentication does not seem to work anymore. When subscribing to a topic, the following error comes up in the logs:
21:10:29.577 Error subscribing to '123456'-topic: googleapi: Error 401: The request does not have valid authentication credentials., unauthorized
This error is new. It worked fine before. We cannot trace it back to any code change on our part.
This error is also not occuring on the devappserver with the PubSub-Emulator.
The subscriber is in the same project as the PubSub-Service. Again, everything here is happening in one project.
Here is some insight into what we are doing:
client, err := pubsub.NewClient(ctx, "lol123", option.WithTokenSource(google.AppEngineTokenSource(ctx, pubsub.ScopePubSub)), option.WithHTTPClient(urlfetch.Client(ctx)))
[...] more code [...]
_, err := client.NewSubscription(ctx, subname, topic, 0, &pubsub.PushConfig{
Endpoint: endpoint,
})
Pretty standard stuff.
Here are things we tried to play with:
Adding a second scope:
option.WithTokenSource(google.AppEngineTokenSource(ctx, pubsub.ScopePubSub, pubsub.ScopeCloudPlatform)
Using the default tokensource:
tokenSource, _ := google.DefaultTokenSource(ctx, pubsub.ScopePubSub)
option.WithTokenSource(tokenSource)
Adding an endpoint, in case https://experimental.pubsub.googleapis.com/ is somehow used:
option.WithEndpoint("https://pubsub.googleapis.com/")
Also, we switched from the previous cloud.WithTokenSource(...)-stuff to the new option.WithTokenSource(...)-libraries. This migration to the new Cloud-Libraries worked well. However, they did not help with the permission problem.
We also played around with every imaginable combination of Service Accounts and Permissions. E.g., the default Appengine Service Account has been added as Owner to all PubSub-Topics. Remember, that this error is new and it worked before and we cannot trace it back to changes on our part.
Any ideas and suggestions? We'll try pretty much anything at this point. ;-)
I'm using a library (gotwilio) that makes http requests to a server using a http.Client object:
client := &http.Client{}
However this fails on appengine with the following error message:
http.DefaultTransport and http.DefaultClient are not available in App Engine
This makes sense. However since the lib is using these directly there is no way for me to tell the library to use the urlfetch based transport that appengine provides.
Is there a way to make &http.Client{} always return a client that works with urlfetch.... In java this is also possible on appengine. (URLConnection just works)
Sadly no, OTOH you can just copy the library, change func (twilio *Twilio) post in github.com/sfreiberg/gotwilio/gotwilio.go and use your local copy until it's implemented properly.
I'm writing a Go application to run on App Engine's Go runtime.
I notice that pretty much any operation which uses an App Engine service (such as Datastore, Mail, or even Capabilities) requires that you pass it an instance of appengine.Context which must be retrieved using the function appengine.NewContext(req *http.Request) Context.
While I am writing this app for App Engine, I want to be able to move it to some other platform (possibly one which doesn't support any of the App Engine API) easily and quickly if I should so choose.
So, I'm abstracting away the actual interaction with App Engine services and API's by writing little wrappers around any App-Engine-specific interaction (including request handling functions). With this approach, if I ever do wish to move to a different platform, I'll just rewrite those specific modules which tie my application to App Engine. Easy and straightforward.
The only problem is that appengine.Context object. I can't pass it down from my request handlers through my layers of logic to the modules which handle these API's without tying pretty much all of my code to App Engine. I could pass the http.Request object from which the appengine.Context object can be derived, but that would require coupling things that probably shouldn't be coupled. (I think it's best practice for none of my application to even know it's a web application except those portions specifically dedicated to handling HTTP requests.)
The first solution that sprang to mind was to just create a persistent variable in some module. Something like this:
package context
import (
"appengine"
)
var Context appengine.Context
Then, in my request handlers, I can set that variable with context.Context = appengine.NewContext(r) and in the modules that directly use App Engine services, I can fetch the context by accesing context.Context. None of the intervening code would need to know of the appengine.Context object's existence. The only problem is that "multiple requests may be handled concurrently by a given instance", which can lead to race conditions and unexpected behavior with this plan. (One request sets it, another sets it, the first one accesses it and gets the wrong appengine.Context object.)
I could in theory store the appengine.Context to datastore, but then I'd have to pass some request-specific identifier down the logic layers to the service-specific modules identifying which appengine.Context object in datastore is the one for the current request, which would again couple things I don't think should be coupled. (And, it would increase my application's datastore usage.)
I could also pass the appengine.Context object down the whole logic chain with the type interface{} the whole way and have any module which doesn't need the appengine.Context object ignore it. That would avoid tying most of my application to anything specific. That also seems highly messy, however.
So, I'm at a bit of a loss how to cleanly ensure the App-Engine-specific modules which need the appengine.Context object can get it. Hopefully you folks can give me a solution I have yet to think of myself.
Thanks in advance!
This is tricky because your self-imposed scoping rule (which is a sensible one) means not passing a Context instance around, and there is nothing similar to Java's ThreadLocal to achieve the same ends by sneaky means. That's actually a Good Thing, really.
Context combines logging support (easy) with a Call to appengine services (not easy). There are I think ten appengine functions that need a Context. I can't see any clean solution other than wrapping all of these behind your own facade.
There is one thing that can help you - you can include a configuration file with your app that indicates whether it's in GAE or otherwise, using a flag of some sort. Your global boolean need only store this flag (not a shared context). Your facade functions can then consult this flag when deciding whether to use NewContext(r) to obtain the Context to access GAE services, or use a lookalike structure to access your own substitute services.
Edit: As a final remark, when you solve this may I invite you to share how you did it, possibly even with an open-source project? Cheeky of me to ask, but if you don't ask... ;-)
I (hopefully) solved this issue by wrapping my request handlers (in this example one called "realHandler") like this:
http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
ds := NewDataStore(r)
realHandler(w, r, ds)
})
NewDataStore creates a DataStore, which is a simple wrapper that abstracts the GAE datastore. It has an unexposed field where it stores the context:
type DataStore struct {
c appengine.Context
}
func NewDataStore(req *http.Request) *DataStore {
return &DataStore{appengine.NewContext(req)}
}
Within the request handler, I can just access the abstracted datastore when I need it without worrying about the GAE context, which has already been set:
func realHandler(w http.ResponseWriter, req *http.Request, db *DataStore) {
var s SomeStruct{}
key, err := db.Add("Structs", &s)
...
}
Particularly in case of Datastore you should be able to reuse the same appengine.Context among different requsts. I haven't tried to do it myself but that's the way an alternative API for Datastore called goon works:
Goon differs from the datastore package in various ways: it remembers the appengine Context, which need only be specified once at creation time
The fact that storage should depend on HTTP request sounds ridiculous by the way. I don't think Datastore depends on a particular request in the usual sense. Most probably, it's needed to identify a particular Google App Engine application which obviously stays the same from request to request. My speculations are based on quick skim over the source code of google.golang.org/appengine.
You may do similar observations in regard of the other Google App Engine APIs. Of cause all the details could be implementation specific and I'd performed more deep research before actually using those observations in a real application.
It's worth to note that Go team introduced a golang.org/x/net/context package.
Later on the context was made available in Managed VMs, the repo is here. The documentation states:
This repository supports the Go runtime on App Engine, including both classic App Engine and Managed VMs. It provides APIs for interacting with App Engine services. Its canonical import path is google.golang.org/appengine.
What it means is that you could easily write another packages out of dev environment depending on appengine.
In particular it becomes very easy to wrap around packages like appengine/log (trivial log wrapper example).
But even more important this allows one to create handlers in a form:
func CoolHandler(context.Context, http.ResponseWriter, *http.Request)
There's an article about a context package on Go blog here. I wrote about using context here. If you decide to use the handler with context passing around it's good to create context for all requrests in one place. You can do it by using non standard requrest router like github.com/orian/wctx.
I handled this by wrapping appengine.NewContext behind an interface, and provide different implementations through different packages. That way, I don't have to link GAE into any binaries where it isn't used:
type Context interface {
GetHTTPContext(r *http.Request) context.Context
}
I provide a method for subpackages to register themselves when imported for side effects, sort of database/sql-style:
var _context Context
func Register(c Context) {
_context = c // Nil checking, double registration checking omitted for brevity
}
I start with a default implementation for vanilla, non-GAE binaries, that simply grabs the existing context:
var _context Context = &defaultContext{} // use this by default
type defaultContext struct {}
func (d *defaultContext) GetHTTPContext(r *http.Request) context.Context {
return r.Context()
}
Then I put an App Engine implementation in a package mything/context/appengine:
import(
ctx "mything/context"
)
type aecontext struct {}
func (a *aecontext) GetHTTPContext(r *http.Request) context.Context {
return appengine.NewContext(r)
}
func init() {
ctx.Register(&aecontext{})
}
Then my GAE binary can pull in the subpackage, which registers itself in init:
import(
_ "mything/context/appengine"
)
And my app code uses GetHTTPContext(r) to get an appropriate context to pass into dependencies.