I have a problem on issuing HTTP requests in appengine since it does not support http.Client. I'm creating a slackbot and want to create a delayed response. The logic is once I received slack's POST request, I'm going to respond successfully and spin a goroutine calling an external API and after I finish with it, create a new request to slack.
Seems easy enough, but the problem I encountered is when using appengine's urlfetch and NewContext, because NewContext takes in an *http.Request as an argument but since I'm responding immediately to the first slack request, the response Body closes before I can use it to issue the response to an external API. Is there an alternative?
code:
func Twit(w http.ResponseWriter, r *http.Request) {
defaultResponse := &SlashResponse{ResponseType: "ephemeral", Text: "success"}
// prepare to make a slack delayed response
responseURL := r.FormValue("response_url")
go sendDelayResponse(responseURL, r)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
json.NewEncoder(w).Encode(defaultResponse)
}
func sendDelayResponse(url string, r *http.Request) {
response := &SlashResponse{ResponseType: "in_channel", Text: twit.TweetTCL(r)}
b, _ := json.Marshal(response)
// send request to slack using given response_url
ctx := appengine.NewContext(r)
client := urlfetch.Client(ctx)
req, _ := http.NewRequest("POST", url, bytes.NewBuffer(b))
req.Header.Add("Content-Type", "application/json")
resp, err := client.Do(req)
defer resp.Body.Close()
if err != nil {
log.Println(err)
} else {
log.Println("success")
}
}
Use the delay package to execute functions outside the scope of a request.
You can also use the lower-level taskqueue package. The delay package is layer above the taskqueue package.
Declare package level variable with the delay function:
var laterFunc("dr", func(ctx context.Context, url string) {
response := &SlashResponse{ResponseType: "in_channel", Text: twit.TweetTCL(r)}
b, _ := json.Marshal(response)
// send request to slack using given response_url
client := urlfetch.Client(ctx)
req, _ := http.NewRequest("POST", url, bytes.NewBuffer(b))
req.Header.Add("Content-Type", "application/json")
resp, err := client.Do(req)
defer resp.Body.Close()
if err != nil {
log.Println(err)
} else {
log.Println("success")
}
})
Call it like this:
laterfunc.Call(appengine.NewContext(r), responseURL)
Related
Not so much a question as help for others having this problem. Took a fair amount of beating my head against a wall to make this work. (as much I as love golang, you do have think a little differently) - This will also work as a generic way to do any sort of post to an outside source in AppEngine.
Here is the function I am using to post simple messages to a slack channel via webhook. (assumes you know how to set up a webhook in slack - very easy to do - https://get.slack.help/hc/en-us/articles/115005265063-Incoming-WebHooks-for-Slack ) - NOTE: while there are a fair # of additional parameters you can pass in the json message (see link above) simple things like email addresses and image urls / web addresses will be automatically parsed by slack if passed in the 'text' parameter.
import (
"bytes"
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
"net/http"
)
func postSlackBetaSignup(req *http.Request, msg string) string {
ctx := appengine.NewContext(req);
request := urlfetch.Client(ctx);
data := []byte("{'text': '" + msg + "'}");
body := bytes.NewReader(data);
resp, err := request.Post("https://hooks.slack.com/services/<<<YOUR WEBHOOK HERE>>>", "application/json", body);
if err != nil {
return err.Error();
} else {
return resp.Status;
}
}
import (
"bytes"
"google.golang.org/appengine"
"google.golang.org/appengine/urlfetch"
"net/http"
)
func postSlackBetaSignup(req *http.Request, msg string) string {
ctx := appengine.NewContext(req);
request := urlfetch.Client(ctx);
data := []byte("{'text': '" + msg + "'}");
body := bytes.NewReader(data);
resp, err := request.Post("https://hooks.slack.com/services/<<<YOUR WEBHOOK HERE>>>", "application/json", body);
if err != nil {
return err.Error();
} else {
return resp.Status;
}
}
So I am trying to use https://github.com/mongodb/mongo-go-driver to connect to a mongo database in golang.
Here is my connection handler:
var DB *mongo.Database
func CreateConnectionHandler()(*mongo.Database, error){
fmt.Println("inside createConnection in database package")
godotenv.Load()
fmt.Println("in CreateConnectionHandler and SERVER_CONFIG: ")
fmt.Println(os.Getenv("SERVER_CONFIG"))
uri:=""
if os.Getenv("SERVER_CONFIG")=="kubernetes"{
fmt.Println("inside kubernetes db config")
uri = "mongodb://patientplatypus:SUPERSECRETPASSDOOT#
mongo-release-mongodb.default.svc.cluster.local:27017/
platypusNEST?authMechanism=SCRAM-SHA-1"
}else if os.Getenv("SERVER_CONFIG")=="compose"{
fmt.Println("inside compose db config")
uri = "mongodb://datastore:27017"
}
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, uri)
if err != nil {
return nil, fmt.Errorf("mongo client couldn't connect: %v", err)
}
DB := client.Database("platypusNEST")
return DB, nil
}
And the error I am getting:
api | database/connection.go:29:30: cannot use uri (type
string) as type *options.ClientOptions in argument to mongo.Connect
So I tried replacing uri with the connection string like this:
client, err := mongo.Connect(ctx, "mongodb://datastore:27017")
But I still got the error.
Compare this with what is in the documentation:
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
client, err := mongo.Connect(ctx, "mongodb://localhost:27017")
And it is exactly the same! I'm really not sure why there is this error. Any ideas?
To those who come searching - the docs are out of date as of this posting, but their latest push here: https://github.com/mongodb/mongo-go-driver/commit/32946b1f8b9412a6a94e68ff789575327bb257cf has them doing this with the connect:
client, err := mongo.NewClient(options.Client().ApplyURI(uri))
You will also now need to import the options package. Happy hacking.
EDIT: thanks vcanales for finding this - you're a gentleman and a scholar.
In addition to the accepted answer - this snippet below may be improved by using an environment variable to pass in the Mongodb URL.
package main
import (
"context" //use import withs " char
"fmt"
"time"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
"go.mongodb.org/mongo-driver/mongo/readpref"
)
func ConnectMongo() {
var (
client *mongo.Client
mongoURL = "mongodb://localhost:27017"
)
// Initialize a new mongo client with options
client, err := mongo.NewClient(options.Client().ApplyURI(mongoURL))
// Connect the mongo client to the MongoDB server
ctx, _ := context.WithTimeout(context.Background(), 10*time.Second)
err = client.Connect(ctx)
// Ping MongoDB
ctx, _ = context.WithTimeout(context.Background(), 10*time.Second)
if err = client.Ping(ctx, readpref.Primary()); err != nil {
fmt.Println("could not ping to mongo db service: %v\n", err)
return
}
fmt.Println("connected to nosql database:", mongoURL)
}
func main() {
ConnectMongo()
}
More information on options and readpref respectively:
https://docs.mongodb.com/manual/reference/method/cursor.readPref/index.html
https://docs.mongodb.com/manual/core/read-preference/
Google's AppEngine service provides an eventually consistent database for storage of application data in production. For testing, Google provides a similar database that emulates the consistency characteristics of the production database. Testing requirements may vary so Google supplies command line parameters to their test server, dev_appserver.py, that modify the consistency characteristics as needed for testing.
I am using the goapp tools to run our automated test environment for my AppEngine development. goapp test is responsible for running our automated server API tests. goapp test does not appear to have a way of setting the datastore's consistency level via the command line parameters, unlike dev_appserver.py, even though goapp test launches dev_appserver.py at some point during the testing process.
Is there a way to pass command line parameters to dev_appserver.py from goapp test? If not, is there an alternative method for setting the consistency from the command line?
I presume you are using the aetest package.
If this is the case you should set the StronglyConsistentDatastore member of the aetest.Options struct accordingly.
Here is an example:
hello.go
package hello
import (
"fmt"
"net/http"
"time"
"appengine"
datastore "appengine/datastore"
)
type Employee struct {
FirstName string
LastName string
HireDate time.Time
}
func init() {
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
ctx := appengine.NewContext(r)
q := datastore.NewQuery("Employee").Filter("FirstName =", "Antonio")
var people []Employee
if _, err := q.GetAll(ctx, &people); err != nil {
fmt.Fprintf(w, "Error %v", err)
}
fmt.Fprintf(w, "Matches %v", len(people))
}
hello_test.go
package hello
import (
"time"
"testing"
"net/http/httptest"
"appengine"
"appengine/aetest"
datastore "appengine/datastore"
)
const consistency_strong = true; // set to false and the test will fail
func TestMyHandler(t *testing.T) {
options := &aetest.Options{StronglyConsistentDatastore: consistency_strong}
inst, err := aetest.NewInstance(options)
if err != nil {
t.Fatalf("Failed to create instance: %v", err)
}
defer inst.Close()
req, err := inst.NewRequest("GET", "/", nil)
if err != nil {
t.Fatalf("Failed to create req1: %v", err)
}
ctx := appengine.NewContext(req)
employee := &Employee{
FirstName: "Antonio",
LastName: "Salieri",
HireDate: time.Now(),
}
key := datastore.NewIncompleteKey(ctx, "Employee", nil)
_, err = datastore.Put(ctx, key, employee)
if err != nil {
t.Fatalf("Error setting test data: %v", err)
}
w := httptest.NewRecorder()
handler(w, req)
if w.Body.String() != "Matches 1" {
t.Fatalf("Expected 1 record got %v", w.Body)
}
}
As you have mentioned the go tools merely wrap dev_appserver.py. This means that if you are not using aetest you can always run your app with a chosen consistency policy like so:
/usr/local/go_appengine/dev_appserver.py --datastore_consistency_policy consistent .
My asana oauth redirect URL is something like
https://testapp.appspot.com/api/redirect.asana#access_token=123456789
I am not sure how to fetch access_token now.
Note : if I am changing ? to # then its working fine by using r.FormValue("access_token").
The reason why r.FormValue() does not get it is because URL parameters are separated by ? but in your URL it is not.
The # is used to separate a fragment for references, so your access_token should be in r.URL.Fragment... but it won't.
You can't test it from the browser
Fragments are not sent over to the server, fragments are for the browsers. There was an issue covering this:
net/http: document fields set in Request.URL in handler #3805
It is also included in the doc of http.Request:
For server requests the URL is parsed from the URI supplied on the Request-Line as stored in RequestURI. For most requests, fields other than Path and RawQuery will be empty. (See RFC 2616, Section 5.1.2)
Code to get it from the request
If a non-browser client does send it as part of the request path, you can use simple string operations to get the token value: it is the part after the = character. You can use strings.Index() to find the "=":
raw := r.URL.Path
if idx := strings.Index(raw, "="); idx >= 0 && idx < len(raw)-1 {
token := raw[idx+1:]
fmt.Fprintln(w, "Token:", token)
}
Output:
Token: 123456789
As an alternative solution you can also use strings.Split() to split it by "=", 2nd element will be the value of the token:
parts := strings.Split(r.URL.Path, "=")
fmt.Fprintln(w, "Parts:", parts)
if len(parts) == 2 {
fmt.Fprintln(w, "Token:", parts[1])
}
Output:
[/api/redirect.asana#access_token 123456789]
Token: 123456789
Code to test it
Here is a code using net/http to call your server that will send a path being "/api/redirect.asana#access_token=123456789", and it will print the response body to the standard output (console):
c := &http.Client{}
req, err := http.NewRequest("GET", "http://localhost:8080/", nil)
if err != nil {
panic(err)
}
req.URL.Path = "/api/redirect.asana#access_token=123456789"
resp, err := c.Do(req)
if err != nil {
panic(err)
}
defer resp.Body.Close()
io.Copy(os.Stdout, resp.Body)
I am having issues with urlfetch's timeouts on Google App Engine in Go. The app does not appear to want to take a longer timeout than about 5 seconds (it ignores a longer timeout and times out after its own time).
My code is:
var TimeoutDuration time.Duration = time.Second*30
func Call(c appengine.Context, address string, allowInvalidServerCertificate bool, method string, id interface{}, params []interface{})(map[string]interface{}, error){
data, err := json.Marshal(map[string]interface{}{
"method": method,
"id": id,
"params": params,
})
if err != nil {
return nil, err
}
req, err:=http.NewRequest("POST", address, strings.NewReader(string(data)))
if err!=nil{
return nil, err
}
tr := &urlfetch.Transport{Context: c, Deadline: TimeoutDuration, AllowInvalidServerCertificate: allowInvalidServerCertificate}
resp, err:=tr.RoundTrip(req)
if err != nil {
return nil, err
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, err
}
result := make(map[string]interface{})
err = json.Unmarshal(body, &result)
if err != nil {
return nil, err
}
return result, nil
}
No matter what I try to set TimeoutDuration to, the app times out after about 5 seconds. How prevent it from doing that? Did I make some error in my code?
You need to pass the time duration like this (otherwise it will default to the 5 sec timeout):
tr := &urlfetch.Transport{Context: c, Deadline: time.Duration(30) * time.Second}
Update Jan 2 2016:
With the new GAE golang packages (google.golang.org/appengine/*), this has changed. urlfetch no longer receives a deadline time duration in the transport.
You should now set the timeout via the new context package. For example, this is how you would set a 1 minute deadline:
func someFunc(ctx context.Context) {
ctx_with_deadline, _ := context.WithTimeout(ctx, 1*time.Minute)
client := &http.Client{
Transport: &oauth2.Transport{
Base: &urlfetch.Transport{Context: ctx_with_deadline},
},
}
Try the code below:
// createClient is urlfetch.Client with Deadline
func createClient(context appengine.Context, t time.Duration) *http.Client {
return &http.Client{
Transport: &urlfetch.Transport{
Context: context,
Deadline: t,
},
}
}
Here is how to use it.
// urlfetch
client := createClient(c, time.Second*60)
Courtesy #gosharplite
Looking at the source code of Go's appengine:
http://code.google.com/p/appengine-go/source/browse/appengine/urlfetch/urlfetch.go
and the protobuffer generated code:
http://code.google.com/p/appengine-go/source/browse/appengine_internal/urlfetch/urlfetch_service.pb.go
Looks like there should not be a problem with Duration itself.
My guess is that the whole application inside appengine timeouts after 5 seconds.
for me, this worked:
ctx_with_deadline, _ := context.WithTimeout(ctx, 15*time.Second)
client := urlfetch.Client(ctx_with_deadline)
This is had now changed with the recent updates to the library . Now the Duration of timeout/delay have to carried by the context , urlfetch.transport no more has the Deadline field in it . context.WithTimeout or context.WithDeadline is the method to use , here is the link https://godoc.org/golang.org/x/net/context#WithTimeout