I am trying to create an endpoint test when using appengine. Unfortunately, the tests keep failing because of the lack of a schema (and host) within the url used when creating the test *Request struct. When running appengine tests a server is spawned for that specific test that runs on a semi-random port number, which makes it seemingly impossible to define the full url to perform the test on.
The official docs on running tests like this are very sparse and only give half of an example, so I am left scratching my head on how to get this to work.
This is the error that I get from the marked line within the code snippet
Error: Received unexpected error "Post /auth: unsupported protocol scheme \"\""
Test Code
func TestEndpoints_Auth(t *testing.T) {
// input data
account := Account{
AuthProvider: "facebook",
AuthProviderId: "123345456",
}
b, _ := json.Marshal(&account)
reader := bytes.NewReader(b)
// test server
inst, err := aetest.NewInstance(nil)
if !assert.NoError(t, err) { return }
defer inst.Close()
// request
client := http.Client{}
req, err := inst.NewRequest("POST", "/auth", reader)
if !assert.NoError(t, err) { return }
req.Header.Add(AppAuthToken, "foobar")
resp, err := client.Do(req)
if !assert.NoError(t, err) { return } // <=== Where the error occurs
// tests
if !assert.Nil(t, err) { return }
assert.Equal(t, http.StatusCreated, resp.StatusCode)
}
Logs
[GIN-debug] POST /auth --> bitbucket.org/chrisolsen/chriscamp.(*endpoints).Auth-fm (5 handlers)
[GIN-debug] GET /accounts/me --> bitbucket.org/chrisolsen/chriscamp.(*endpoints).GetMyAccount-fm (7 handlers)
INFO 2016-04-22 13:23:39,278 devappserver2.py:769] Skipping SDK update check.
WARNING 2016-04-22 13:23:39,278 devappserver2.py:785] DEFAULT_VERSION_HOSTNAME will not be set correctly with --port=0
WARNING 2016-04-22 13:23:39,345 simple_search_stub.py:1126] Could not read search indexes from c:\users\chris\appdata\local\temp\appengine.testapp\search_indexes
INFO 2016-04-22 13:23:39,354 api_server.py:205] Starting API server at: http://localhost:54461
INFO 2016-04-22 13:23:41,043 dispatcher.py:197] Starting module "default" running at: http://localhost:54462
INFO 2016-04-22 13:23:41,046 admin_server.py:116] Starting admin server at: http://localhost:54466
I was really hoping to perform api blackbox tests, but that seems to be undoable with appengine. Instead I am now performing the tests on the endpoint directly.
req, _ := inst.NewRequest("POST", "/auth", reader)
req.Header.Add(AppAuthToken, "foobar")
resp := httptest.NewRecorder()
handlePostAuth(resp, req)
assert.Equal(t, http.StatusCreated, resp.Code)
Related
I'm using Google App Engine Standard Environment with the Go 1.11 runtime. The documentation for Go 1.11 says "Write your application logs using stdout for output and stderr for errors". The migration from Go 1.9 guide also suggests not calling the Google Cloud Logging library directly but instead logging via stdout.
https://cloud.google.com/appengine/docs/standard/go111/writing-application-logs
With this in mind, I've written a small HTTP Service (code below) to experiment logging to Stackdriver using JSON output to stdout.
When I print plain text messages they appear as expected in the Logs Viewer panel under textPayload. When I pass a JSON string they appear under jsonPayload. So far, so good.
So, I added a severity field to the output string and Stackdriver Log Viewer successfully categorizes the message according to the levelled logging NOTICE, WARNING etc.
https://cloud.google.com/logging/docs/reference/v2/rest/v2/LogEntry
The docs say to set the trace identifier to correlate log entries with the originating request log. The trace ID is extracted from the X-Cloud-Trace-Context header set by the container.
Simulate it locally using curl -v -H 'X-Cloud-Trace-Context: 1ad1e4f50427b51eadc9b36064d40cc2/8196282844182683029;o=1' http://localhost:8080/
However, this does not cause the messages to be threaded by request, but instead the trace property appears in the jsonPayload object in the logs. (See below).
Notice that severity has been interpreted as expected and does not appear in the jsonPayload. I had expected the same to happen for trace, but instead it appears to be unprocessed.
How can I achieve nested messages within the original request log message? (This must be done using stdout on Go 1.11 as I do not wish to log directly with the Google Cloud logging package).
What exactly is GAE doing to parse the stdout stream from my running process? (In the setup guide for VMs on GCE there is something about installing an agent program to act as a conduit to Stackdriver logging- is this what GAE has installed?)
app.yaml file looks like this:
runtime: go111
handlers:
- url: /.*
script: auto
package main
import (
"fmt"
"log"
"net/http"
"os"
"strings"
)
var projectID = "glowing-market-234808"
func parseXCloudTraceContext(t string) (traceID, spanID, traceTrue string) {
slash := strings.Index(t, "/")
semi := strings.Index(t, ";")
equal := strings.Index(t, "=")
return t[0:slash], t[slash+1 : semi], t[equal+1:]
}
func sayHello(w http.ResponseWriter, r *http.Request) {
xTrace := r.Header.Get("X-Cloud-Trace-Context")
traceID, spanID, _ := parseXCloudTraceContext(xTrace)
trace := fmt.Sprintf("projects/%s/traces/%s", projectID, traceID)
warning := fmt.Sprintf(`{"trace":"%s","spanId":"%s", "severity":"WARNING","name":"Andy","age":45}`, trace, spanID)
fmt.Fprintf(os.Stdout, "%s\n", warning)
message := "Hello"
w.Write([]byte(message))
}
func main() {
http.HandleFunc("/", sayHello)
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
log.Fatal(http.ListenAndServe(fmt.Sprintf(":%s", port), nil))
}
Output shown in the Log viewer:
...,
jsonPayload: {
age: 45
name: "Andy"
spanId: "13076979521824861986"
trace: "projects/glowing-market-234808/traces/e880a38fb5f726216f94548a76a6e474"
},
severity: "WARNING",
...
I have solved this by adjusting the program to use logging.googleapis.com/trace in place of trace and logging.googleapis.com/spanId in place of spanId.
warning := fmt.Sprintf(`{"logging.googleapis.com/trace":"%s","logging.googleapis.com/spanId":"%s", "severity":"WARNING","name":"Andy","age":45}`, trace, spanID)
fmt.Fprintf(os.Stdout, "%s\n", warning)
It seems that GAE is using the logging agent google-fluentd (a modified version of the fluentd log data colletor.)
See this link for a full explanation.
https://cloud.google.com/logging/docs/agent/configuration#special-fields
[update] June 25th, 2019: I've written a Logrus plugin that will help to thread log entries by HTTP request. It's available under on GitHub https://github.com/andyfusniak/stackdriver-gae-logrus-plugin.
[update]] April 3rd, 2020: I've since switched to using Cloud Run and the Logrus plugin appears to work fine with this platform also.
I'm trying to send requests to Apple's APNS service using their HTTP/2 API, and my service is working fine locally, but once it's on a Managed VM it seems the underlying sockets are dying after a few minutes and the Go HTTP library is unable to handle it gracefully.
What I see is the requests working fine (getting responses) for a while, but if it's idle for a few minutes the connections will take minutes to time out with read tcp 172.17.0.4:35395->17.172.234.19:443: read: connection timed out (apparently ignoring the 10 second timeout I specified).
I've previously had keep-alive issues with Managed VMs specifically, but Google has indicated it should be fixed. Does anyone know how to avoid this issue?
I'm creating an HTTP/2 client in this way:
func NewClient() *http.Client {
cert, err := tls.LoadX509KeyPair("secrets/prod_voip.pem", "secrets/prod_voip.key")
if err != nil {
log.Fatalln(err)
}
config := &tls.Config{
Certificates: []tls.Certificate{cert},
}
config.BuildNameToCertificate()
dialer := &net.Dialer{
Timeout: 10 * time.Second,
}
transport := &http.Transport{
Dial: dialer.Dial,
TLSClientConfig: config,
}
// Explicitly enable HTTP/2 as TLS-configured clients don't auto-upgrade.
// See: https://github.com/golang/go/issues/14275
if err := http2.ConfigureTransport(transport); err != nil {
log.Fatalln(err)
}
return &http.Client{Transport: transport}
}
When I'm testing my website locally everything works. But in production, I get the following error: Couldn't lease a task: API error 1 (taskqueue: UNKNOWN_QUEUE)
I'm pretty sure my code is correct since it works locally. My best guess is that there is something wrong with the queue.yaml file, but it's dead simple:
queue:
- name: daemonQueue
mode: pull
What could I be doing wrong?
EDIT:
Turns out enqueueing fails as well: TickTask enqueue error: Failed to insert task: API error 1 (taskqueue: UNKNOWN_QUEUE)
Here is how I'm enqueueing the task.
// Add the task to the queue.
func EnqueueWithName(c sessions.Context, task interface{}, tag string, name string) (err error) {
buffer := new(bytes.Buffer)
err = gob.NewEncoder(buffer).Encode(task)
if err != nil {
return
}
newTask := &taskqueue.Task{
Method: "PULL",
Payload: buffer.Bytes(),
Tag: tag,
Name: name}
newTask, err = taskqueue.Add(c, newTask, "daemonQueue")
return err
}
You don't have the queue configured in production. Make sure you are deploying your whole app directory so queue.yaml get uploaded. Point goapp deploy or appcfg.py at the directory holding app.yaml and queue.yaml, not at app.yaml directly.
package helloworld
import (
"fmt"
"net/http"
"appengine"
"appengine/user"
)
func init() {
fmt.Print("hello")
http.HandleFunc("/", handler)
}
func handler(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
u := user.Current(c)
if u == nil {
url, err := user.LoginURL(c, r.URL.String())
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Set("Location", url)
w.WriteHeader(http.StatusFound)
return
}
fmt.Fprintf(w, "Hello, %v!", u)
}
Throw the following error in goapp serve output
(saucy)adam#localhost:~/projects/ringed-land-605/default$ goapp serve -host 0.0.0.0 .
INFO 2014-06-08 23:57:47,862 devappserver2.py:716] Skipping SDK update check.
INFO 2014-06-08 23:57:47,877 api_server.py:171] Starting API server at: http://localhost:48026
INFO 2014-06-08 23:57:47,923 dispatcher.py:182] Starting module "default" running at: http://0.0.0.0:8080
INFO 2014-06-08 23:57:47,925 admin_server.py:117] Starting admin server at: http://localhost:8000
ERROR 2014-06-08 23:57:48,759 http_runtime.py:262] bad runtime process port ['hello46591\n']
Removing the fmt.Print() fixes the issue. My question is why does that happen?
When starting the runtime process, the Go Development Server (in the App Engine Go SDK) is reading the single line response found in your helloworld's init.
This modifies the _start_process_flavor flag in http_runtime.py; consequently, the HTTP runtime attempts to read the line for direction on which port to listen to.
Read the single line response expected in the start process file. [...] The START_PROCESS_FILE flavor uses a file for the runtime instance to report back the port it is listening on.
In this case, hello isn't a valid port to listen on.
Try using Go's log package instead:
log.Print("hello")
I'm getting a permission denied error when I make a call to another web service from within my go code.
part of the handler code on my server side go program:
client := http.Client{}
if resp, err := client.Get("http://api.wipmania.com/" + r.RemoteAddr); err != nil {
c.Errorf("Error getting location from ip: %s", err.Error())
}
From the logs:
Error getting location from ip: Get http://api.wipmania.com/30.30.30.30: permission denied
I saw a similar qn here. Still unable to figure it out. Can you please tell me what is the right way to do this and if any permissions have to be set on the appengine console?
App Engine applications can communicate with other applications and can access other resources on the web by fetching URLs. An app can use the URL Fetch service to issue HTTP and HTTPS requests and receive responses.
Try:
import (
"fmt"
"net/http"
"appengine"
"appengine/urlfetch"
)
func handler(w http.ResponseWriter, r *http.Request) {
c := appengine.NewContext(r)
client := urlfetch.Client(c)
if resp, err := client.Get("http://api.wipmania.com/" + r.RemoteAddr); err != nil {
c.Errorf("Error getting location from ip: %s", err.Error())
}
// ...
}