Fetching a URL From the init() func in Go on AppEngine - google-app-engine

Background: I'm running Go on GAE and using Mux for my router. In order to fetch a URL GAE requires that I use its built in urlFetch capability. I want to make this URL fetch happen during my modules init() but as far as I can tell I can only use urlFetch when invoked via a handler.
func init() {
r := mux.NewRouter()
r.HandleFunc("/", homeHandler)
r.HandleFunc("/about", anotherHandler)
http.Handle("/", r)
}
GAE suggests the following code for making a urlFetch:
c := appengine.NewContext(r)
client := urlfetch.Client(c)
... but its argument is an http router, and it doesn't want to work if I pass my mux router. So I'm out of ideas of how to make this urlFetch happen outside the scope of a URL handler.
Error when passing the mux router: "cannot use r (type *mux.Router) as type *http.Request in argument to "appengine".NewContext"

You can't use AppEngine services that require a Context outside of handlers (because the creation of a Context requires an *http.Request value). This by nature means you can't use them in package init() functions either.
Note that you can use them from cron jobs and tasks added to task queues, because tasks and cron jobs are executed by issuing HTTP GET requests.
You have to restructure your code so that the service (urlFetch in your case) gets called from a handler.
A possible solution is to check if init completed in handlers that serve user requests. If not, perform the initialization function you would otherwise put in init() before proceeding to serve the request.
Yes, this may cause first requests to take considerably longer to serve. For this purpose (to avoid this) I recommend you to utilize Warmup requests. A warmup request is issued to a new instance before it goes "live", before it starts serving user requests. In your app.yaml config file you can enable warmup requests by adding -warmup to the inbound_services directive:
inbound_services:
- warmup
This will cause the App Engine infrastructure to first issue a GET request to /_ah/warmup. You can register a handler to this URL and perform initialization tasks. As with any other request, you will have an http.Request in the warmup handler.
But please note that:
..you may encounter loading requests, even if warmup requests are enabled in your app.
Which means that in rare cases it may happen a new instance will not receive a warmup request, so its best to check initialization state in user handlers too.

Related

Sveltekit: Sharing service objects (e.g. configured http client) across pages

I am moving my app from Svelte SPA (original) to Sveltekit multi page app (new).
In the original app, I configure a http client up top and put it context using:
setContext(HTTP_CLIENT, httpClient)
Now the entire app can get that http client using
const httpClient = getContext(HTTP_CLIENT)
I do this because my app can be started with debug parameters than turn on http request logging.
I'm not clear how to do similar in Sveltekit, because it seems that pages do not share a context.
I tried sticking the http client in the session like this:
import { session } from "$app/stores";
$session.httpClient = httpClient
and I got:
Error: Failed to serialize session data: Cannot stringify arbitrary non-POJOs
So $session is meant to be serialized, ok. Does that mean that I need to put whatever debug parameters a user supplied in $session, and each page needs to freshly instantiate its own http client? Or is there some other idiomatic sveltekit way of doing this?
PS I know sveltekit has its own fetch so you might want to say "don't use your own http client", but my app uses many different service objects (graphql client for example) that can be configured in debug (and other) modes, so please don't zero in on the fact that my example is a http client.
One way around this could be to send down the configuration in the top __layout file, create the http client there and store in a store. Since stores are shared across all pages the client can then freely use this store.

Controlling how each catch in $httpBackend.when() is handled

I'm trying to build some jasmine unit tests around my angular project. I've run into a situation I'm not sure how best to work around.
I use a response interceptor that's capable of retrying a request if it determines the error was a 401 error. It will make a call to renew the authorization token, and then reissue the request transparently.
(Orginal Call) -> 401 ? (recert and retry) : (return fail)
My problem lies with:
$httpBackend.whenPOST('/mymockedendpoint').respond(401, null);
This is the behavior I want the FIRST time it is queried. However, because it also controls all subsequent queries, my token renewal works, and then it reissues this request, but instead of returning a 200 like it would in production, it still returns the 401.
How can I extend the handling of that whenPOST so that I can control the behavior in a handler of some sort? Is that even possible?
Right now executing my test produces an infinite loop because the both reissues the request (since it successfully renewed the token) and catches again because the would-be-200 requests returns a 401).
Instead of using the "when" based functions of $httpBackend you can use the "expect" versions. This will let you assert that a particular request is made in a particular order. The documentation for $httpBackend describes the differences pretty well:
$httpBackend.expect - specifies a request expectation
$httpBackend.when - specifies a backend definition
Request Expectations vs Backend Definitions
Request expectations provide a way
to make assertions about requests made by the application and to
define responses for those requests. The test will fail if the
expected requests are not made or they are made in the wrong order.
Backend definitions allow you to define a fake backend for your
application which doesn't assert if a particular request was made or
not, it just returns a trained response if a request is made. The test
will pass whether or not the request gets made during testing.
Given that, try this in your test:
$httpBackend.expectPOST('/mymockedendpoint').respond(401, null);
$httpBackend.expectPOST('/mymockedendpoint').respond(200, { });
$httpBackend.flush();
Also, note the function $httpBackend.resetExpectations(), which could be useful in this type of scenario.

Request to App Engine Backend Timing Out

I created an App Engine backend to serve http requests for a long running process. The backend process works as expected when the query references an input of small size, but times out when the input size is large. The query parameter is the url of an App Engine BlobStore blob, which is the input data for the backend process. I thought the whole point of using App Engine backends was to avoid the timeout restricts that App Engine frontends possess. How can I avoid getting a timeout?
I call the backend like this, setting the connection timeout length to infinite:
HttpURLConnection connection = (HttpURLConnection)(new URL(url + "?" + query).openConnection());
connection.setRequestProperty("Accept-Charset", charset);
connection.setRequestMethod("GET");
connection.setConnectTimeout(0);
connection.connect();
InputStream in = connection.getInputStream();
int ch;
while ((ch = in.read()) != -1)
json = json + String.valueOf((char) ch);
System.out.println("Response Message is: " + json);
connection.disconnect();
The traceback (edited for anonymity) is:
Uncaught exception from servlet
java.net.SocketTimeoutException: Timeout while fetching URL: http://my-backend.myapp.appspot.com/somemethod?someparameter=AMIfv97IBE43y1pFaLNSKO1hAH1U4cpB45dc756FzVAyifPner8_TCJbg1pPMwMulsGnObJTgiC2I6G6CdWpSrH8TrRBO9x8BG_No26AM9LmGSkcbQZiilhC_-KGLx17mrS6QOLsUm3JFY88h8TnFNer5N6-cl0iKA
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.convertApplicationException(URLFetchServiceImpl.java:142)
at com.google.appengine.api.urlfetch.URLFetchServiceImpl.fetch(URLFetchServiceImpl.java:43)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.fetchResponse(URLFetchServiceStreamHandler.java:417)
at com.google.apphosting.utils.security.urlfetch.URLFetchServiceStreamHandler$Connection.getInputStream(URLFetchServiceStreamHandler.java:296)
at org.someorg.server.HUDXML3UploadService.doPost(SomeService.java:70)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
As you can see, I'm not getting the DeadlineExceededException, so I think something other than Google's limits is causing the timeout, and also making this a different issue from similar stackoverflow posts on the topic.
I humbly thank you for any insights.
Update 2/19/2012: I see what's going on, I think. I should be able to have the client wait indefinitely using GWT [or any other type of client-side async framework] async handler for an any client request to complete, so I don't think that is the problem. The problem is that the file upload is calling the _ah/upload App Engine system endpoint which then, once the blob is stored in the Blobstore) calls the upload service's doPost backend to process the blob. The client request to _ah/upload is what is timing out, because the backend doesn't return in a timely fashion. To make this timeout problem go away, I attempted to make the _ah_upload service itself a public backend accessible via http://backend_name.project_name.appspot.com/_ah/upload, but I don't think that google allows a system service (like _ah/upload) to be run as a backend. Now my next approach is to just have ah_upload immediately return after triggering the backend processing, and then call another service to get the original response I wanted, after processing is finished.
The solution was to start a backend process as a tasks and add that to the task queue, then returning a response to client before it waits to process the backend task (which can take a long time). If I could have assigned ah_upload to a backend, this would have also solved the problem, since the clien't async handler could wait forever for the backend to finish, but I do not think Google permits assigning System Servlets to backends. The client will now have to poll persisted backend process response data, as Paul C mentioned, since tasks can not respond like a normal servlet.

HTTPException: Deadline exceeded while waiting for HTTP response from URL: #deadline

We are using the developers python guide with Python data 2.15 library and as per example stated for app engine.
createSite("test site one", description="test site one", source_site =("https://sites.google.com/feeds/site/googleappsforus.com/google-site-template" ))
We are getting an un-predictable response every time we use.
Exception: HTTPException: Deadline exceeded while waiting for HTTP response from URL: https://sites.google.com/feeds/site/googleappsforyou.com
Did someone experience the same issue? Is it AppEngine or Sites API related?
Regards,
Deadline exceeded while waiting for HTTP response from URL is actually a DeadlineExceededError. When you are making a HTTP request, App Engine maps this request to URLFetch. URLFetch has its own deadline that is configurable. See the URLFetch documentation.
It seems that your client library catches DeadlineExceededError and throws HTTPException. Your client library either passes a deadline to URLFetch (which you would need to change) or the default deadline is insufficient for your request.
Try setting the default URLFetch deadline like this:
from google.appengine.api import urlfetch
urlfetch.set_default_fetch_deadline(45)
Also make sure to check out Dealing with DeadlineExceededErrors in the official Docs.
Keep in mind that any end-user initiated request must complete in 60 seconds or will encounter a DeadlineExceededError. See App Engine Python Runtime Request Timer.
The accepted solution did not work for me when working with the very recent versions of httplib2 and googleapiclient. The problem appears to be that httplib2.Http passes it's timeout argument all the way through to urlfetch. Since it has a default value of None, urlfetch sets the limit for that request to 5s irrespective of whatever default you set with urlfetch.set_default_fetch_deadline. It appears that you have a few options.
First option -- You can explicitly pass an instance of httplib2.Http around:
http = httplib2.Http(timeout=30)
...
client.method().execute(http=http)
Second option, you can set the default value using sockets1:
import socket
socket.setdefaulttimeout(30)
Third option, you can tell appengine to use sockets for requests2. To do that, you modify app.yaml:
env_variables:
GAE_USE_SOCKETS_HTTPLIB : 'anyvalue'
1This might only work for paid apps since the socket api might not be present for unpaid apps...
2I'm almost certain this will only work for paid apps since the socket api won't be functional for unpaid apps...

getCurrentUser Magic in AppEngine - how does it work?

How does Google allow userService.getCurrentUser() to be called without having it passed HttpServletRequest request param? I am looking to extend UserService but it does not allow me to do that easily.
I have created my own authentication mechanism which overlaps UserService so that I can allow Google supported users and users who will login automatically. But I am interested in knowing how can I send the getCurrentUser when called from any function.
This is the current workflow
Request to secure resource -> Filter -> set current user somewhere but I am not sure WHERE so that I can call it from the rest of the application without passing httprequests..
in my code
GetCurrentUser() should return the user set in the filter. Note that I will not be passing the original HttpServletRequest
Thanks.
After few days of research, we can safely save the session info in the ThreadLocal. This is set in the servlet filter.
Static ThreadLocal variable in a WebApp - Are there any Security/Performance issues?
Since each servlet is executed in a single thread, we can safely call getCurrentUser from the code which then reads from the threadlocal.
Caveat though. Threads are recycled. So you can get a thread which has threadlocal set for a different user. You have to make sure that it is reset everytime.

Resources