Getting huge random latency in app engine - google-app-engine

Question

I run an API in app engine. Sometimes it takes a request only ~50ms to complete and sometimes it takes 10-15 seconds!
Here's what it looks like in the Google Chrome console:
As you can see, some requests are very fast, and some very slow.
Using StackDrive Trace I can confirm that it takes the API 10 seconds or longer sometimes. I tried automatically making requests each second to see if it speeds up after the first request, but it still seems random.
So the next thing I tried is measuring if the API itself is slow due to my own code. I tested it, but it seems to be very fast and not the cause of the problem. Neither do i make any requests inside my API that could be slowing it down (other than a database request).
I am still trying to figure out what it is that is causing this massive latency, but it seems like it happens in between the request being made on the frontend and the request being received on the backend.
I would highly appreciate any help and suggestions!
EDIT 1
Seems like the 204 No Content responses are also slow sometimes.
Here's more strange behavior. On the frontend I make several requests at once to load a page. For every request there is almost exactly a one second delay:
I still have not even figured out the cause of this problem, help is still appreciated.
EDIT 2
My timeline doesn't seem to break down the way it does for Alex:
I tried adding this to all http headers:
'Cache-Control': 'no-cache',
Pragma: 'no-cache'
Which sadly is not solving my problem either.
EDIT 3
The 10 second latency is probably caused by 10 requests all being fired at once, each taking 1 second.
So my first question is:
Can a single app engine f1 instance not handle multiple (concurrent) requests at once?
And my second question:
Why does it take over 1 second (sometimes over 2 seconds) to process a single request?
I did another test to find out if it is my code that is slowing down the requests. I deployed a .net core MVC controller with only 1 task. All it does is return "Hello world". Here are the results (using this method):
> curl.exe -s -o --url "http://api.---.com/test" -w "#curl-format.txt"
time_namelookup: 0,000001
time_connect: 0,109000
time_appconnect: 0,000000
time_pretransfer: 0,109000
time_redirect: 0,000000
time_starttransfer: 1,203000
--------
time_total: 1,203000

Answer 1

In your fast requests, the responses are 204 (No Content) and they are 45 bytes. The slow requests are responding with 200 and are actually returning something.
Is there some kind of caching that's effecting this?
EDIT 1: Since your server was returning 204's I was more referring to any caching that you implemented on the client-side. I see that you found the trace screen (https://console.cloud.google.com/traces/traces), have you tried clicking on one of them? It gives you a breakdown like this:
that should tell you where the request is spending its time