I have a site with around 4000 pages which is using fine grained permissions on around 400 of those pages. For some users accessing the /admin/ page the page loads in a second or so. For others the page can take around 5 minutes to load.
Profiling and debugging into the code seems to show that the area of concern is the wagtail\core\models.py explorable_pages() function, in particular the
page_permissions = Page.objects.filter(group_permissions__in=self.permissions)
line. This seems to return, for my data at least, over 2000 results with each reults repeated 6 or 7 times (returning a result for each permission type?). There are only around 400 distinct results.
Adding a distinct() improves matters
page_permissions = Page.objects.filter(group_permissions__in=self.permissions).distinct()
getting the number down to around 400. My page loads are now around 90 seconds - not ideal but better.
Is the above a real issue with wagtail or am I missing/not understanding something?
Is there something I can do to speed up the page loading? Are the site permissions being set in a way that would be expected for a typical wagtail site?
(Excuse my ignorance and I am probably using the wrong terminology - I am Java programmer new to Python/Django/Wagtail)
Related
I run an API in app engine. Sometimes it takes a request only ~50ms to complete and sometimes it takes 10-15 seconds!
Here's what it looks like in the Google Chrome console:
As you can see, some requests are very fast, and some very slow.
Using StackDrive Trace I can confirm that it takes the API 10 seconds or longer sometimes. I tried automatically making requests each second to see if it speeds up after the first request, but it still seems random.
So the next thing I tried is measuring if the API itself is slow due to my own code. I tested it, but it seems to be very fast and not the cause of the problem. Neither do i make any requests inside my API that could be slowing it down (other than a database request).
I am still trying to figure out what it is that is causing this massive latency, but it seems like it happens in between the request being made on the frontend and the request being received on the backend.
I would highly appreciate any help and suggestions!
EDIT 1
Seems like the 204 No Content responses are also slow sometimes.
Here's more strange behavior. On the frontend I make several requests at once to load a page. For every request there is almost exactly a one second delay:
I still have not even figured out the cause of this problem, help is still appreciated.
EDIT 2
My timeline doesn't seem to break down the way it does for Alex:
I tried adding this to all http headers:
'Cache-Control': 'no-cache',
Pragma: 'no-cache'
Which sadly is not solving my problem either.
EDIT 3
The 10 second latency is probably caused by 10 requests all being fired at once, each taking 1 second.
So my first question is:
Can a single app engine f1 instance not handle multiple (concurrent) requests at once?
And my second question:
Why does it take over 1 second (sometimes over 2 seconds) to process a single request?
I did another test to find out if it is my code that is slowing down the requests. I deployed a .net core MVC controller with only 1 task. All it does is return "Hello world". Here are the results (using this method):
> curl.exe -s -o --url "http://api.---.com/test" -w "#curl-format.txt"
time_namelookup: 0,000001
time_connect: 0,109000
time_appconnect: 0,000000
time_pretransfer: 0,109000
time_redirect: 0,000000
time_starttransfer: 1,203000
--------
time_total: 1,203000
In your fast requests, the responses are 204 (No Content) and they are 45 bytes. The slow requests are responding with 200 and are actually returning something.
Is there some kind of caching that's effecting this?
EDIT 1: Since your server was returning 204's I was more referring to any caching that you implemented on the client-side. I see that you found the trace screen (https://console.cloud.google.com/traces/traces), have you tried clicking on one of them? It gives you a breakdown like this:
that should tell you where the request is spending its time
Background:
I have built a tool using angularjs. The user is able to view a dynamic page based on the data. The page can be really long with lots of various types of fields within it. There are many and various angularjs components used within the app.
Issue:
If a user has got lots of data (which is shown within various input fieds/date fields etc; around 500 fields ) then the page takes around 3 minutes to load in IE browser (IE11 is my browser). The same page loads within 20 seconds on chrome and firefox. I have tried now almost a month to find the issue ..but still no luck.
I am very desperate for a solution. I coudn't find any tool that would show me what is causing my page to take so long to load.
Well, first things first, you'll need to profile what is actually taking so long. I suggest you check out
https://github.com/angular/batarang
to do this. A good article that goes over it's use is available at http://daginge.com/technology/2013/08/11/debugging-performance-problems-in-angularjs-with-batarang/ -
It's too long to include in this answer, but the general flow of resolving this is probably going to involve streamlining the watchers involved in each of those component fields etc - rendering 500 fields at once seems somewhat unnecessary - pagination would probably be an easy fix. Limit the data to maybe 50 fields per page? You'll need to track if it's the actual queries you're running to get the data that is taking so long, although based on the difference betweeen IE and Chrome, I would guess it's going to be something in the browser. It could also be that IE is being forced to use some polyfills for the functionality that Chrome and FF are supplying natively. Maybe link to your repo for us to have a look at?
I'm facing some performance issue.
Indeed, the app itself reacts pretty well when it's loaded but my problem is more like when I perform a F5 ( and app reload again), then I see some strange idle time that are killing my app loading time.
When starting my app :
I do some queries to my webserver : take around 500ms - 700ms
I present the requested datas
I have some translation to every text (that generates a lot of watchers)
I use angular-translate for that. If you have a better solution of internationalisation ;)
When app is started, I have 1200 Watchers. I have tried removed ALL translation and have around 700 watchers but do not see any difference on behaviour.
I have around 25 Modules loaded, 18 are mandatory for app to run
Well, let's dig to the issue, here is a chrome Timeline representing my problem.
I presented 2 different ones.
Any idea why I have those IDLE Times ?
Any Idea on how I can investigate more on the root cause of that issue ?
Edit : Information 09/06/2015
all my JS code is into appcache, therefore, all my code is loaded at
app first init, and then into cache. I have added into appcache all
vendors / lib / js / fonts / img etc...
there is during that time 5 or 6 requests that are mainly done into
the first block. This is a corporation private angularJS app
from those requests, we granted to the user rights and init the app
we have tried to remove all HTML generation from the app, and keep only the JS to see if the blocking point was the rendering, no
effects
we have tried to remove angular-translate, that generates a lot of watchers for "on the go" translation, no effects
Thank for support,
Have you tried using one-time binding for parts you only want to show for reading not editing e.g. dynamically generated Tables / Lists ?
Hello {{::name}}!
more here
I am experiencing a performance bottle-neck in this website: http://oceanosdecolor.es/ and I'm not able to find it. If you try, you'll see any page (for example, homepage) takes a long time to load.
The first time you execute a page, the site reloads to detect client device, but that's only the first time and then it keeps client device so it doesn't reload again.
I log traces of the execution to database but I don't get useful information as, according to the log, the execution of the whole homepage happens in the same 1 second, but I can see that the homepage takes more time to load.
The IIS log (when trying locally) doesn't help as this also gives information in seconds, not miliseconds, and again, it says everything happens in the same second, and anyway running locally is much faster than on the server.
So, I ask for help in any tool to monitor performance with more accuracy or any technique I could use.
Thank you
I think your answer might not lie with IIS or SQL Server. According to the Developer Tools in Chrome, your actual page execution and sending out the HTML takes 400ms on first load from my location. The problem is you have a tangle of CSS files (many of which are not being found and causing extremely long delays). Also you have a lot of requests.
I would install Yahoo's YSlow for your favourite browser. This will give you a whole bunch of recommendations for what is running slow on your site from an end-user perspective.
To use the Developer Tools on Chrome: right click on your page, hit "Inspect Element" and then go to the "Network" tab and then hard-refresh your browser (shift-F5).
A few of the problems I see are: commun.css (2.5 seconds and failed), layout.css (400ms and failed), jquery-ui-1.8.10.custom.min.js (800ms and failed).
Find the reason for these failing and fix it and I'm sure your site will load faster. Also try use CSS image sprites wherever possible to cut down on the number of requests.
I am working on an application on Google App Engine. Roughly this is what I do:
The user screen is split into 2 parts (actually 3, but lets leave that out for now). The left part (this takes upto 75% of the screen) has a document with some words highlighted. When one of these highlighted words are clicked the right part displays various meanings of it, example usage etc. The way this works is clicking the word send an XML HTTP Request to the server, where the sample usage(s)/meaning(s) are retrieved from the datastore. This data is returned and displayed.
My problem:
After I click on a few words consecutively, the application seems to "hang" - say, I click on 5 words in quick succession, clicking on the 6th word (or any word after that) doesn't replace the info regarding the 5th word on my right panel.
Since some data store columns (at least single valued properties) are indexed by default I'm guessing retrieval is not the bottleneck here. It is probably the requests.
Is such an issue known with the GAE? Any workarounds possible?
Kind of in a soup with this - the application was supposed to go live today. Urgent help required!
Thanks! :)
You're probably being limited to two simultaneous requests by your browser - not by appengine. If you click on a third link before the first two have had a chance to return, make sure your app can deal with requests returning for links that should no longer be displayed.
If you were hitting a limit on appengine, you'd see exceptions in your server logs. If you're not seeing those exceptions, it's probably a client-side issue.
Sorry for the late ack (for some reason I received a notification for the responses a day late, by which we had managed to fix a few things). It does look like the problem was at the data end - our code was doing some inserts, and it turns out you can't do too many of them quickly - the logs reported a transaction time-out error. The reason we couldn't spot it earlier in the logs was we were writing simply too much info out and this was buried in somewhere.
The clicks on the user-side were pulling data from this table.
Unfortunately, the GAE simulator doesn't simulate any timeout error - so even though we had tested with comparable volumes of data before deployment this error never happened during development.
Thanks again for your responses!
And yet again, I apologize for responding late.