I have developed a flutter app for android and for Ios, both with the same code. I followed the official guides to implement the database with sqflite. The problem comes when I have to work with my database (inserts, deletes...). My app has to do about 12 inserts, that are instant on IOS devices, but take about 10 seconds on different average Android devices. Both are realese versions.
As I don't have prior experience with databases I don't know if it's normal to take that long or what is going on here. As I mentioned the code is the same for Ios and for Android, I use the function insert(), about 12 of them, and the data I insert is quite simple.
await db.insert(data);
await db.insert(data);
await db.insert(data);
...
In addition to my problem I found the same problem here https://github.com/tekartik/sqflite/issues/202 , but nothing more.
It is not that Sqflite is particularly slow it is because you are using await with multiple db.insert. Basically await is a thread-blocking process which means that your 2nd db.insert won't be performed until the 1st one is done, same goes for the 3rd one it won't be completed while the 2nd is still running.
Moreover you are chaining calls to db.insert, instead you should use db.batch() it will allows you to plan and perform your db.insert all at once without blocking your database and avoid ping-pong between native code as it is said in the package documentation.
Here is a code sample of how you could do your insertions:
final batch = db.batch();
batch.insert('myTable', data);
batch.insert('myTable', data);
batch.insert('myTable', data);
await batch.commit();
Just by doing this you will only need one await which should improve your performances.
Related
I'm using MongoDB in combination with Meteor + React and the result fetching takes like 5 sec even on a small database.
This happens only on the production server (AWS) and it works instantly on the local machine.
In order to fetch the results, I'm using the following code.
return{ cand : Job.find({thejob:props.id}).fetch() };
and to see if the array has been loaded, I use the following code on the frontend side.
if(!this.props.cand){return(<div>Loading....</div>)}
but the Loading.... takes like 5 sec on the server always. The database is a small one with less than 1000 records.
I have had similar experiences. The performance is pretty good when you run the queries in the local machine. If the query is slower in platforms like AWS and not on the local, it's mostly due to the Network latency.
I suspect there isn't an index on the thejob field.
First check if there is an index on thejob field
db.job.getIndexes()
If there is none, simply create one
db.job.createIndex({thejob:1})
I have a script that, using Remote API, iterates through all entities for a few models. Let's say two models, called FooModel with about 200 entities, and BarModel with about 1200 entities. Each has 15 StringPropertys.
for model in [FooModel, BarModel]:
print 'Downloading {}'.format(model.__name__)
new_items_iter = model.query().iter()
new_items = [i.to_dict() for i in new_items_iter]
print new_items
When I run this in my console, it hangs for a while after printing 'Downloading BarModel'. It hangs until I hit ctrl+C, at which point it prints the downloaded list of items.
When this is run in a Jenkins job, there's no one to press ctrl+C, so it just runs continuously (last night it ran for 6 hours before something, presumably Jenkins, killed it). Datastore activity logs reveal that the datastore was taking 5.5 API calls per second for the entire 6 hours, racking up a few dollars in GAE usage charges in the meantime.
Why is this happening? What's with the weird behavior of ctrl+C? Why is the iterator not finishing?
This is a known issue currently being tracked on the Google App Engine public issue tracker under Issue 12908. The issue was forwarded to the engineering team and progress on this issue will be discussed on said thread. Should this be affecting you, please star the issue to receive updates.
In short, the issue appears to be with the remote_api script. When querying entities of a given kind, it will hang when fetching 1001 + batch_size entities when the batch_size is specified. This does not happen in production outside of the remote_api.
Possible workarounds
Using the remote_api
One could limit the number of entities fetched per script execution using the limit argument for queries. This may be somewhat tedious but the script could simply be executed repeatedly from another script to essentially have the same effect.
Using admin URLs
For repeated operations, it may be worthwhile to build a web UI accessible only to admins. This can be done with the help of the users module as shown here. This is not really practical for a one-time task but far more robust for regular maintenance tasks. As this does not use the remote_api at all, one would not encounter this bug.
To my understanding, we should do database calls async since it will block IOLoop, and I also found some library/tools that working on this.
But recently I just found lots of posts about this question on stackoverflow
Tornado Asynchronous Handler
How does async work in Tornado?
and something from wiki https://github.com/tornadoweb/tornado/wiki/Threading-and-concurrency
Do it synchronously and block the IOLoop. This is most appropriate for
things like memcache and database queries that are under your control
and should always be fast. If it's not fast, make it fast by adding
the appropriate indexes to the database, etc.
So should we do database calls async?
and a related question: as I know, there are two methods to do database calls async
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
For async database(motor), we can use it directly.
what's the different of them based on tornado?
I had a similar doubt and went through some research and digging and finally have some grip over this.
Short Answer - Yes, Do it Asynchronously.
Long Answer :
Database calls are generally blocking calls. Tornado is best suited as a framework when we do async/Non-blocking I/O calls to completely utilize its full potential.
Now, You can do Database Calls in Tornado in the following ways, I am giving the How and Why for each method.
Method 1 : Synchronous Database Calls - Start a number of Tornado instances behind a load balancer like for example nginx, and make synchronous database calls(as mentioned in the wiki). In this case, the process with the particular request making the blocking database call will block. So nginx will load balance the other requests by directing them to the other running tornado instances to handle the requests.
NOTE : That wiki is old, and hasn't been updated so that method mentioned is prior to the addition of coroutines in the newer versions of Tornado as mentioned by Ben Darnell, one of the lead developers of Tornado.
Update : The wiki has been updated by Ben recently and you can check it now.
Method 2 : By using multiprocessing or multithreading, concurrent.futures, similar to what you have mentioned:
For sync database(mysqldb), we can
executor = ThreadPoolExecutor(4)
result = yield executor.submit(mysqldb_operation)
Method 3 : By using Async libraries, like for instance if you are using MySql, then you can use libraries like TorMySql, Tornado-MySQL, other links given here-Tornado 3rd-party libraries, go to the MySql section. These libraries are to be used with Tornado's async decorators, tornado.gen.coroutine or tornado.web.asynchronous. Similarly, you need to use other async libraries for different databases.
NOTE : You cannot use blocking libraries otherwise its all the same as Method 1.
CONCLUSION : Method 3 is what I mentioned as my Short Answer. Method 2 is the next best solution, followed by Method 1. This is in accordance with your performance requirements. If you requirement is for not-so-heavy(few to moderate) request processing per second or per minute, than going for the easy and general Method 1 will suffice.
I have a gatling test for an application that will answer a survey and upon answering this survey, the application will identify possible answers that may pose a risk and create what we call riskareas. These riskareas are normally created in the background as soon as the survey answering is finished. My question is I have a gatling test with ten users who will go and answer the survey and logout, I used recorder to record the test; now after these ten users are finished I do not see any riskareas being created in the application. Am I missing something--should the survey be really answered by gatling (like it does in selenium) user or is it just the urls that the gatling test will touch ?
I am new to gatling please help.
Gatling should be indistinguishable from a user in a web browser (or Selenium) as far as the server is concerned, so the end result should be exactly the same as if you'd gone through the process yourself. However, writing a Gatling script is a little more work than writing a Selenium script.
For performance reasons, Gatling operates at a lower level than Selenium. Gatling works with the actual data that is sent and received from the server (i.e, the actual GETs and POSTs sent to the server), rather than with user-level interactions (such as clicking links and filling forms).
The recorder will generally produce a relaitvely "dumb" script. It records the exact data that was sent to the server, and makes no attempt to account for things that may change from run to run. For example, the web application you are testing might have hidden form fields that contain session information, or the link addresses might contain a unique identifier or a session id.
This means that your script may not be doing what you think it's doing.
To debug the script, the first thing to do is to add checks on each of the requests, to validate that you are getting the response you expect (for example, check that when you submit page 1 of the survey, you are taken to page 2 - check for something that you'd only expect to find on page 2, like a specific question).
Once you know which requests are failing, look at what data was sent with the request, and try to figure out where it came from. You will probably find that there are session ids, view state, or similar, that must be extracted from the previous page.
It will help to enable request and response logging, as per the documentation.
To simplify testing of web apps, we wrote some helper functions to allow tests to be written in a more Selenium-like way. Once you understand what your application is doing, you may find that it simplifies scripting for you too. However, understanding why your current script doesn't work the way you expect should be your first step.
I'm working on a rather large classic asp / SQL Server application.
A new version was rolled out a few months ago with a lot of new features, and I must have a very nasty bug somewhere : some very basic pages randomly take a very long time to execute.
A few clues :
It isn't the database : when I run the query profiler, it doesn't detect any long running query
When I launch IIS Diagnostic tools, reqviewer shows that the request is in state "processing"
This can happen on ANY page
I can't reproduce it easily, it's completely random.
To have an idea of "a very long time" : this morning I had a page take more than 5 minutes to execute, when it normaly should be returned to the client in less than 100 ms.
The application can handle rather large upload and download of files (up to 2 gb in size). This is also handled with a classic asp script, using SoftArtisan FileUp. Don't think it can cause the problem though, we've had these uploads for quite a while now.
I've had the problem on two separate servers (in two separate locations, with different sets of data). One is running the application with good ol' SQL Server 2000 and the other runs SQL Server 2005. The web server is IIS 6 in both cases.
Any idea what the problem is or on how to solve that kind of problem ?
Thanks.
Sebastien
Edit :
The problem came from memory fragmentation. Some asp pages were used to download files from the server. File sizes could go from a few kb to more than 2 gb. These variations in size induced memory fragmentation. The asp pages could also take quite some time to execute (the time for the user to download the pages minus what is put in cache at IIS's level), which is not really standard for server pages that should execute quickly.
This is what I did to improve things :
Put all the download logic in a single asp page with session turned off
That allowed me to put that asp page in a specific pool that could be recycled every so often (download would now disturb the rest of the application no more)
Turn on LFH (Low Fragmention Heap), which is not by default on Windows 2003, in order to reduce memory fragmentation
References for LFH :
http://msdn.microsoft.com/en-us/library/aa366750(v=vs.85).aspx
Link (there is a dll there that you can use to turn on LFH, but the article is in French. You'll have to learn our beautiful language now!)
I noticed the same thing on a classic ASP + ajax application that I worked on. Using Timer, I timed the page load to be 153 milliseconds but in the firebug waterfall chart it randomly says 3.5 seconds. The Timer output is on the response and the waterfall chart claims that it's Firefox waiting for a response from the server. Because the waterfall chart also shows the response, I can compare the waterfall chart to the timer and there's a huge discrepancy 'every so often'
Can you establish whether this is a problem for all pages or a common subset of pages?
If a subset examine what these pages have in common, for example they all use a specific COM dll, that other pages don't.
Does this problem affect multiple clients or just a few?
IOW is there an issue with a specific browser OS version.
Is this public or intranet?
Can you reproduce the problem from a client you own?
Is there any chance there are some full-text search queries going on SQL Server?
Because if so, and if SQL Server has no access to internet, it may cause a 45-second delay every few hours or so when it tries to check the certifications (though this does not apply to SQL Server 2000).
For a detailed explanation of what I'm referring to, read this.
Are any other apps running on your web server? If so, is your problematic in the same app pool as any of them? If so, try creating a dedicated app pool for it. Maybe one of the other apps is having a problem and is adversely affecting yours.
One thing to watch out for is if you have server side debugging turned on in IIS, the web server will run in single threaded mode.
So if you try to load a page, and someone else has hit that url at the same time, you will be queued up behind them. It will seem like pages take a long time to load, but its simply because the server is doling out page requests in a single file line and sometimes you aren't at the front of the line.
You may have turned this on for debugging and forgot to turn it off for production.