I'm using MongoDB in combination with Meteor + React and the result fetching takes like 5 sec even on a small database.
This happens only on the production server (AWS) and it works instantly on the local machine.
In order to fetch the results, I'm using the following code.
return{ cand : Job.find({thejob:props.id}).fetch() };
and to see if the array has been loaded, I use the following code on the frontend side.
if(!this.props.cand){return(<div>Loading....</div>)}
but the Loading.... takes like 5 sec on the server always. The database is a small one with less than 1000 records.
I have had similar experiences. The performance is pretty good when you run the queries in the local machine. If the query is slower in platforms like AWS and not on the local, it's mostly due to the Network latency.
I suspect there isn't an index on the thejob field.
First check if there is an index on thejob field
db.job.getIndexes()
If there is none, simply create one
db.job.createIndex({thejob:1})
Related
Using AWS' DocumentDB, I've deployed a test database cluster in Germany. When I run a test-query on it from an EC2 instance in Germany, it takes less than a 2 seconds. When I query it from my country (middle-east) it takes more then a minute.
This is the simple test I run (it just goes through the whole collection):
var last_doc= db.dev.find()
while(last_doc.hasNext()){ last_doc = last_doc.next(); }
Thing is, we also have an RDS (MySQL) DB in AWS and a query of the same size takes less than 2 seconds from my country to RDS (Also in Germany).
I tried viewing the logs so I followed this document but I can't seem to find any logs from docdb in CloudWatch. These are my cluster parameters. I also tried opening a ticket with AWS but apparently our basic subscription doesn't allow creating tickets.
Does anyone happen to have suggestion on how to tackle this? What should I do/look for?
Thanks ahead!
I have a MongoDB instance running on Mongo Atlas and I have a local machine.
I want to execute a script but I would like this script to be execute on the mongo instance.
I have tried several things like Robo3T or Mongo shell. And it looks like the behaviour is not the one I want.
Suppose we have this script :
print(db.users.find({}).toArray().length);
My users collection has around 30k rows. I voluntarily use toArray() to force the creation of a js array. But I want this array to be created... In the MongoDB instance or close to it ; not on the instance where I launched the mongo shell (or Robo3T).
This is obviously not my use case to count the number of users, if I really just wanted the number of users, I would have used .count() and it would have been faster. But I just want to illustrate the fact that the code is run not at the location I want it to be run.
Suppose you connect to a remote ssh. You have a very poor connection.
If you do something like
wget http://moviedatabase.com/rocky.mp4
which is a 1 To movie.
You will take the same time if your connection is blazing fast or amazingly slow : what counts is the bandwith of the server you are connecting to.
With my example, all depends on the connection of the instance you are launching Mongo shell on.
If it has a good connection, it will be faster than if it has a good connection.
What is the way to execute js code "closer" to the MongoDB instance?
How this behaviour not a problem when you administrate a MongoDB instance?
Thanks in advance,
Jerome
It depends on what you are trying to do.
There is no generic context where you can run arbitrary code, but you can store a javascript function on the server, which can then be used in $where or mapReduce.
Note that server-side javascript can be disabled with the security.javascriptEnable configuration parameter.
I would expect that Atlas disables this for it's free and shared tiers.
I'm profiling my rails 3.2 app with miniprofiler, and it shows me a
SELECT 1
query at the beginning of each page load or ajax call. It only takes 0.4ms, but it is still a seemingly unnecessary database query.
Anyone know why this query is happening or how to get rid of it?
SELECT 1 is like a ping - the cheapest query to test whether the session is alive and kicking. Various clients use it for that purpose. It may be useless in your case ...
For Postgres you can find it in this line on Github.
Or, if you using MySQL, you can see the solution in this groupon engineering blog.
I am using Django 1.4 on GAE + Google Cloud SQL - my code works perfectly fine (on dev with local sqlite3 db for Django) but chocks with Server Error (500) when I try to "refresh" DB. This involves parsing certain files and creating ~10K records and saving them (I'm saving them in batch using commit_on_success).
Any advise ?
This error is raised for front end requests after 60 seconds. (its increased)
Solution options:
Use task queue (again a time limit of 10 minutes is imposed, which is practically enough).
Divide your task in smaller batches.
How we do it: we divide it on client side in smaller chunks and call them repeatedly.
Both the solutions work fine, depends on how you make these calls and want the results. Taskqueue doesn't return back the results to the client.
For tasks that take longer than 30s you should use task queue.
Also, database operations can also timeout when batch operations are too big. Try to use smaller batches.
Google app engine has a maximum time allowed for a request. If a request takes longer than 30 seconds, this error is raised. If you have a large quantity of data to upload, either import it directly from the admin console, or break up the request into smaller chunks, or use the command line python manage.py dbshell to upload the data from your computer.
I'm working on a rather large classic asp / SQL Server application.
A new version was rolled out a few months ago with a lot of new features, and I must have a very nasty bug somewhere : some very basic pages randomly take a very long time to execute.
A few clues :
It isn't the database : when I run the query profiler, it doesn't detect any long running query
When I launch IIS Diagnostic tools, reqviewer shows that the request is in state "processing"
This can happen on ANY page
I can't reproduce it easily, it's completely random.
To have an idea of "a very long time" : this morning I had a page take more than 5 minutes to execute, when it normaly should be returned to the client in less than 100 ms.
The application can handle rather large upload and download of files (up to 2 gb in size). This is also handled with a classic asp script, using SoftArtisan FileUp. Don't think it can cause the problem though, we've had these uploads for quite a while now.
I've had the problem on two separate servers (in two separate locations, with different sets of data). One is running the application with good ol' SQL Server 2000 and the other runs SQL Server 2005. The web server is IIS 6 in both cases.
Any idea what the problem is or on how to solve that kind of problem ?
Thanks.
Sebastien
Edit :
The problem came from memory fragmentation. Some asp pages were used to download files from the server. File sizes could go from a few kb to more than 2 gb. These variations in size induced memory fragmentation. The asp pages could also take quite some time to execute (the time for the user to download the pages minus what is put in cache at IIS's level), which is not really standard for server pages that should execute quickly.
This is what I did to improve things :
Put all the download logic in a single asp page with session turned off
That allowed me to put that asp page in a specific pool that could be recycled every so often (download would now disturb the rest of the application no more)
Turn on LFH (Low Fragmention Heap), which is not by default on Windows 2003, in order to reduce memory fragmentation
References for LFH :
http://msdn.microsoft.com/en-us/library/aa366750(v=vs.85).aspx
Link (there is a dll there that you can use to turn on LFH, but the article is in French. You'll have to learn our beautiful language now!)
I noticed the same thing on a classic ASP + ajax application that I worked on. Using Timer, I timed the page load to be 153 milliseconds but in the firebug waterfall chart it randomly says 3.5 seconds. The Timer output is on the response and the waterfall chart claims that it's Firefox waiting for a response from the server. Because the waterfall chart also shows the response, I can compare the waterfall chart to the timer and there's a huge discrepancy 'every so often'
Can you establish whether this is a problem for all pages or a common subset of pages?
If a subset examine what these pages have in common, for example they all use a specific COM dll, that other pages don't.
Does this problem affect multiple clients or just a few?
IOW is there an issue with a specific browser OS version.
Is this public or intranet?
Can you reproduce the problem from a client you own?
Is there any chance there are some full-text search queries going on SQL Server?
Because if so, and if SQL Server has no access to internet, it may cause a 45-second delay every few hours or so when it tries to check the certifications (though this does not apply to SQL Server 2000).
For a detailed explanation of what I'm referring to, read this.
Are any other apps running on your web server? If so, is your problematic in the same app pool as any of them? If so, try creating a dedicated app pool for it. Maybe one of the other apps is having a problem and is adversely affecting yours.
One thing to watch out for is if you have server side debugging turned on in IIS, the web server will run in single threaded mode.
So if you try to load a page, and someone else has hit that url at the same time, you will be queued up behind them. It will seem like pages take a long time to load, but its simply because the server is doling out page requests in a single file line and sometimes you aren't at the front of the line.
You may have turned this on for debugging and forgot to turn it off for production.