We're currently evaluating ArangoDB Oasis with two deployments. Both have a database with a collection of mock sensor data that has been filled with over 75,000,000 documents. When we attempt to query the collection with an aggregation window, we are getting ETIMEDOUT errors with the JavaScript driver more often than we would like. Normally, the query only takes a few milliseconds (30-40ms) with a stream cursor, but when the timeout happens, it takes around 130000ms to recover. Arango recently switched us to a different load balancer, which seemed to help slightly, but didn't eliminate the problem. We haven't had any ETIMEDOUT errors when working with a local Docker container image of Arango. Our Oasis deployments and our test environment are both hosted in the same region on AWS.
Has anyone else had problems with ETIMEDOUT errors? If so, were you able to solve them?
Related
I've deployed a Django app on AWS-ec2-micro instance and a React app on GCP-e2-micro instance before, but I encountered almost the exact same problem: Server will randomly become unresponsive and unreachable while doing some heavy I/O operations. It happens almost all the time if I try to install some large packages such as tesseract, but it sometimes freezes even when I'm just trying to run a react app using npm start. I've looked at the monitoring and they all have one thing in common: super high CPU usage. Especially after the server becomes unreachable, the CPU meters continue to rise. AWS-ec2 usually will reach almost 100% while GCP-e2 instance will reach beyond 100% to something like 140%. At a certain time, the CPU usage will become stabilized at about 50%, but the server is still unreachable using SSH.
The server sometimes recovers itself after hours of being unreachable, but usually, it ends up having to force stop and restart the server. This will cause the public ipv4 to change which I really don't like, so I want to find out why my server is constantly unresponsive.
Here is what I've installed on my server:
ssh-server
vscode-server
And then on GCP-e2, I've also installed npm, react and some UI packages. A simple react app should not have such a high I/O operation that will directly makes the server unresponsive, so I begin to think if I have something configured wrong, but I have no clue what that will be. Please help me. Thank you!
I had the same issue, I used the free tier t2.micro and it was not keeping up with all the processes that needed to be handled when executing npx create-react-app react-webapp. I had to reboot it at least 2 times to be able to ssh into it again.
Upgrading the instance type to c5a.large solved the problem, hope this helps.
I have Zeppelin 0.7.2 installed and connected to Spark 2.1.1 standalone cluster.
It has been running fine for quite a while until I changed the Spark workers' settings, to double the workers' cores and executor memory. I also tried to change the parameters SPARK_SUBMIT_OPTIONS and ZEPPELIN_JAVA_OPTS on zeppelin-env.sh, to make it request for more "Memory per node" on the Spark workers but it always requests only 1GB per node so I removed them.
I had an issue while developing a paragraph so I tried set zeppelin.spark.printREPLOutput to true on the web interface. But when I tried to save that setting, I only got a small transparent red box at right side of my browser window. So it fails to save that setting. I also got that small red box when I tried to restart the Spark interpreter. The same actually happens when I tried to change the parameters of all other interpreters or restart them.
There is nothing on the log files. So I am quite puzzled on this issue. Do any of you has ever experienced this issue? If so, what kind of solutions that you applied to fix it? If not, do you have any suggestions on how to debug this issue?
I am sorry for the noise. The issue was actually due to my silly mistake.
I actually have Zeppelin behind nginx. I recently played around with a new CMS. I didn't separate the configuration of the CMS and the proxy to Zeppelin. So any access to location containing /api/, like restarting Zeppelin interpreters or saving the interpreters' settings, got blocked. Separating the site configuration of the CMS and the proxy to Zeppelin on nginx solves the problem.
Background
From the docs, at https://github.com/GoogleCloudPlatform/gradle-appengine-plugin
I see that by putting my functionalTests in /src/functionalTests/java does the following:
Starts the Local GAE instance
runs tests in the functionalTests directory
Stops the Local instance after the tests are complete
My Issue
For my microservices, I need to have 2 local servers for running my tests. 1 server is responsible for a lot of auth operations, and the other microservices talk to this server for some verification operations.
I've tried
appengineFunctionalTest.dependsOn ':authservice:appengineRun'
this does start the dependent server, but then it hangs and the tests don't continue. I see that I can set deamon = true and start the server on a background thread, but I can only seem to do that in isolation.
Is there a way to have a 'dependsOn' also be able to pass parameters to the dependent task? I haven't found a way to make that happen.
Or perhaps there is another way to accomplish this.
Any help appreciated
I have developed an app in Twilio which I would like to run from the cloud. I tried learning about AWS and Google App Engine but am quite confused at this stage:
I have 2 questions which I hope to get your help on:
1) How can I store my scripts and database in the cloud? Right now, everything is running out of my local machine but I would like to transfer the scripts and db to another server and run my app at a predetermined time of day. What would be the best way to do this?
2) How can I write a batch file to run my app at a predetermined time of day in the cloud?
I understand this does not have code, but I really hope someone can point me to the right direction. I have spent lots of time trying to understand this myself but still am unsure. Tks in adv.
Update: The application is a Twilio app that makes calls to people, the script simply applies an algorithm to make calls in a certain fashion and the database is a mysql db that provides the details of people to be called.
This is quite difficult to provide an exact answer without understanding what is the application, what is the DB or what is the script that you wish to run.
I can give you a couple of ideas that might be helpful in such cases.
OpsWorks (http://aws.amazon.com/opsworks/) is a managed service for managing applications. You can define your stack (multiple layers like web, workers, DB...) and what are the chef recipes that should run in various points in the life of the instances in each layer (startup, shutdown, app deployment or stack modification..). Then you can use the ability to add instances to each layer in specific days and hours, to implement the functionality of running at predetermined times as you requested.
In such a solution you can either have some of your instances (like DB) always on, or even to bootstrap them using the chef recipes every day, with restore from snapshot on start and create snapshot on shutdown.
Another AWS service that you use is Data Pipeline (http://aws.amazon.com/datapipeline/). It is designed to move data periodically between data sources, for example from a MySQL database to Amazon Redshift, the Data warehouse service. But you can use it to trigger scripts and run random shell scripts that you wish (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-object-shellcommandactivity.html), and schedule it to run in various conditions like every hour/day or specific times (http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/dp-concepts-schedules.html).
A simple path here would be just to create an EC2 instance in AWS, and put the components needed to run your app there. A thorough walk through is here:
http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/get-set-up-for-amazon-ec2.html
Essentially you will create an EC2 virtual machine, which you can for most purposes treat just like any other Linux server. You can install MySQL on it, copy your script there, and run it. Of course whatever container or support libraries your code requires will need to be installed as well.
You don't say what OS you are using locally, but if it is Mac or Linux, you should be able to follow almost the same process to get your script running on an EC2 instance that you used on your local machine.
As you get to know AWS, there are sophisticated services you can use for deployment, infrastructure orchestration, database services, and so on. But just to get started running a script from a virtual machine should be pretty straightforward.
I recently developed a Twilio application using Ruby on Rails for the backend and found Heroku extremely simple to setup and launch. While Heroku does cost more than AWS, I found that the time I saved using Heroku more than made up this. As an early stage startup, we wanted to spend our time developing important features, and not "wasting" time optimizing our AWS cloud.
However, while I believe Heroku is ideal for early-stage websites/startups I do believe hosting should be reevaluated once a company reaches a certain size. At some point it becomes economically viable to devote resources into optimizing an AWS cloud solution because it will be cheaper than Heroku in the long run.
My workplace is connected via cable internet with limited up/downstream, so for load-testing a web application I am using a distributed approach:
An instance of jmeter-server (v2.4) is running on a remote linux box which has proper internet connectivity.
The controlling jmeter GUI (also 2.4) is running on my desktop.
The two are connected via a VPN.
The SUT is a web-application hosted in another datacenter.
This works fine - I am able to start the remote thread execution from the jmeter GUI and receive live results in the Summary Report.
The weird part is that my ability to put real load on the SUT is limited by my local internet connection (inbound, downstream). Tracing the link between the local jmeter GUI and the remote jmeter-server, I can see that all HTTP-traffic from SUT towards jmeter-server is sent to the local jmeter GUI. So if the jmeter-server is retrieving a 1MB file from the SUT, it's then sending it to my local jmeter GUI (over my slow internet link).
Is that how it should be? I was expecting that only the key measurements would be transported (success, latency, etc).
[PS: I know I can scp my testplan across to the remote box and run it in a headless jmeter. But then I don't see live results...]
This is probably because you are having a "View Tree Results" in your jmeter script. This tells the server to send all responses back to the master so you can view them. That will quickly fill your bandwith and is memory intensive. Try to disable all listeners in your script and only have the "View Aggregate results" one in there.
With high load, you're not interested in individual results. There is also no need to open up your firewall in your datacenter if you use ssl tunneling. I've documented that in a blogpost because I couldn't find a simple step-by-step online, you might want to check that out too at http://rolfje.com/2012/02/16/distributed-jmeter-through-vpn-and-ssl/
I would recommend you to write your results to a file. Before doing so, you can optionally perform a select / deselect of result save configuration by clicking on configure button and/or by checking write success/error to log. By unselecting result sampler data, save response messages etc you can reduce the amount of data being sent to your console.
I THINK the way you have it set up, both the Local instance of Jmeter and the remote are running the script at the same time, hence traffic is being directed to your local box.
What I've found to be more effective is to remote INTO the machine running JMeter. Everything is then local to the proper machine, and I don't have to worry about bottle necks.
I would recommend having a simple data writer element that saves to a file on the REMOTE linux box.
In the GUI, you could still have just the aggregate report to keep an eye on what's happening.
Hope this helps. If not, I would recommend posting to the JMeter user group:
http://jmeter.512774.n5.nabble.com/