I usually have the following development branches:
Production (corresponding to the main server, such as www.example.com
Development/Staging (corresponding to a test server, such as test.example.com)
Local (the user will just use local resources, no server required)
I have a Cloud Spanner instance for Production, and for Local I use the Spanner Emulator.
What is the suggested best practice for the Development/Staging database? Is it considered bad practice to have a second database within the primary Instance, such as:
my-spanner-instance (Instance)
prod (Database)
dev (Database)
First of all: There's not one correct answer to this, as it really depends on your specific needs and requirements.
But in general I would say:
Having a shared instance for production and development/staging will save you some costs. It will however also mean that for example load from tests and experiments on the staging database could have an impact on your production environment.
Having separate instances for production and development/staging would make the setup more like your traditional setup with a separate test server, as development/staging would physically and logically be separated from your production environment.
Although the analogy is not 1-on-1, when comparing your Cloud Spanner to setup to a more traditional setup using servers, the best comparison would be to consider a Cloud Spanner instance a 'server'. So if in a traditional setup you would always use a separate (virtual) server for your staging environment, then it would also make sense to do so in your Cloud Spanner setup.
Related
Assuming we are using a micro service architecture for a product and we decide to use 'Database per service' model, and deploy in cloud servers by provider like AWS.
It is convenient to have databases running as a container for development and test environments.
But can same be implemented for Production environment! If so, how safe it would be?
Or is it proper to go with cloud solution as AWS RDS-DB instead!!
This blog post lists some reasons why you should not run production databases in containers. It also references another blog post describing problems with updating docker and unstable storage drivers.
The main points here for me boil down to this:
Dodgy storage drivers. This may be less of a problem when you write your database state to the host system but Docker for example explicitly encourages users to use volumes for exactly that (see the docs: Citation: "Volumes are the best way to persist data in Docker"). It may just work fine under normal circumstances, but what about the edge-cases like power-failures or read-errors for example?
Managing databases in production is hard. Many companies employ full-time DBAs to ensure smooth operation of production databases. The devops paradigm (every dev creates a plethora of DB servers in containers) makes it nearly impossible for a DBA to do his job. That is if the DBA even has access to these DBs.
In conclusion: Containers are fine for certain tasks and a bad idea for others. Running production databases in containers is one of those bad ideas.
We containerise our db in production (on-premises enterprise application). Many do. It's perfectly stable and the deployment is much simplified. Of course our db is not under stress; we're dealing with hundreds of concurrent users, not tens of thousands. We just make sure that the container has enough RAM and is monitored well.
If we did need to dedicate an entire VM to the db alone, then yes I would skip docker.
According to link below, it is not a good idea to use database container in Production.
But as I have experienced; if you isolate your container from your app and update your container regularly and also manage networking stuff, there seems to be no problem.
Link: https://www.quora.com/Is-it-not-advisable-to-use-database-in-Docker-container
As you are using Database Per Service Model for Microservice, in Production perfect solution can be AWS RDS instance for database, Now you have 2 approaches :
You can create single RDS Instance and can have different databases for different services on same RDS insatnce, it will save cost a lot but you need to take care of database connections and load you will be having on database based on that you have to choose RDS instance type like 4xlarge etc, better the instance type more connection it will provide and more database load it can handle effectively.
Second solution can be creating several RDS instance and number of RDS instance will be equivalent to your microservice count as each service will be using one RDS instance for its database independently, this is not the effective solution, it will incur lot of cost and this solution will under utilize AWS RDS instances.
What is the best practice to deploy database in microservices architecture, more precisely in distributed environment, such as docker swarm? Microservices principles states each service should be stateless to enable scaling. As database obviously has a state, should it live at fixed position outside of cluster, deployed and configured before the cluster is initialized?
I'm confused, because all docker compose examples includes database container in the service definition. But things aren't that simple. Often the database needs a lot of configuration before it's ready to use. Also, docker sucks at coordinating the service starting order.
If it's really a good practice to deploy the database alongside with services to docker swarm, how to ensure consistency and persistence of cricial data?
This is a good question and one I think a lot of people are still thinking through as far as best practices are concerned. The answer really depends on your needs. There are several ways to crack this nut but these are the two I'm using right now:
Running the database in the typical manner on dedicated machine(s) with replication, etc
I am currently experimenting with running the database as a service on a Docker Swarm cluster with the data persisted across the cluster with GlusterFS
I have three machines in the cluster labeled as database machines
These database machines all run a GlusterFS container providing the GlusterFS capabilities
When the database service is started I map the GlusterFS share into the container and specify that the service should only run on a machine labeled as a database node. With this setup it doesn't matter which node the database service starts on and if a machine fails the database service is automatically migrated to another node labeled as a database node. The GlusterFS replication of data ensures the integrity of the persisted data.
As mentioned, it is my understanding that there is still a lot of experimentation going on with this and 'best practices' are not entirely established. Those best practices will ultimately depend on your needs and risk tolerances.
I have an application in Rails that displays a lot of information to the user.
Using new relic, I notice that the database is working intensively and that this will probably limit my ability to scale (assume for now that the SQL is fine)
Is there a way I can have several databases which will be in sync, and the requests will be load-balanced between them?
Does Heroku provide such a system?
Maybe more importantly - Should I rely on Heroku for an app which needs to scale? (is the architecture one web server connects to one database server or can it do more?)
Look in to heroku follower database.
https://devcenter.heroku.com/articles/heroku-postgres-follower-databases
It will keep your database sync and for load balancing you will need to configure octopus.
Moreover regarding scalability its quite easy (application level scalability just increase the dynos) and on database they are having multiple models (with different cache sizes) and its quite ease with to switch between these models (with ignoreable down time)
thanks
I'm researching cloud services to host an e-commerce site. And I'm trying to understand some basics on how they are able to scale things.
From what I can gather from AWS, Rackspace, etc documentation:
Setup 1:
You can get an instance of a webserver (AWS - EC2, Rackspace - Cloud Server) up. Then you can grow that instance to have more resources or make replicas of that instance to handle more traffic. And it seems like you can install a database local to these instances.
Setup 2:
You can have instance(s) of a webserver (AWS - EC2, Rackspace - Cloud Server) up. You can also have instance(s) of a database (AWS - RDS, Rackspace - Cloud Database) up. So the webserver instances can communicate with the database instances through a single access point.
When I use the term instances, I'm just thinking of replicas that can be access through a single access point and data is synchronized across each replica in the background. This could be the wrong mental image, but it's the best I got right now.
I can understand how setup 2 can be scalable. Webserver instances don't change at all since it's just the source code. So all the http requests are distributed to the different webserver instances and is load balanced. And the data queries have a single access point and are then distributed to the different database instances and is load balanced and all the data writes are sync'd between all database instances that is transparent to the application/webserver instance(s).
But for setup 1, where there is a database setup locally within each webserver instance, how is the data able to be synchronized across the other databases local to the other web server instances? Since the instances of each webserver can't talk to each other, how can you spin up multiple instances to scale the app? Is this setup mainly for sites with static content where the data inside the database is not getting changed? So with an e-commerce site where orders are written to the database, this architecture will just not be feasible? Or is there some way to get each webserver instance to update their local database to some master copy?
Sorry for such a simple question. I'm guessing the documentation doesn't say it plainly because it's so simple or I just wasn't able to find the correct document/page.
Thank you for your time!
Update:
Moved question to here:
https://webmasters.stackexchange.com/questions/32273/cloud-architecture
We have one server setup to be the application server, and our database installed across a cluster of separate machines on AWS in the same availability zone (initially three but scalable). The way we set it up is with a "k-safe" replication. This is scalable as the data is distributed across the machines, and duplicated such that one machine could disappear entirely and the site continues to function. THis also allows queries to be distributed.
(Another configuration option was to duplicate all the data on each of the database machines)
Relating to setup #1, you're right, if you duplicate the entire database on each machine with load balancing, you need to worry about replicating the data between the nodes, this will be complex and will take a toll on performance, or you'll need to sacrifice consistency, or synchronize everything to a single big database and then you lose the effect of clustering. Also keep in mind that when throughput increases, adding an additional server is a manual operation that can take hours, so you can't respond to throughput on-demand.
Relating to setup #2, here scaling the application is easy and the cloud providers do that for you automatically, but the database will become the bottleneck, as you are aware. If the cloud provider scales up your application and all those application instances talk to the same database, you'll get more throughput for the application, but the database will quickly run out of capacity. It has been suggested to solve this by setting up a MySQL cluster on the cloud, which is a valid option but keep in mind that if throughput suddenly increases you will need to reconfigure the MySQL cluster which is complex, you won't have auto scaling for your data.
Another way to do this is a cloud database as a service, there are several options on both the Amazon and RackSpace clouds. You mentioned RDS but it has the same issue because in the end it's limited to one database instance with no auto-scaling. Another MySQL database service is Xeround, which spreads the load over several database nodes, and there is a load balancer that manages the connection between those nodes and synchronizes the data between the partitions automatically. There is a single access point and a round-robin DNS that sends the requests to up to thousands of database nodes. So this might answer your need for a single access point and scalability of the database, without needing to setup a cluster or change it every time there is a scale operation.
Sometimes we have upwards to 4-6 people either RDPed looking at data in SQL Management Studio, or hitting the server with LINQpad, Toad, etc from various locations while developing in mostly ASP.NET and Flex with WebOrb. Is this bad? Bad in the sense that we are trying to keep our live production app stable and as lag free as possible for global users?
i don't think i'd do that. if it was just me, then sure:) but if there's a bunch of people god only knows what queries they might run. we always use a test server for such things.
best regards,
don
Best practice would be separate servers. Next best, separate instances on same server. Next best, separate databases on a instance.
However, I wouldn't be letting any developers RDP into a production SQL Server (or production anything), regardless of choice of segregation mechanism. Use a separate terminal server with tools and everything there.
You can have dev and prod db on the same instance. Just make sure the permission are setup so that developers cannot touch the prod db. The negative is a long running query in dev will impact prod.
In SQL SERVER 2005 a better solution is to have a dev "instance" and a prod "instance".
Then is someone mis-behaves on the dev instance you and just bring down that insance.
In SQL server 2008 you can setup up CPU usage plans which can help throttle how much resources can be used. You should investigate that.
It depends on a lot of variables. It's generally better to have them on different servers. This is really depending on how you use sql server. If you just have databases, don't use a lot of the management tools, like nightly processes to alter data and other jobs you might be OK. You are running a real risk of having bleed over code from developing on the dev database to the production one though. It's safer to have them separated out, especially for the small amount of money needed to create a dev instance of sql server.
I find this a poor practice for several reasons:
First suppose one of your devs messes up and does something that ends up taking all of the processing power of your server. Oops prod is down for no good reason.
Second, devs could easliy change the wrong database. Oops prod is down for no good reason. At least you can avoid this by not giving any production rights to devs (which you should be doing anyway as a best practice.)
Third, if the database is the on the same server it has to have a different name, this can make moving things to prod difficult and error prone. I think it also means it will be less likely that you deploy correctly through source controlled scripts. If you choses to copy objects from one database to the other, then you can have issues with that as well. First if there is data in the object already, you may accidentally wipe it out (hope you have a backup) or you may move the new table structure but miss things like the PKs and FKS and default values and triggers and constraints and indexes or the wizard might take much longer to do the move because in the background it is creating and populating a new table and then droping the old and renaming the new one rather than using alter table. Oops prod is down or seriously slowed for no good reason.
I tend to agree with the "separate servers" folks, although with my company we actually do most of our day to day development work on our local machines -- so we have SQL Server installed locally. This can be a pain, of course, if you're developing reporting or something that needs production data. In that scenario, developers here usually get a subset of production data exported to work with.
For acceptance testing vs. deployment though, we do use separate instances.
Developers probably shouldn't have production access UNLESS they're also the ones who do application deployments (as can be the case with small teams like the one I'm in). If you do end up using separate DBs on the same server, I would at least lock down RDP access and grant access to each development DB on an individual basis. That's how it works here -- I don't have admin rights to any of our servers at this time, and can only admin databases for applications that belong specifically to my team.
It depends how much you value your live service. I know I wouldn't trust me and my fat hands running SQL on the same hardware as a live application.
Even if the application is not business critical, and the app is not data-bound, you can set up a development environment on an unused desktop machine, so why wouldn't you do that instead of take the risk?
The set up I use is typically DEV database on a local instance of SQL Server (Development Version for me, but Express would probably also work), a QA database on a test instance of SQL Server. In our environment, this is located on a virtual instance of W2K3 -- soon to be W2K8. Production databases live either on dedicated instances of SQL server or on one of various clustered instances. We don't mix PROD/QA/DEV at all. I use RedGate SQL Compare to synchronize schemas between the various systems, including different developer instances of the database.
It will be 'OK' as much as the team don't had any administrator privileges over the server (either SQL or Windows), and their user log-ins just grant access to potentially destroy just the development database and it's associated files, having denied access to production databases
For other application testing reasons, we created a copy of our production server (which is a virtual server) on a separate domain. This allowed the Windows Server Name, SQL Serer Name, Database name to be exactly the same (lots of settings on 3rd party apps require this level of configuration to get different processes to work.). Now we can rebuild a test environment by creating an exact virtual image of our production server.
I was sceptical about running SQL Server on a virtual machine, but it has given our small company a lot of flexibility. We like to think our databases are critical, but it is for internal uses and having some down time would just have workers shift their lunch hour.