Let me make a statement first: I'm new to Kubernetes, please take it easy if I'm asking wrong questions.
Ok, here is what I'm gonna do. I'm planning to build a Kubernetes for my project using some physical machines.
I have 1 server for master and 2 worker nodes. My service dockers (pods) will be allocated by Kubernetes master, they will need storage for the database (MySQL).
After searching around, I came up with a solution of Persistent Volume, but I don't want to use those online cloud services out there such as Google Cloud or Azure Cloud, etc. It leads me to another solution - Local Persistent Volume (LPV), this is where I stuck currently.
The problem with LPV is it's attached with a specific node, so I wouldn't be able to replicate (backup) the database on other nodes, if something happens to this node, or something wrong with the physical disk, I'm gonna lose all the databases, right?
The question is, are there any solutions to set up replication on the database using Local Persistent Volume? For example, I have a database on Node 1, and a backup version on Node 2, so when Node 1 is not available, the pods will mount to the backup database on Node 2.
Thanks in advance!
You can deploy the database as statefulset using local volumes on nodes.Just create the volumes and put them in a StorageClass
For backup , you need to setup the replication on the database level ( not volume level ) to some other cluster /other database instance running somewhere else/or on some other cluster.
Pod failures are handled by kubernetes anyway , it will restart the pod if goes unhelathy.
Node failures can't be handled in statefulset ( one node can't replace another , in other words , in statefulset a pod will not be restarted on other node , kubernetes will wait for node to come back )
If you are going for simple single pod deployement rather than statefulset , you can deploy the database as single pod and another instance as single pod and use node selector to run those on different nodes , then setup the replication from one instance to another instance on database level , and configure your client app to failover to fallback instance in case the primary is not available , this needs to be synchronous replication.
Links:
Run a Single-Instance Stateful Application ( MYSQL)
Run a Replicated Stateful Application ( MYSQL )
Related
I am running database inside Kubernetes pod. i am planning to run K8s job to take automatic backup of databases from pod.
There is also i can write shell script to take snapshot of volume(PV).
which method will be better to use? in emergency which one will save time restore data ?
You can use Stash by AppsCode which is a great solution to backup Kubernetes volumes.
For supported versions check here
Stash by AppsCode is a Kubernetes operator for restic. If you are
running production workloads in Kubernetes, you might want to take
backup of your disks. Traditional tools are too complex to setup and
maintain in a dynamic compute environment like Kubernetes. restic is a
backup program that is fast, efficient and secure with few moving
parts. Stash is a CRD controller for Kubernetes built around restic to
address these issues.
Using Stash, you can backup Kubernetes volumes mounted in following
types of workloads:
Deployment, DaemonSet, ReplicaSet, ReplicationController, StatefulSet
After installing stash using Script or HELM you would want to follow
Instructions for Backup and Restore if you are not familiar
I find it very useful
Migrating a Postgres database from Heroku to Google Cloud in a Kubernetes and Docker setup.
Trying to decide what is a better approach.
1st approach - Use a persistent disc on the VM that is used by a deployed Postgres instance in the Kubernetes cluster.
2nd approach - Use a managed Postgres SQL database that the cluster deployments connect to.
I assume the main differences would be for the maintenance and updating of the database? Are there any big trade-offs of one setup vs the other?
This is an opinion question so I'll answer with an option.
Kubernetes Postgres
Pros:
You can manage your own Postgres cluster.
No vendor lock-in.
Postgres is local to your cluster. (It may not be too much of a difference)
Do your own maintenance.
Raw cost is less.
Cons:
If you run into any Postgres cluster problems you are responsible to fix them.
You have to manage your own storage
No vendor lock-in but you still need to move the data if you decide to switch providers.
You have to do your own backups.
Managed postgres SQL database
Pros:
GCP does it all for you
Any problems will be handled by GCP
Maintenance also handled by GCP.
Storage handled by GCP.
Backups performed by GCP
Cons:
Vendor lock-in
Postgres not local to your cluster.
Will probably cost more.
I'm learning Docker Swarm mode and I managed to create a Swarm locally with a web application and a PostgreSQL database. I can scale them and I see Swarm creating replicas.
I think I understand how Docker Swarm can load balance regular web servers, but how does it deal out of the box with database containers?
Outside of the Swarm context, usually databases have their own ways to deal with replication, in the form of plugins or extended products like MySQL cluster. Other databases like Cassandra have replication built directly into their product.
On a Swarm context, do we still need to rely on those database plugins and features?
What is the expected pattern to handle data consistency between replicas of a database container?
I know it's a very open-ended question, but Docker's documentation is very open-ended too and I can't seem to find anything specific to this.
How does it deal out of the box with database containers?
It doesn't.
There is a pretty good description of Swarm services here: How services work (emphasis mine)
When you deploy the service to the swarm, the swarm manager accepts your service definition as the desired state for the service. Then it schedules the service on nodes in the swarm as one or more replica tasks.
Swarm has no idea what's inside the task, all it knows is how many instances of it there are, whether those instances are passing their health checks, and if there are enough of them to satisfy the task definition you gave it. The word overlap between this and database replicas is a little unfortunate, but they are different concepts.
What is the expected pattern to handle data consistency between replicas of a database container?
Setting up data replication is on you. These are probably as good a place to start as any
How to Set Up PostgreSQL for High Availability and Replication with Hot Standby
PostgreSQL Replication Example
Docker swarm currently scales well for the stateless applications. For database replication, you have to rely on every database's own replication mechanism. Swarm could not manage the datatbase replication. The volume or file system level replication could provide the protection for a single instance database, but are not aware of database replication/cluster.
For databases such as PostgreSQL, the additional works are required. There are a few options:
Use host's local directory. You will need to create one service for every replica, and use constraint to schedule the container to one specific host. You will also need custom postgresql docker image to set up the postgresql replication among replicas. While, when one node goes down, one PostgreSQL replica will go down. You will need to work to bring up another replica. See crunchydata's example.
Use the volume plugin, such as flocker, REX-Ray. You will still need to create one service for every replica, and bind one volume to one service. You need to create all services in the same overlay network and configure the PostgreSQL replicas to talk with each other via the dns name (the docker service name of the replica). You will still need to set up the postgresql replication among replicas.
With the introduction of the new 'swarm mode' with Docker 1.12, we've been trying to migrate our application on containers and make use of the swarm mode's orchestration & clusters.
Our application requires some initial database scripts to be run for it to start.
We're not packaging the database inside our dockerized application so that it could follow a stateless microservice architecture and multiple containers would eventually talk to a single (at the moment) database instance.
While creating the service, we cannot use --replicas with the create service command as multiple instances would try and create tables on a single database and fail. Although our scripts would check if the database has been set-up and skip the creation but since all containers start simultaneously, it could not be used.
We couldn't find any wait-for kind of mechanism that we could leverage with dockers for this issue. It would have been good if we could only start the second container when the first one had created the database (and exposed the ports) but how can we configure inter-container communication for this?
Alternatively, can tools like flywaydb help in some way?
How should this be used in production?
From the Flyway FAQ:
Can multiple nodes migrate in parallel?
Yes! Flyway uses the locking technology of your database to coordinate multiple nodes. This ensures that even if even multiple instances of your application attempt to migrate the database at the same time, it still works. Cluster configurations are fully supported.
There is no easy way to coordinate this among containers. It basically requires a distributed lock solution. The first container that gets the lock could create db, while, other containers that not get the lock need to wait.
In AWS, you could leverage DynamoDB for it. DynamoDB supports conditional update. The container first tries to create the lock key in DynamoDB with "attribute_not_exists(yourKey)". The first creation will succeed and other creations will be rejected. The first container needs to create another key in DynamoDB to indicate the db is ready. Other containers simply waits till the ready key is created.
Or you could do it in your service deployment script. The script could create the service with 1 replica. Then keep checking if db is created. If yes, scale the service, such as docker service update yourservicce --replicas 5.
I'm researching cloud services to host an e-commerce site. And I'm trying to understand some basics on how they are able to scale things.
From what I can gather from AWS, Rackspace, etc documentation:
Setup 1:
You can get an instance of a webserver (AWS - EC2, Rackspace - Cloud Server) up. Then you can grow that instance to have more resources or make replicas of that instance to handle more traffic. And it seems like you can install a database local to these instances.
Setup 2:
You can have instance(s) of a webserver (AWS - EC2, Rackspace - Cloud Server) up. You can also have instance(s) of a database (AWS - RDS, Rackspace - Cloud Database) up. So the webserver instances can communicate with the database instances through a single access point.
When I use the term instances, I'm just thinking of replicas that can be access through a single access point and data is synchronized across each replica in the background. This could be the wrong mental image, but it's the best I got right now.
I can understand how setup 2 can be scalable. Webserver instances don't change at all since it's just the source code. So all the http requests are distributed to the different webserver instances and is load balanced. And the data queries have a single access point and are then distributed to the different database instances and is load balanced and all the data writes are sync'd between all database instances that is transparent to the application/webserver instance(s).
But for setup 1, where there is a database setup locally within each webserver instance, how is the data able to be synchronized across the other databases local to the other web server instances? Since the instances of each webserver can't talk to each other, how can you spin up multiple instances to scale the app? Is this setup mainly for sites with static content where the data inside the database is not getting changed? So with an e-commerce site where orders are written to the database, this architecture will just not be feasible? Or is there some way to get each webserver instance to update their local database to some master copy?
Sorry for such a simple question. I'm guessing the documentation doesn't say it plainly because it's so simple or I just wasn't able to find the correct document/page.
Thank you for your time!
Update:
Moved question to here:
https://webmasters.stackexchange.com/questions/32273/cloud-architecture
We have one server setup to be the application server, and our database installed across a cluster of separate machines on AWS in the same availability zone (initially three but scalable). The way we set it up is with a "k-safe" replication. This is scalable as the data is distributed across the machines, and duplicated such that one machine could disappear entirely and the site continues to function. THis also allows queries to be distributed.
(Another configuration option was to duplicate all the data on each of the database machines)
Relating to setup #1, you're right, if you duplicate the entire database on each machine with load balancing, you need to worry about replicating the data between the nodes, this will be complex and will take a toll on performance, or you'll need to sacrifice consistency, or synchronize everything to a single big database and then you lose the effect of clustering. Also keep in mind that when throughput increases, adding an additional server is a manual operation that can take hours, so you can't respond to throughput on-demand.
Relating to setup #2, here scaling the application is easy and the cloud providers do that for you automatically, but the database will become the bottleneck, as you are aware. If the cloud provider scales up your application and all those application instances talk to the same database, you'll get more throughput for the application, but the database will quickly run out of capacity. It has been suggested to solve this by setting up a MySQL cluster on the cloud, which is a valid option but keep in mind that if throughput suddenly increases you will need to reconfigure the MySQL cluster which is complex, you won't have auto scaling for your data.
Another way to do this is a cloud database as a service, there are several options on both the Amazon and RackSpace clouds. You mentioned RDS but it has the same issue because in the end it's limited to one database instance with no auto-scaling. Another MySQL database service is Xeround, which spreads the load over several database nodes, and there is a load balancer that manages the connection between those nodes and synchronizes the data between the partitions automatically. There is a single access point and a round-robin DNS that sends the requests to up to thousands of database nodes. So this might answer your need for a single access point and scalability of the database, without needing to setup a cluster or change it every time there is a scale operation.