I recently had to add clickhouse to our technology stack but unfortunately I didn't find any good, simple and fast tutorials for my needs and after some trial and error I could manage to do it by myself. To help others I decided to share my experience.
So how to deploy and configure a remote ClickHouse DB instance with docker?
ClickHouse setup
This is a setup guide for deploying ClickHouse with docker on remote servers.
Installation
You have to have docker preinstalled on your remote system.
Server
Run the following command:
$ docker run -d --name some-clickhouse-server -p 8123:8123 --ulimit nofile=262144:262144 --volume=$HOME/some_clickhouse_database:/var/lib/clickhouse yandex/clickhouse-server
ClickHouse server uses port 8123 as the default port but you can uses any other open port but remember to expose the port to the external network. The server comes with a default users with no password.
Client
Run the following command in the server to connect to the clickhouse server with the default user.
$ docker run -it --rm --link some-clickhouse-server:clickhouse-server yandex/clickhouse-client --host clickhouse-server
Configuration
Open the clickhouse server docker container
$ docker exec -it some-clickhouse-server bash
1. Enable SQL-driven access control and account management for the default user.
In the clickhouse server docker container:
$ cd etc/clickhouse-server
Comment out the following line in user.xml file to enable access control:
<access_management>1</access_management>
Note that this operation is unsafe and after finishing you work, you should change the access control to :
<access_management>0</access_management>
2. Listening to other networks:
In the etc/clickhouse-server/config.xml comment out <listen_host>::</listen_host> to allow remote connections.
You should see the port is open in the systems network:
root#myvm:~# lsof -i :8123
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
docker-pr 1141768 root 4u IPv6 53989091 0t0 TCP *:8123 (LISTEN)
Creating a DB with users
In this part we create a simple database and a users. Then we Grant privileges of the database to that user.
There are two types of synchronization and they can complement each other:
1. Creating an users
CREATE USER IF NOT EXISTS user1 IDENTIFIED WITH PLAINTEXT_PASSWORD BY 'pass1'
You can check the users with the SHOW USERS command.
2. Creating a database
CREATE DATABASE IF NOT EXISTS db1
You can check the databases with the SHOW DATABASES command.
3. Grant database privileges to the user
You can grant limited privileges or all privileges to a users.
GRANT ALL PRIVILEGES ON db1.* TO user1
4. Connect with the new users to the database
Now we can connect to the server with the created account.
$ docker run -it --rm --link some-clickhouse-server:clickhouse-server yandex/clickhouse-client --host clickhouse-server -u user1 --password pass1
Make sure the users has all the permissions and has access to the databases:
:) SHOW GRANTS
:) SHOW DATABASES
Creating a Sample Table
Clickhouse supports SQL .
To create a table:
:) USE db1
:) CREATE TABLE names (
id UInt64,
name String,
created DateTime
) ENGINE = MergeTree()
PRIMARY KEY id
ORDER BY id;
:) SHOW TABLES
There you have it. You can connect to the databases from other networks with the clickhouse-client and your user/password.
Please clone this project and follow the instructions as per the README.md file. After running the docker compose you will have 2 shards and 2 replica setup of clickhouse.
https://github.com/vinitk95/clickhouse-cluster.git
Related
I have a server that uses neo4j 3.5.x with docker. Now I want to move that database to another server.
This time I see that neo4j released 4.0. I just copied data folder which contains only graph.db
I run the script I used last time
sudo docker run --name vis --restart unless-stopped --log-opt max-size=50m --log-opt max-file=10 -p3001:7474 -p3002:7473 -p3003:3003 -d -v /data:/data -v /conf:/conf -v /logs:/logs --env NEO4J_AUTH=none neo4j
When I run this I see that I can reach it from 7474 which is fine. BUT it asks password to see the data. Though I didn't set a password WHY IT ASKS?
I tried everything possible like neo4j, 123, 1234, test or live it empty. none worked.
it gives error
neo4j-driver.chunkhash.bundle.js:1 WebSocket connection to 'ws://0.0.0.0:7687/' failed: Error in connection establishment: net::ERR_ADDRESS_INVALID
Is there a proper/robust way to import data between neo4j database servers? Can I use this https://neo4j.com/developer/kb/export-sub-graph-to-cypher-and-import/
If you go to the Neo4j desktop, and then select the graph. Open up Manage options. Then choose Open Terminal.
Once there you can use the database backup command. (Here is an example)
bin\neo4j-admin backup --backup-dir=c:/backup --name=graph.db-20200107
This will backup the database to the specified backup directory.
Then you can zip that backup directory, and then unzip the backup directory, and restore it on the new server.
(Here is an example)
bin\neo4j-admin restore --from=c:/backup/graph.db-20200107 --database=graph.db --force=true
Note: The 'graph.db-20200107' is an example of the name that you give the database backup. You can name it whatever you want.
-yyyguy
I am new to docker concept. I have followed this to start and able to install db2 under docker environment.
My questions are
1)i need to load data in to this docker based db2. Data dump is from the db2 instance present remotely in linux machines.How can i load the dump from the docker db2 console?
2)every time i start docker quick start terminal i need to do all the procedure done in above link again. can't i just run db2 container and go to db2 console?
3) su - db2inst1(from link) is asking for password non of attempts to that password succeed(will give my machine admin password,db2 password etc).i need to restart or start again the process to go in to db2 container.
I have a problem with my database. I installed postgreSQL 9.5 on my Ubuntu server. I changed the postgresql.conf file to allow binding the postgreSQL server to the localhost. This allows me to run pgAdmin and connect to my database by forwarding also the port 5432, where I run my postgreSQL.
The problem I am experiencing is that I only see the default table 'postgres', but not my newly created one 'games' (I created this table by running create database games with the postgres user connected to the server).
And here is my screen shot of the pgAdmin application with all the property value I use to connect to my server.
As you can see from the first picture I use the same permissions as for the postgres database - it is blank, which should grant access to everyone. I know I have to change that later and limit it to the postgres user I have, but for now I will let it that way. Once I manage to see my 'games' database, then I will start to tighten the security more.
UPDATE I granted all access to the database 'games', which is visible right on the third screen shot down. The access privilege is different. This did not help me, I would still not see the database, when connecting to the server with pgAdmin. A saw someone had a similar problem and run the right click on the server and clicked 'New database'. This seems created a new database, because as you can see from the pgAdmin, the application manage to find the score table I create inside pgAdmin. The reason I believe this is the case is, because running the same SQL connected to the server postgres=# select * from score; results in ERROR: relation "score" does not exist LINE 1: select * from score;.
I manage to find the problem. One of my problems was that I had (unaware of that) installed a postgreSQL server on my machine. Seems I installed it with my pgAdmin install. So everytime I would connect to my server, I would establish a connection to my localhost server and not my remote server. So I just uninstalled the server and installed only the pgAdmin client.
The second problem I had was that the file /etc/postgresql/9.5/main/pg_hba.conf had to be changed. So I run:
sudo vi /etc/postgresql/9.5/main/pg_hba.conf
and changed the line
# Database administrative login by Unix domain socket
local all postgres peer
to
# Database administrative login by Unix domain socket
local all postgres md5
Once that was changed, I had to restart the configuration by executing:
sudo /etc/init.d/postgresql reload
I would also point out that it is important to have postgres user as a unix and DB user with same passwords. I found all this information here.
Try granting access privileges explicitly for your new table.
I believe a blank access privileges column means the table has DEFAULT access privileges. The default could be no public access for tables, columns, schemas, and tablespaces. For more info: http://www.postgresql.org/docs/9.4/static/sql-grant.html
I read some topics on restoring and copying mysql database from 1 server to another
But I wanted to be make sure the impact it might have on my production web app.
So basically here is my situation:
Server A has a database called enrollment.
Server B has a database called enrollment.
Through the command line, how do I do the following:
1.Create a backup copy of 'enrollment' on Server A
2. Drop database enrollment on Server A
3. Copy/Dump database enrollmentt from Server B to Server A( do I need to ssh or copy the sql file or can do i do it throug mysql?)
The databse size is about 64 MB.
While i do the above, how long will the production web app be impacted?
based on my research, this was my thinking, but I wanted to be careful since I am dealing with production data
On server B, mysqldump --databases enrollment > enrollment_backup.sql
scp enrollment_backup.sql from Server B to Server A
drop database enrollment
mysqldump < enrollment_backup.sql
Note: I have root access on server A & server B.
You have to do the drop database in the last step:
1) backup server A
2) dump data A on server B
3) change the web app to point to B
4) if everything is ok you can drop server A
You can dump any remote server to your local one. Or even any remote server to any other remote server.
mysqldump -hproduction -uroot1 -p --add-drop-database --add-drop-table --extended-insert --lock-all-tables --routines --databases mydatabase | \
mysql -hlocalhost -uroot2 -ppassword
This will connect to the server production with user root1 and a password you will enter via a prompt. The database mydatabase (you can give a list of databases as well) will be dumped.
The standard output with all the commands is then redirected to another MySQL server running on localhost (can be any other host as well), using user root2 and the password password. The second password has to be given on the command line.
Advanced possibilities:
If both systems are not in a VPN or secured by any other means, you can create an SSH tunnel and then dump using this tunnel.
If you do not want to pass the password via command line or do not want to enter it at all, let mysqldump or mysql read it from an individual options file via --defaults-extra-file.
Apparently there is a database "postgres" that is created by default on each postgresql server installation. Can anyone tell me or point me to documentation what it is used for?
When a client application connects to a Postgres server, it must specify which database that it wants to connect to. If you don't know the name of a database (within the cluster serviced by the postmaster to which you connect), you can find a list of database names with the command:
psql -l
When you run that command, psql connects to the server and queries pg_database for a list of database names. However, since psql is a Postgres client application, it can't connect to the server without knowing the name of at least one database: Catch-22. So, psql is hard-coded to connect to a database named "postgres" when you run psql -l, but you can specify a template database in that case:
psql -l -d template1
It appears that it does not really have a well-defined purpose. According to the docs:
Creating a database cluster consists of creating the directories in which the database data will live, generating the shared catalog tables (tables that belong to the whole cluster rather than to any particular database), and creating the "template1" and "postgres" databases.
[...]
The postgres database is a default database meant for use by users, utilities and third party applications.
(Source: http://www.postgresql.org/docs/current/app-initdb.html )
There is also the database template0, your safety net when you screw up all others.
postgres is your default database to
connect with.
template1 is your default for
creating new databases, these are
created just like template1
template0 is usefull when template1
is corrupted (wrong settings etc.)
and you don't want to spend a lot of
time to fix this. Just drop
template1 and create a new template1
using the database template0.
The comment above asked: "Is it safe to delete the postgres database if you're not using it?" - CMCDragonkai Oct 22 '16 at 10:37
From the PostgreSQL documentation
After initialization, a database cluster will contain a database named postgres, which is meant as a default database for use by utilities, users and third party applications. The database server itself does not require the postgres database to exist, but many external utility programs assume it exists.
[Note: A database cluster is a collection of databases that is managed by a single instance of a running database server.]
If you are using multiple database connections when creating new databases, then all the connections cannot be done to template1 or template0.
Postgresql will throw an error if the source DB while creating new DB is accessed by other connections.
So for creating new DBs it is better to connect postgres.