Setting UTF-8 in MariaDB server ignores character_set_system - database

I have the following Dockerfile definition file for a MariaDB server:
version: "3.8"
...
services:
database:
command: ["mysqld", "--character-set-server=utf8mb4", "--collation-server=utf8mb4_unicode_ci"]
container_name: mariadb
environment:
MARIADB_DATABASE: sample_database
MARIADB_INITDB_SKIP_TZINFO: "true"
MARIADB_PASSWORD_FILE: /run/secrets/mariadb_user_password
MARIADB_ROOT_PASSWORD_FILE: /run/secrets/root_user_password
MARIADB_USER: mariadb
TZ: "Etc/UTC" # https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
image: docker.io/library/mariadb:10.6-focal # https://hub.docker.com/_/mariadb/
networks:
- global-network
ports:
- "3306:3306"
restart: on-failure
secrets:
- mariadb_user_password
- root_user_password
stdin_open: true
tty: true
volumes:
- "database-volume:/var/lib/mysql"
I'm not quite versed on this topic, but if I would like to check the character- and collation-set used by the server, I usually do execute these queries:
> show variables like 'char%';
+------------------------+--------------------------+
|Variable_name |Value |
+------------------------+--------------------------+
|character_set_client |utf8mb4 |
|character_set_connection|utf8mb4 |
|character_set_database |utf8mb4 |
|character_set_filesystem|binary |
|character_set_results |utf8mb4 |
|character_set_server |utf8mb4 |
|character_set_system |utf8mb3 |
|character_sets_dir |/usr/share/mysql/charsets/|
+------------------------+--------------------------+
> show variables like 'collation%';
+--------------------+------------------+
|Variable_name |Value |
+--------------------+------------------+
|collation_connection|utf8mb4_unicode_ci|
|collation_database |utf8mb4_unicode_ci|
|collation_server |utf8mb4_unicode_ci|
+--------------------+------------------+
My understanding is that the settings that I have: --character-set-server=utf8mb4 and --collation-server=utf8mb4_unicode_ci are the correct ones to set both, the character set and collation to UTF-8.
Now my questions is: how do I get rid of that utf8mb3 value for character_set_system? It should be set to utf8mb4 along with the other values — or that's what I think ;)
Any clues how to set this correctly?
I would like to avoid using any configuration file(s) (like my.cnf) since I'm using the stock Docker image without any modifications.

character_set_system specifies the character set which will be used to store identifiers and other internal information.
It is 3byte utf8, and you cannot change it unless you change sources and recompile MariaDB.
Beginning with 10.6, utf8 was mapped to utf8mb3 (and will be mapped to utf8mb4 in later versions). See MDEV-8334

I am having a similar problem in MariaDB 10.6.5, I was trying to load a dump from AWS and had this error:
ERROR 1253 (42000) at line 26: COLLATION 'utf8mb3_general_ci' is not valid for CHARACTER SET 'utf8mb4'
My config looks like this:
MariaDB [(none)]> SHOW VARIABLES LIKE '%char%';
+--------------------------+----------------------------+
| Variable_name | Value |
+--------------------------+----------------------------+
| character_set_client | utf8mb4 |
| character_set_connection | utf8mb4 |
| character_set_database | utf8mb4 |
| character_set_filesystem | binary |
| character_set_results | utf8mb4 |
| character_set_server | utf8mb4 |
| character_set_system | utf8mb3 |
| character_sets_dir | /usr/share/mysql/charsets/ |
+--------------------------+----------------------------+
and according to this page https://mariadb.com/kb/en/old-mode/ I had to set old-mode to empty in /etc/mysql/mariadb.conf.d/50-server.cnf
old-mode=
Therefore it changed from
MariaDB [(none)]> SHOW VARIABLES LIKE '%old%';
+------------------------------------------+-----------------+
| Variable_name | Value |
+------------------------------------------+-----------------+
| old_mode | UTF8_IS_UTF8MB3 |
+--------------------------+---------------------------------+
to
MariaDB [(none)]> SHOW VARIABLES LIKE '%old%';
+------------------------------------------+-----------------+
| Variable_name | Value |
+------------------------------------------+-----------------+
| old_mode | |
+--------------------------+---------------------------------+
So I've managed to load the SQL dump even if character_set_system was still showing utf8mb3.
HTH somebody.

Related

FATAL: database "Employees" does not exist

Here is the console input-output, where I'm trying to access DB but psql cannot find it. I have tried changing capitalization, but result is the same
(base) username#MacBook-Pro-Ruslan ~ % psql -U username Employees
psql: error: connection to server on socket "/tmp/.s.PGSQL.5432" failed: FATAL: database "Employees" does not exist
But, I have this DB in my Psql app, here the screenshoot of it
UPD: I just checked in command line all the databases
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
----------------+----------------+----------+---------+-------+-----------------------------------
postgres | ruslanpilipyuk | UTF8 | C | C |
ruslanpilipyuk | ruslanpilipyuk | UTF8 | C | C |
template0 | ruslanpilipyuk | UTF8 | C | C | =c/ruslanpilipyuk +
| | | | | ruslanpilipyuk=CTc/ruslanpilipyuk
template1 | ruslanpilipyuk | UTF8 | C | C | =c/ruslanpilipyuk +
| | | | | ruslanpilipyuk=CTc/ruslanpilipyuk
(4 rows)
And it looks like that my psql app has not connected or transferred db between console. How could I get access to this db in app? That I already have. Db that are shown in list in console - are the Dbs that I created manually in console

How to timestamp data copied into a postgresql database

I have a shell script pulling data from a server into a postgresql table.
df -g | awk 'BEGIN{OFS=","}NR>1{$1=$1; print}' > /data/metric.csv
psql -h localhost -d metrics -U postgres -c "copy tablename from STDIN with delimiter as ',';" < /data/metric.csv
Displays as:
filesystem | gb_blocks | free | %used | iused | %iused | mounted_on
/dev/hd2 | 16.75 | 12.60 | 25% | 79098 | 3% | /usr
/dev/hd9var | 8.00 | 6.00 | 25% | 11965 | 1% | /var
/dev/hd3 | 36.75 | 18.83 | 49% | 5614 | 1% | /tmp
/dev/hd1 | 3.25 | 3.11 | 5% | 674 | 1% | /home
/dev/hd11admin | 0.25 | 0.25 | 1% | 16 | 1% | /admin
/proc | - | - | - | - | - | /proc
I'm working with Postgresql on an Ubuntu OS and pulling the info from an AIX server. I'd like to add a column with a timestamp for every time new data is added to the table because right now it just all blends together. I've tried to add another column for timestamp and give it a timestamp value but the timestamp isn't in the csv file and I'm not sure how to add it either. I appreciate an help I can get to solve this.
Create table and add date column with Default value like current_Date/now() ).
CREATE TABLE IF NOT EXISTS metrics
(
filesystem text ,
gb_blocks text ,
free text ,
per_used text ,
iused text ,
per_iused text ,
mounted_on text ,
load_dttm timestamp without time zone DEFAULT now()
);
mention columnswith table while loading data as below command
psql -h localhost -d metrics -U postgres -c "copy metrics(filesystem,gb_blocks,free,per_used,iused,per_iused,mounted_on) from STDIN with delimiter as ',';" < /data/metric.csv

Can't connect to postgres on ubuntu

I have a ubuntu 20 on dreamcompute (which is cloud computing).
I create a user and a database. Here is the list of database and users (for some reason, I can't see database under a matt username).
I went into:
nano /etc/postgresql/13/main/postgresql.conf &
nano /etc/postgresql/13/main/pg_hba.conf and did the whole '*' and '0.0.0.0/0'
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
strapi | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | hossein=CTc/postgres
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
as you can see you can't see Superuser and database strapi under the matt username.
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
matt | | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
ubuntu | | {}
I'm using my dream compute's ip address as my host and use my database and user and password but get error message: connection attempt time out.
Could someone please give me a pointer on why this is happening? I have been working onthis for 2 weeks now and I can't get it to work.
Error message is connection time out. That usually means that the port is blocked by a firewall. Check your cloud provider firewall settings and iptablesin your Linux box in case you have installed it.
If there was problem with permissions, the error message would be something else.
as ex4 mentioned above I needed to reachout to the company that I was renting my cloud computer from but I still could not connect to the database.
The way I went around it is that you can ssh into your database and then connect to your database as a localhost since you are ssh into your cloud computer.
In DBeaver you have a ssh tab and you can connect and then you got back to your postgres tab and fill the localhost, user, database name, and user password area and simply click connect.
Sadly this took weeks to come to this :/

MemSQL database not available for reads while leaf node is in replicating mode

I have a CentOS server running a local memsql cluster (aggregator and leaf on the same machine). I have a databse named offers. For some reason, I cannot execute any queries against tables in my database.
Everything was working fine until I tried to add another machine to the cluster. I had the IT team at my place replicate the server I was working on (completely). I went over to the replicated server, deleted the database in question and then registered the server using the memsql-toolbox-config register-node command. Then the database showed it was under the transition state. I restarted memsql using memsql-ops and got to this situation.
Running a simple query yields:
memsql> select * from table;
ERROR 2261 (HY000): Query `select * from table` couldn't be executed because of an in progress failover operation. Check the status of the leaf nodes in the cluster (error 1049:'Leaf Error (172.26.32.20:3307): Unknown database 'offers_5'')
The output for the the cluster status command is:
memsql> show cluster status;
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
| Node ID | Host | Port | Database | Role | State | Position | Master Host | Master Port | Metadata Master Node ID | Metadata Master Host | Metadata Master Port | Metadata Role | Details |
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
| 1 | 172.26.32.20 | 3306 | cluster | master | online | 0:181 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 1 | 172.26.32.20 | 3306 | offers | master | online | 0:156505 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 2 | 172.26.32.20 | 3307 | cluster | async slave | replicating | 0:180 | 172.26.32.20 | 3306 | 1 | 172.26.32.20 | 3306 | Reference | stage: packet wait, state: x_streaming, err: no |
| 2 | 172.26.32.20 | 3307 | offers | sync slave | replicating | 0:156505 | 172.26.32.20 | 3306 | 1 | 172.26.32.20 | 3306 | Reference | |
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
4 rows in set (0.00 sec)
So it seems that the the second node is replicating. Also note the details column saying:
stage: packet wait, state: x_streaming, err: no
Running the replication status command gives:
memsql> show replication status;
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
| Role | Database | Master_URI | Master_State | Master_CommitLSN | Master_HardenedLSN | Master_ReplayLSN | Master_TailLSN | Master_Commits | Connected | Slave_URI | Slave_State | Slave_CommitLSN | Slave_HardenedLSN | Slave_ReplayLSN | Slave_TailLSN | Slave_Commits |
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
| master | cluster | NULL | online | 0:181 | 0:181 | 0:177 | 0:181 | 86 | yes | 172.26.32.20:3307/cluster | replicating | 0:180 | 0:181 | 0:180 | 0:181 | 84 |
| master | offers | NULL | online | 0:156505 | 0:156505 | 0:156505 | 0:156505 | 183 | yes | 172.26.32.20:3307/offers | replicating | 0:156505 | 0:156505 | 0:156505 | 0:156505 | 183 |
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
2 rows in set (0.00 sec)
I never initiated any fail over or replication. Anyone knows why this is happening? How could I solve this?
EDIT:
Using memsql-ops I get:
[me#memsql ~]$ memsql-ops memsql-list
ID Agent Id Process State Cluster State Role Host Port Version
33829AF Af13af7 RUNNING CONNECTED MASTER 172.26.32.20 3306 6.5.18
BBA1B61 Af13af7 RUNNING CONNECTED LEAF 172.26.32.20 3307 6.5.18
But with memsql-admin, with the new memsql tools:
[me#memsql ~]$ memsql-admin list-nodes
✘ Failed to list nodes on all hosts: failed to list nodes on 1 host:
172.26.32.20
No nodes found
Making my question a bit clearer - How can I get my server to respond to queries again? And after I do, How should I act to add another host? Should I clean the replicated server completely of any memsql data?
2nd EDIT:
I managed to solve this problem by delete my database and cluster data, and setting up a new one using the new MemSQL tools, throwing away MemsqlOps. Read my answer.
It looks like there are a couple things that might be causing problems. Generally speaking, cloning a memsql server is not something that is supported nor the best way to go about adding nodes. It also looks like you may be using both the older Ops management tool and the newer MemSQL tools. I would recommend not installing or using Ops and sticking to just the new MemSQL tools instead.
A good place to start would be to try recreating the nodes after cloning; a cloned memsql node won't correctly become part of the cluster. You should also verify that you don't have more than one master aggregator in the cluster. If you can start with that and see if it resolves your issues I'm happy to help with any other problems that you run into.
I managed to set up a working cluster.
As micahbhakti mentioned in his answer, I tried using only the newer MemSQL tools, instead of the deprecated MemSQL Ops. It required deleting the MemSQL agent existing on both servers and then following the tutorial in the MemSQL documentation. Here are the steps I took for anyone struggling with this issue which is better described as: My MemSQL-Ops-managed-MemSQL-cluster is not responding. How can I upgrade it to a working MemSQL-tools-managed-cluster?
1. Save what data you can
The following step is to delete all memsql data, so it would be best if you could save your data. The table data could be stored in CSV files easily with a simple
SELECT * FROM important_data_containing_table INTO OUTFILE '/home/yourfolder/yourcsvfile.csv';
Do this for all tables containing important data. You could also save the scheme itself. You can do that by viewing and copying to another file all the create queries you used to create the table originally, to re-execute them later. Use this
SHOW CREATE TABLE your_table_name
The documentation for mysql is described here. It might not be similar to the syntax used in mem, but the above base command works. For exact information, read about MySQL Features Unsupported in MemSQL.
2. Delete anything to do with Memsql-Ops
As it is said here about the uninstall command:
Stops the local MemSQL Ops agent and deletes all its data.
If MemSQL nodes are already installed in the local host, this command will prompt users to delete those nodes first before proceeding with the uninstall.
And indeed, if there is a node runnning (in my case there were), you will be prompted to run another command to delete those nodes: memsql-ops memsql-delete --all. This WILL delete all data in your database as said in it's documentation:
Deletes all data for a MemSQL node. This operation is not reversible and may lead to data loss. Users who want to perform this operation are prompted to explicitly type ‘DELETE’ to be sure of their decision.
That's why I asked you to save what ever you need :)
This should be done for each host you want to include in your new shiny cluster.
3. Follow the instructions to create the new cluster using MemSQL tools
After you cleaned your servers from the deprecated MemSQL ops agent and data, you can follow the instructions here. I chose to set up a multiple host comprehensive set up. The process will ask you to register your hosts, and then set up the nodes roles (master aggregator, aggreators and leafs), ip addresses, passwords, ports and etc.
After that, you can try to test the cluster, making changes in one machine and view them in another. Also the output for memsql-admin list-nodes on the deploying machine for my cluster was:
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group |
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
| AAAAAAAAAA | Master | 172.26.32.20 | 3306 | Running | True | 6.7.16 | Online | |
| BBBBBBBBBB | Aggregator | 172.26.32.22 | 3306 | Running | True | 6.7.16 | Online | |
| CCCCCCCCCC | Leaf | 172.26.32.20 | 3307 | Running | True | 6.7.16 | Online | 1 |
| DDDDDDDDDD | Leaf | 172.26.32.22 | 3307 | Running | True | 6.7.16 | Online | 1 |
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
4. Restore the data
Re-execute all the create table queries you saved in step 1, and import all data exported to a csv using this syntax:
LOAD DATA INFILE '/home/yourfolder/yourcsvfile.csv' INTO TABLE your_table;
And that's it! Now you can manage your cluster using the new MemSQL studio that run on the default http://your_deployment_machine:8080.
Enjoy :)

Database not created

I installed postgresql and tried to create new database but couldn't succeed in creating database:
Server [localhost]:
Database [postgres]:
Port [5433]:
Username [postgres]:
psql (9.2.17)
WARNING: Console code page (850) differs from Windows code page (1252)
8-bit characters might not work correctly. See psql reference
page "Notes for Windows users" for details.
Type "help" for help.
postgres=# createdb gps_heatmap
postgres-# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------------------+---------------------+-----------------------
database4 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 |
postgres | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 |
template0 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | English_Canada.1252 | English_Canada.1252 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
postgres-#
Database is not created . I couldnt solve this problem. Help needed.
You are missing ; use
createdb gps_heatmap;

Resources