tdengine is deployed in k8s, through the way of http query, the query statement uses functions such as max, min, last, etc. I want to know where the parameters are not set properly, what is the meaning of tables and qId in the log, and what can be done Can you check the reasons for this query over and stop read? thanks.
Version is version 2.4
The log is as follows:
TAOS_ADAPTER info "| 200 | 19.518869ms | 10.233.94.78 | POST | /rest/sql/indc_point_data_40 " model=web sessionID=8660772
01/30 10:47:50.236770 00000189 TDB 0x7f8779a12c30 LIMIT_READ query is over and stop read. tables=1 qId=0x89f00496e6ac46
01/30 10:47:50.237723 00000200 TDB 0x7f87a5d47860 LIMIT_READ query is over and stop read. tables=1 qId=0x89f00496e6b449
01/30 10:47:50.239013 00000127 TAOS_ADAPTER info "| 200 | 9.334362ms | 10.233.94.37 | POST | /rest/sql/indc_point_data_40 " model=web sessionID=8660775
01/30 10:47:50.243209 00000207 TDB 0x7f87c18036c0 LIMIT_READ query is over and stop read. tables=1 qId=0x89f00496e6ec4e
01/30 10:47:50.245438 00000127 TAOS_ADAPTER info "| 200 | 19.027155ms | 10.233.101.104 | POST | /rest/sql/indc_point_data_40 " sessionID=8660774 model=web
01/30 10:47:50.252404 00000191 TDB 0x7f87519dfde0 LIMIT_READ query is over and stop read. tables=1 qId=0x89f00496e73c57
Related
Please help with ODBC connection from Clickhouse to SQL Server databases.
I configured ODBC on the Clickhouse server.
Сonnection from clients such as isql, tsql is successful.
But it is not possible to connect from the clickhouse client's.
Operation system – Ubuntu 20.04
Clickhouse Server – version 22.1.3
Clickhouse Client – version 18.16.1
MS SQL Server 2016 on Windows Server.
/etc/freetds/freetds.conf
[TSQL_NE]
host = 10.72.82.72
port = 1433
tds version = 7.4
client charset = UTF-8
/etc/odbcinst.ini
[FreeTDS]
Description=FreeTDS
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage=1
UsageCount=8
/etc/odbc.ini
[TSQL_NE]
Description=FreeTDS NE
Driver=FreeTDS
Server=10.72.82.72
Database=ASU
UID=user
PWD=password
Port=1433
Checking the connection to the MSSQL database via ODBC
root#srv:/# isql -v TSQL_NE "user" "password"
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL> SELECT top 10 v1 from asu.dbo.data
+-------------------------+
| V1 |
+-------------------------+
| 1.45 |
| 1.5062500000000001 |
| 1.385 |
| 1.4237500000000001 |
| 1.3712500000000001 |
| 1.425 |
| 1.39625 |
| 1.6487499999999999 |
| 1.28 |
| 1.2037500000000001 |
+-------------------------+
SQLRowCount returns 10
10 rows fetched
root#srv:/# tsql -v -S TSQL_NE -U user –P password
locale is "C.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> SELECT top 10 v1 from asu.dbo.data
…
10 rows fetched
Connection with clickhouse-client and the error
root#srv:~# clickhouse-client
ClickHouse client version 18.16.1.
Password for user :
Connecting to localhost:9000.
Connected to ClickHouse server version 22.2.2 revision 54455.
b7e1d742cbd0 :) SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', 'asu', 'dbo.data')
0 rows in set. Elapsed: 0.290 sec.
Received exception from server (version 22.2.2):
> Code: 86. DB::Exception: Received from localhost:9000, 127.0.0.1.
DB::Exception: Received error from remote server /columns_info?connection_string=DSN%3DTSQL_NE%3B%20Uid%3Duser%3BPwd%3Dpassword%3B&table=dbo.data&external_table_functions_use_nulls=true.
HTTP status code: 500 Internal Server Error, body: Error getting columns from ODBC
'Code: 36. DB::Exception: Table dbo.data not found. (BAD_ARGUMENTS) (version 22.2.2.1)'
You command in clickhouse-client should be
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;Database=asu', 'dbo', 'data')
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', '', 'data')
When you remove schema and database you will get success. I saw this method in here
After searching TDengine online documentation: https://www.taosdata.com/en/documentation/, I found the the command to change the default database parameter "keep", which indicates how long data will be preserved in databases. However after I have typed in that command from shell, "show variables" command still shows the old value. How would I know if changing this parameter is taking effect?
taos> alter database test keep 50;
Query OK, 0 of 0 row(s) in database (0.019087s)
taos> show variables;
name | value |
============================================================
version | 2.1.5.0 |
buildinfo | Built at 2021-08-05 23:49:17 |
walLevel | 1 |
comp | 2 |
precision | 0 |
maxRows | 4096 |
minRows | 100 |
keep | 3650 |
the alter command is effective at the DataBase level, and show varibles is show the global parameters.
you can use show databases; to check the database's parameter.
if you want change the show variables;'s show, you should modify the config file /etc/taos.cfg
and there are only serval parameters can modify by alter command.
I'm having issues testing a data unload flow from Snowflake using the GET command to store the files on my local machine.
Following the documentation here, it should be as simple as creating a stage, copying the data I want to that stage, and then running a snowsql command locally to retrieve the files.
I'm on Windows 10, running the following snowsql command to try and unload the data, against a database populated with the test TCP-H data that snowflake provides:
snowsql -a <account id> -u <username> -q "
USE DATABASE TESTDB;
CREATE OR REPLACE STAGE TESTSNOWFLAKESTAGE;
copy into #TESTSNOWFLAKESTAGE/supplier from SUPPLIER;
GET #TESTSNOWFLAKESTAGE file://C:/Users/<local user>/Downloads/unload;"
All commands run successfully, except for the final GET:
SnowSQL * v1.2.14
Type SQL statements or !help
+----------------------------------+
| status |
|----------------------------------|
| Statement executed successfully. |
+----------------------------------+
1 Row(s) produced. Time Elapsed: 0.121s
+-------------------------------------------------+
| status |
|-------------------------------------------------|
| Stage area TESTSNOWFLAKESTAGE successfully created. |
+-------------------------------------------------+
1 Row(s) produced. Time Elapsed: 0.293s
+---------------+-------------+--------------+
| rows_unloaded | input_bytes | output_bytes |
|---------------+-------------+--------------|
| 100000 | 14137839 | 5636225 |
+---------------+-------------+--------------+
1 Row(s) produced. Time Elapsed: 7.548s
+-----------------------+------+--------+------------------------------------------------------------------------------------------------------+
| file | size | status | message |
|-----------------------+------+--------+------------------------------------------------------------------------------------------------------|
| supplier_0_0_0.csv.gz | -1 | ERROR | An error occurred (403) when calling the HeadObject operation: Forbidden, file=supplier_0_0_0.csv.gz |
+-----------------------+------+--------+------------------------------------------------------------------------------------------------------+
1 Row(s) produced. Time Elapsed: 1.434s
This 403 looks like it's coming from the S3 instance backing my Snowflake account, but that's part of the abstracted service layer provided by Snowflake, so I'm not sure where I would have to go to flip auth switches.
Any guidance is much appreciated.
You need to use Windows-based slashes in your local file path. So, assuming that to #NickW's point, you are filling your local user correctly, the format should be like the following:
file://C:\Users\<local user>\Downloads
There are some examples in the documentation for this here:
https://docs.snowflake.com/en/sql-reference/sql/get.html#required-parameters
I have a ubuntu 20 on dreamcompute (which is cloud computing).
I create a user and a database. Here is the list of database and users (for some reason, I can't see database under a matt username).
I went into:
nano /etc/postgresql/13/main/postgresql.conf &
nano /etc/postgresql/13/main/pg_hba.conf and did the whole '*' and '0.0.0.0/0'
postgres=# \l
List of databases
Name | Owner | Encoding | Collate | Ctype | Access privileges
-----------+----------+----------+---------+---------+-----------------------
postgres | postgres | UTF8 | C.UTF-8 | C.UTF-8 |
strapi | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =Tc/postgres +
| | | | | postgres=CTc/postgres+
| | | | | hossein=CTc/postgres
template0 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
template1 | postgres | UTF8 | C.UTF-8 | C.UTF-8 | =c/postgres +
| | | | | postgres=CTc/postgres
(4 rows)
as you can see you can't see Superuser and database strapi under the matt username.
List of roles
Role name | Attributes | Member of
-----------+------------------------------------------------------------+-----------
matt | | {}
postgres | Superuser, Create role, Create DB, Replication, Bypass RLS | {}
ubuntu | | {}
I'm using my dream compute's ip address as my host and use my database and user and password but get error message: connection attempt time out.
Could someone please give me a pointer on why this is happening? I have been working onthis for 2 weeks now and I can't get it to work.
Error message is connection time out. That usually means that the port is blocked by a firewall. Check your cloud provider firewall settings and iptablesin your Linux box in case you have installed it.
If there was problem with permissions, the error message would be something else.
as ex4 mentioned above I needed to reachout to the company that I was renting my cloud computer from but I still could not connect to the database.
The way I went around it is that you can ssh into your database and then connect to your database as a localhost since you are ssh into your cloud computer.
In DBeaver you have a ssh tab and you can connect and then you got back to your postgres tab and fill the localhost, user, database name, and user password area and simply click connect.
Sadly this took weeks to come to this :/
I have a CentOS server running a local memsql cluster (aggregator and leaf on the same machine). I have a databse named offers. For some reason, I cannot execute any queries against tables in my database.
Everything was working fine until I tried to add another machine to the cluster. I had the IT team at my place replicate the server I was working on (completely). I went over to the replicated server, deleted the database in question and then registered the server using the memsql-toolbox-config register-node command. Then the database showed it was under the transition state. I restarted memsql using memsql-ops and got to this situation.
Running a simple query yields:
memsql> select * from table;
ERROR 2261 (HY000): Query `select * from table` couldn't be executed because of an in progress failover operation. Check the status of the leaf nodes in the cluster (error 1049:'Leaf Error (172.26.32.20:3307): Unknown database 'offers_5'')
The output for the the cluster status command is:
memsql> show cluster status;
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
| Node ID | Host | Port | Database | Role | State | Position | Master Host | Master Port | Metadata Master Node ID | Metadata Master Host | Metadata Master Port | Metadata Role | Details |
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
| 1 | 172.26.32.20 | 3306 | cluster | master | online | 0:181 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 1 | 172.26.32.20 | 3306 | offers | master | online | 0:156505 | NULL | NULL | NULL | NULL | NULL | Reference | |
| 2 | 172.26.32.20 | 3307 | cluster | async slave | replicating | 0:180 | 172.26.32.20 | 3306 | 1 | 172.26.32.20 | 3306 | Reference | stage: packet wait, state: x_streaming, err: no |
| 2 | 172.26.32.20 | 3307 | offers | sync slave | replicating | 0:156505 | 172.26.32.20 | 3306 | 1 | 172.26.32.20 | 3306 | Reference | |
+---------+--------------+------+----------+-------------+-------------+----------+--------------+-------------+-------------------------+----------------------+----------------------+---------------+-------------------------------------------------+
4 rows in set (0.00 sec)
So it seems that the the second node is replicating. Also note the details column saying:
stage: packet wait, state: x_streaming, err: no
Running the replication status command gives:
memsql> show replication status;
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
| Role | Database | Master_URI | Master_State | Master_CommitLSN | Master_HardenedLSN | Master_ReplayLSN | Master_TailLSN | Master_Commits | Connected | Slave_URI | Slave_State | Slave_CommitLSN | Slave_HardenedLSN | Slave_ReplayLSN | Slave_TailLSN | Slave_Commits |
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
| master | cluster | NULL | online | 0:181 | 0:181 | 0:177 | 0:181 | 86 | yes | 172.26.32.20:3307/cluster | replicating | 0:180 | 0:181 | 0:180 | 0:181 | 84 |
| master | offers | NULL | online | 0:156505 | 0:156505 | 0:156505 | 0:156505 | 183 | yes | 172.26.32.20:3307/offers | replicating | 0:156505 | 0:156505 | 0:156505 | 0:156505 | 183 |
+--------+----------+------------+--------------+------------------+--------------------+------------------+----------------+----------------+-----------+---------------------------+-------------+-----------------+-------------------+-----------------+---------------+---------------+
2 rows in set (0.00 sec)
I never initiated any fail over or replication. Anyone knows why this is happening? How could I solve this?
EDIT:
Using memsql-ops I get:
[me#memsql ~]$ memsql-ops memsql-list
ID Agent Id Process State Cluster State Role Host Port Version
33829AF Af13af7 RUNNING CONNECTED MASTER 172.26.32.20 3306 6.5.18
BBA1B61 Af13af7 RUNNING CONNECTED LEAF 172.26.32.20 3307 6.5.18
But with memsql-admin, with the new memsql tools:
[me#memsql ~]$ memsql-admin list-nodes
✘ Failed to list nodes on all hosts: failed to list nodes on 1 host:
172.26.32.20
No nodes found
Making my question a bit clearer - How can I get my server to respond to queries again? And after I do, How should I act to add another host? Should I clean the replicated server completely of any memsql data?
2nd EDIT:
I managed to solve this problem by delete my database and cluster data, and setting up a new one using the new MemSQL tools, throwing away MemsqlOps. Read my answer.
It looks like there are a couple things that might be causing problems. Generally speaking, cloning a memsql server is not something that is supported nor the best way to go about adding nodes. It also looks like you may be using both the older Ops management tool and the newer MemSQL tools. I would recommend not installing or using Ops and sticking to just the new MemSQL tools instead.
A good place to start would be to try recreating the nodes after cloning; a cloned memsql node won't correctly become part of the cluster. You should also verify that you don't have more than one master aggregator in the cluster. If you can start with that and see if it resolves your issues I'm happy to help with any other problems that you run into.
I managed to set up a working cluster.
As micahbhakti mentioned in his answer, I tried using only the newer MemSQL tools, instead of the deprecated MemSQL Ops. It required deleting the MemSQL agent existing on both servers and then following the tutorial in the MemSQL documentation. Here are the steps I took for anyone struggling with this issue which is better described as: My MemSQL-Ops-managed-MemSQL-cluster is not responding. How can I upgrade it to a working MemSQL-tools-managed-cluster?
1. Save what data you can
The following step is to delete all memsql data, so it would be best if you could save your data. The table data could be stored in CSV files easily with a simple
SELECT * FROM important_data_containing_table INTO OUTFILE '/home/yourfolder/yourcsvfile.csv';
Do this for all tables containing important data. You could also save the scheme itself. You can do that by viewing and copying to another file all the create queries you used to create the table originally, to re-execute them later. Use this
SHOW CREATE TABLE your_table_name
The documentation for mysql is described here. It might not be similar to the syntax used in mem, but the above base command works. For exact information, read about MySQL Features Unsupported in MemSQL.
2. Delete anything to do with Memsql-Ops
As it is said here about the uninstall command:
Stops the local MemSQL Ops agent and deletes all its data.
If MemSQL nodes are already installed in the local host, this command will prompt users to delete those nodes first before proceeding with the uninstall.
And indeed, if there is a node runnning (in my case there were), you will be prompted to run another command to delete those nodes: memsql-ops memsql-delete --all. This WILL delete all data in your database as said in it's documentation:
Deletes all data for a MemSQL node. This operation is not reversible and may lead to data loss. Users who want to perform this operation are prompted to explicitly type ‘DELETE’ to be sure of their decision.
That's why I asked you to save what ever you need :)
This should be done for each host you want to include in your new shiny cluster.
3. Follow the instructions to create the new cluster using MemSQL tools
After you cleaned your servers from the deprecated MemSQL ops agent and data, you can follow the instructions here. I chose to set up a multiple host comprehensive set up. The process will ask you to register your hosts, and then set up the nodes roles (master aggregator, aggreators and leafs), ip addresses, passwords, ports and etc.
After that, you can try to test the cluster, making changes in one machine and view them in another. Also the output for memsql-admin list-nodes on the deploying machine for my cluster was:
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
| MemSQL ID | Role | Host | Port | Process State | Connectable? | Version | Recovery State | Availability Group |
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
| AAAAAAAAAA | Master | 172.26.32.20 | 3306 | Running | True | 6.7.16 | Online | |
| BBBBBBBBBB | Aggregator | 172.26.32.22 | 3306 | Running | True | 6.7.16 | Online | |
| CCCCCCCCCC | Leaf | 172.26.32.20 | 3307 | Running | True | 6.7.16 | Online | 1 |
| DDDDDDDDDD | Leaf | 172.26.32.22 | 3307 | Running | True | 6.7.16 | Online | 1 |
+------------+------------+--------------+------+---------------+--------------+---------+----------------+--------------------+
4. Restore the data
Re-execute all the create table queries you saved in step 1, and import all data exported to a csv using this syntax:
LOAD DATA INFILE '/home/yourfolder/yourcsvfile.csv' INTO TABLE your_table;
And that's it! Now you can manage your cluster using the new MemSQL studio that run on the default http://your_deployment_machine:8080.
Enjoy :)