Fail to export from voltdb to kafka - export

VoltDB is enterprise 5.1.2
Kafka is 2.9.2-0.8.1.1, also tried 2.10-0.8.2.1
VoltDB is at 192.168.56.101
Kafka is at 192.168.56.102
Here is my deployment configuration for VoltDB:
<deployment>
<cluster hostcount="1" sitesperhost="4" kfactor="0" />
<commandlog enabled="true">
<frequency time="400" transactions="1000" />
</commandlog>
<export>
<configuration enabled="true" type="kafka" stream="archive">
<property name="metadata.broker.list">192.168.56.102:9092</property>
<property name="producer.type">sync</property>
<property name="batch.mode">true</property>
</configuration>
</export>
</deployment>
The schema is defined as:
drop table person if exists;
create table person (
ic varchar(9) not null,
first_name varchar(20) not null,
last_name varchar(20) not null,
middle_name varchar(20),
gender tinyint not null,
dob timestamp not null,
date_created timestamp default now
);
partition table person on column ic;
export table person to stream archive;
And for server.properties of Kafka, I only added this line
auto.create.topics.enable=true
I first started Kafka as following:
bin/zookeeper-server-startsh config/zookeeper.properties
bin/kafka-server-start.sh config/server.properties
When I start VoltDB, I encounter this exception:
david#u14voltdb:~$ voltdb create catalog.jar --deployment=config.xml
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option PermSize=64m; support was removed in 8.0
Initializing VoltDB...
_ __ ____ ____ ____
| | / /___ / / /_/ __ \/ __ )
| | / / __ \/ / __/ / / / __ |
| |/ / /_/ / / /_/ /_/ / /_/ /
|___/\____/_/\__/_____/_____/
--------------------------------
Build: 5.1.2 voltdb-5.1.2-0-g6d05c33-local Enterprise Edition
Connecting to VoltDB cluster as the leader...
Host id of this node is: 0
Starting VoltDB with trial license. License expires on May 31, 2015.
Initializing the database and command logs. This may take a moment...
WARN: Failed to send producer request with correlation id 2 to broker 0 with data for partitions [voltdbexportPERSON,0]
java.nio.channels.ClosedChannelException
at kafka.network.BlockingChannel.send(BlockingChannel.scala:97)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
at kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
at kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
at kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
at scala.collection.Iterator$class.foreach(Iterator.scala:631)
at scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
at scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
at kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
at kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
at kafka.producer.Producer.send(Producer.scala:76)
at kafka.javaapi.producer.Producer.send(Producer.scala:42)
at org.voltdb.exportclient.KafkaExportClient$KafkaExportDecoder.onBlockCompletion(KafkaExportClient.java:217)
at org.voltdb.export.processors.GuestProcessor$2.run(GuestProcessor.java:223)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.voltcore.utils.CoreUtils$7$1.run(CoreUtils.java:735)
at java.lang.Thread.run(Thread.java:745)
At the Kafka side, I keep getting this:
[2015-05-14 00:40:08,197] INFO Closing socket connection to /192.168.56.101. (kafka.network.Processor)
Any suggestions?

This is purely an issue related to Kafka setting. In the setting, there is a commented setting:
advertised.host.name=something
just need to replace "something" to the IP address of the server in which Kafka is running. This is found at Kafka - Unable to send a message to a remote server using Java

Related

ODBC connection from Clickhouse to MSSQL databases

Please help with ODBC connection from Clickhouse to SQL Server databases.
I configured ODBC on the Clickhouse server.
Сonnection from clients such as isql, tsql is successful.
But it is not possible to connect from the clickhouse client's.
Operation system – Ubuntu 20.04
Clickhouse Server – version 22.1.3
Clickhouse Client – version 18.16.1
MS SQL Server 2016 on Windows Server.
/etc/freetds/freetds.conf
[TSQL_NE]
host = 10.72.82.72
port = 1433
tds version = 7.4
client charset = UTF-8
/etc/odbcinst.ini
[FreeTDS]
Description=FreeTDS
Driver=/usr/lib/x86_64-linux-gnu/odbc/libtdsodbc.so
Setup=/usr/lib/x86_64-linux-gnu/odbc/libtdsS.so
FileUsage=1
UsageCount=8
/etc/odbc.ini
[TSQL_NE]
Description=FreeTDS NE
Driver=FreeTDS
Server=10.72.82.72
Database=ASU
UID=user
PWD=password
Port=1433
Checking the connection to the MSSQL database via ODBC
root#srv:/# isql -v TSQL_NE "user" "password"
+---------------------------------------+
| Connected! |
| |
| sql-statement |
| help [tablename] |
| quit |
| |
+---------------------------------------+
SQL> SELECT top 10 v1 from asu.dbo.data
+-------------------------+
| V1 |
+-------------------------+
| 1.45 |
| 1.5062500000000001 |
| 1.385 |
| 1.4237500000000001 |
| 1.3712500000000001 |
| 1.425 |
| 1.39625 |
| 1.6487499999999999 |
| 1.28 |
| 1.2037500000000001 |
+-------------------------+
SQLRowCount returns 10
10 rows fetched
root#srv:/# tsql -v -S TSQL_NE -U user –P password
locale is "C.UTF-8"
locale charset is "UTF-8"
using default charset "UTF-8"
1> SELECT top 10 v1 from asu.dbo.data
…
10 rows fetched
Connection with clickhouse-client and the error
root#srv:~# clickhouse-client
ClickHouse client version 18.16.1.
Password for user :
Connecting to localhost:9000.
Connected to ClickHouse server version 22.2.2 revision 54455.
b7e1d742cbd0 :) SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', 'asu', 'dbo.data')
0 rows in set. Elapsed: 0.290 sec.
Received exception from server (version 22.2.2):
> Code: 86. DB::Exception: Received from localhost:9000, 127.0.0.1.
DB::Exception: Received error from remote server /columns_info?connection_string=DSN%3DTSQL_NE%3B%20Uid%3Duser%3BPwd%3Dpassword%3B&table=dbo.data&external_table_functions_use_nulls=true.
HTTP status code: 500 Internal Server Error, body: Error getting columns from ODBC
'Code: 36. DB::Exception: Table dbo.data not found. (BAD_ARGUMENTS) (version 22.2.2.1)'
You command in clickhouse-client should be
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;Database=asu', 'dbo', 'data')
SELECT top 10 v1 from odbc('DSN=TSQL_NE;Uid=user;Pwd=password;', '', 'data')
When you remove schema and database you will get success. I saw this method in here

Liquibase and distributed transactions across a linked server, SQL Server 2019

A technical question concerning distributed transactions via a SQL Server "linked server" definition, using Liquibase.
background/context:
SQL script is being run by Liquibase, through a changelog file.
SQL Server 2019
Oracle 11g Express on the far side of the db link RPTDB_XE.
MSDTC is turned off on the SQL Server host.
liquibase.properties
classpath=C:\\Program Files\\Microsoft SQL Server\\sqljdbc_10.2\\enu\\mssql-jdbc-10.2.0.jre8.jar
driver=com.microsoft.sqlserver.jdbc.SQLServerDriver
url=jdbc:sqlserver://localhost:1433;database=SSReporting;Enlist=false
liquibaseSchemaName=dbo
username=deploy_dbbuild
password=deploy_dbbuild
changeLogFile=changelog.xml
liquibase.hub.mode=off
integratedSecurity=false
changelog.xml
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog xmlns="http://www.liquibase.org/xml/ns/dbchangelog" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd">
<changeSet runOnChange="true" author="smcintosh" id="dw-1042-distributed-trans-error-v2.sql">
<sqlFile path="dw-1042-distributed-trans-error-v2.sql" relativeToChangelogFile="true" endDelimiter="GO"/>
</changeSet>
</databaseChangeLog>
dw-1042-distributed-trans-error-v2.sql
use SSReporting
go
set xact_abort on;
go
set nocount on;
go
begin
-- distributed transaction test
print formatmessage('%s START: %s', convert(varchar(20),getdate(),20), convert(varchar(67),##version));
update RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT set OBJECT_ID=OBJECT_ID+1;
print formatmessage('%s Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: %d rows', convert(varchar(20),getdate(),20), ##rowcount);
insert into dbo.TASK_DETAIL_LOG (BATCH_ID, LOG_DTM, CATEGORY, SEVERITY, ACTION, TASK, SUBTASK, MSG)
values (0,getdate(),'DISTRIB_TRANS','INFO','LIQUIBASE_MIGRATE','PERFORMANCE',null,
'DISTRIB_TRANSACTION_TEST_SCOTT has been updated.');
print formatmessage('%s Inserted dbo.TASK_DETAIL_LOG : %d rows',convert(varchar(20),getdate(),20), ##rowcount);
print formatmessage('%s END', convert(varchar(20),getdate(),20));
end;
go
Run the script manually to verify that it works, using sqlcmd...
C:\>sqlcmd -Usa -Pxxxx -S tcp:SCOTT-MCINTOSH,1433 -dSSReporting -w 1000 -i dw-1042-distributed-trans-error-v2.sql
Changed database context to 'SSReporting'.
2022-03-23 13:52:08 START: Microsoft SQL Server 2019 (RTM-GDR) (KB4583458) - 15.0.2080.9 (X64)
2022-03-23 13:52:08 Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: 3 rows
2022-03-23 13:52:08 Inserted dbo.TASK_DETAIL_LOG : 1 rows
2022-03-23 13:52:08 END
Run the script using Liquibase...
C:\>liquibase --defaults-file=liquibase.properties update
####################################################
## _ _ _ _ ##
## | | (_) (_) | ##
## | | _ __ _ _ _ _| |__ __ _ ___ ___ ##
## | | | |/ _` | | | | | '_ \ / _` / __|/ _ \ ##
## | |___| | (_| | |_| | | |_) | (_| \__ \ __/ ##
## \_____/_|\__, |\__,_|_|_.__/ \__,_|___/\___| ##
## | | ##
## |_| ##
## ##
## Get documentation at docs.liquibase.com ##
## Get certified courses at learn.liquibase.com ##
## Free schema change activity reports at ##
## https://hub.liquibase.com ##
## ##
####################################################
Starting Liquibase at 13:55:46 (version 4.8.0 #1581 built at 2022-02-18 21:43+0000)
Liquibase Version: 4.8.0
Liquibase Community 4.8.0 by Liquibase
Running Changeset: changelog.xml::dw-1042-distributed-trans-error-v2.sql::smcintosh
Unexpected error running Liquibase: MSDTC on server 'SCOTT-MCINTOSH' is unavailable. [Failed SQL: (8501) begin
-- distributed transaction test
print formatmessage('%s START: %s', convert(varchar(20),getdate(),20), convert(varchar(67),##version));
update RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT set OBJECT_ID=OBJECT_ID+1;
print formatmessage('%s Updated RPTDB_XE..OWBTGT.DISTRIB_TRANSACTION_TEST_SCOTT: %d rows', convert(varchar(20),getdate(),20), ##rowcount);
insert into dbo.TASK_DETAIL_LOG (BATCH_ID, LOG_DTM, CATEGORY, SEVERITY, ACTION, TASK, SUBTASK, MSG)
values (0,getdate(),'DISTRIB_TRANS','INFO','LIQUIBASE_MIGRATE','PERFORMANCE',null,
'DISTRIB_TRANSACTION_TEST_SCOTT has been updated.');
print formatmessage('%s Inserted dbo.TASK_DETAIL_LOG : %d rows',convert(varchar(20),getdate(),20), ##rowcount);
print formatmessage('%s END', convert(varchar(20),getdate(),20));
end;]
For more information, please use the --log-level flag
The Question:
Why can't Liquibase run this script, when sqlcmd can? In fact, other database tools like TOAD and DataGrip can also successfully run this script, and they too use a JDBC connection, just like Liquibase.
I can see that you're running the sqlFile change type in your changelog. Liquibase has two different modes of operation. The first splits commands on the delimiter (I see you set it to GO) and executes each statement via a JDBC call to the database. However, in your script the commands are not database commands, they are SQLCMD commands. When those statements get executed on the database they will fail.
However, I found this page on the Liquibase docs site that explains the second mode of operation, which is how to run native SQLCMD scripts via Liquibase.
https://docs.liquibase.com/concepts/changelogs/attributes/using-sqlcmd-integration.html

Why does create table operation attach owner as 'yugabyte' to a new table yet the database to which am connected has a different owner?

I have installed yugabytedb in minikube on my laptop and created a database with owner 'Rodgers'.
Then I run the ysqlsh to execute ysql commands from the terminal, one of which is 'CREATE DATABASE ...'.
Problem
When I try connecting to the database using an external Go application by providing the application with user as 'Rodgers' and the set password, it fails to connect.
I have found out that the tables created were attached to owner 'yugabyte', not 'Rodgers'.
But the database to which I have connected and from where am running the CREATE DATABASE command belongs to Rodgers.
What's going on here?
It's best to rehearse all this using "ysqlsh". When everything works there, connecting from any client program (Python, go, ...) etc will work — as long as you have the right driver. The PostgresSQL drivers work with YugabyteDB.
The following is mainly commands for "ysqlsh" — both SQLs and so-called metacommands (the ones starting with backslash). But occasionally, there are commands that you do from the O/S prompt. So you must read the following carefully and then do what it says after each comment — mainly in "ysqlsh" but a couple of times at the O/S prompt. So you can't simply run the script "lights out".
Start with virgin YB single-node cluster (fresh from "yb-create).
$ ysqlsh -h localhost -p 5433 -d yugabyte -U yugabyte
Now follow the script.
-- Shows two "Superuser" users: "postgres" and "yugabyte" (nothing else).
\du
-- Shows two databases: "postgres" and "yugabyte" (nothing else except "system" databases).
-- Both "postgres" and "yugabyte" databases are owned by "postgres".
\l
-- Create a new "ordinary user and connect as that user.
create user rodgers login password 'p';
alter user rodgers createdb;
-- Now connect to database yugabyte as user rodgers
\c yugabyte rodgers
-- Create a new database and check it's there.
create database rog_db owner rodgers;
\l
-- Name | Owner | Encoding | Collate | Ctype | Access privileges
-- -----------------+----------+----------+---------+-------------+-----------------------
...
-- rog_db | rodgers | UTF8 | C | en_US.UTF-8 |
-- ...
-- Now connect to the new "rog_db" database. Works fine.
\c rog_db rodgers
-- Quit "ysqlsh.
\q
Connect again. Works fine.
$ ysqlsh -h localhost -p 5433 -d rog_db -U rodgers
Now carry on with the script.
-- Works fine.
create table t(k int primary key);
-- Inspect it. First "\d", then "\d t".
\d
-- List of relations
-- Schema | Name | Type | Owner
-- --------+------+-------+---------
-- public | t | table | rodgers
\d t
-- Table "public.t"
Column | Type | Collation | Nullable | Default
-- --------+---------+-----------+----------+---------
-- k | integer | | not null |
-- Indexes:
-- "t_pkey" PRIMARY KEY, lsm (k HASH)
-- This is OK for playing. But terrible for real work.
drop table t;
\c rog_db yugabyte
drop schema public;
\c rog_db rodgers
create schema rog_schema authorization rodgers;
-- For future connect commands.
alter user rodgers set search_path = 'rog_schema';
-- for here and now.
set schema 'rog_schema';
create table t(k int primary key);
\d
-- List of relations
-- Schema | Name | Type | Owner
-- ------------+------+-------+---------
-- rog_schema | t | table | rodgers
--------------------------------------------------------------------------------
I just stepped through all of this using "YB-2.2.0.0-b0" on my laptop (macOS Big Sur). It all worked fine.
Please try this in your minikube env and report back.
Regards, Bryn Llewellyn, Technical Product Manager at Yugabyte Inc.

How to synchronize Redmine database pacemaker/pcs/corosync?

Currently I have 3 servers: one master and 2 clients. I installed redmine 3.3.1.stable with postgresql 9.6, and installed pacemaker on 3 servers. To synchronize a database, I follow the documentation. Every thing is working fine until when I stop the active server. The server2 redmine is showing authentication error.
Redmine error when I try to login form client after connect servers.
Completed 500 Internal Server Error in 11ms (ActiveRecord: 3.5ms)
ActiveRecord::StatementInvalid (PG::ReadOnlySqlTransaction: ERROR: cannot execute UPDATE in a read-only transaction
: UPDATE "users" SET "last_login_on" = '2020-08-17 13:05:11.001886' WHERE "users"."type" IN ('User', 'AnonymousUser') AND "users"."id" = $1):
app/models/user.rb:238:in `try_to_login'
app/controllers/account_controller.rb:204:in `password_authentication'
app/controllers/account_controller.rb:199:in `authenticate_user'
app/controllers/account_controller.rb:40:in `login'
lib/redmine/sudo_mode.rb:63:in `sudo_mode'
so far I unterstand redmine only allow server1 to access permission. why redmine can't give access to server2 or server3
Below I give more information about my step so far.
pcs config
pcs config
Cluster Name: mycluster
Corosync Nodes:
server1 server2 server3
Pacemaker Nodes:
server1 server2 server3
Resources:
Resource: MasterVip (class=ocf provider=heartbeat type=IPaddr2)
Attributes: ip=101.226.189.208 nic=lo cidr_netmask=32 iflabel=pgrepvip
Meta Attrs: target-role=Started
Operations: start interval=0s timeout=20s (MasterVip-start-interval-0s)
stop interval=0s timeout=20s (MasterVip-stop-interval-0s)
monitor interval=90s (MasterVip-monitor-interval-90s)
Resource: Apache (class=ocf provider=heartbeat type=apache)
Attributes: configfile=/etc/apache2/apache2.conf statusurl=http://localhost/server-status
Operations: start interval=0s timeout=40s (Apache-start-interval-0s)
stop interval=0s timeout=60s (Apache-stop-interval-0s)
monitor interval=1min (Apache-monitor-interval-1min)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: Apache
Enabled on: server1 (score:INFINITY) (role: Started) (id:cli-prefer-Apache)
Ordering Constraints:
Colocation Constraints:
Apache with MasterVip (score:INFINITY) (id:colocation-Apache-MasterVip-INFINITY)
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
migration-threshold: 5
resource-stickiness: 10
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: mycluster
dc-version: 1.1.16-94ff4df
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
Quorum:
Options:
master postgresql.conf
# Add settings for extensions here
listen_addresses = '*'
wal_level = hot_standby
synchronous_commit = local
archive_mode = on
archive_command = 'cp %p /var/lib/postgresql/9.6/main/archive/%f'
max_wal_senders = 10
wal_keep_segments = 30
synchronous_standby_names = 'server2'
synchronous_standby_names = 'server3'
hot_standby = on
master pg_hba.conf
# Localhost
host replication postgres 127.0.0.1/32 md5
# PostgreSQL Master IP address
host replication postgres 101.226.189.205/32 md5
# PostgreSQL SLave IP address
host replication postgres 101.226.189.206/32 md5
ho
st replication postgres 101.226.189.207/32 md5
copy config to client from Master
pg_basebackup -h server1 -U postgres -D /var/lib/postgresql/9.6/main -X stream -P
Database connection status
postgres#oreo:/etc/postgresql/9.6/main$ psql -x -c "select * from pg_stat_replication;"
-[ RECORD 1 ]----+------------------------------
pid | 18174
usesysid | 10
usename | postgres
application_name | server3
client_addr | 101.226.189.207
client_hostname |
client_port | 35236
backend_start | 2020-08-17 15:56:40.687282+02
backend_xmin |
state | streaming
sent_location | 0/7005430
write_location | 0/7005430
flush_location | 0/7005430
replay_location | 0/7005430
sync_priority | 1
sync_state | sync
-[ RECORD 2 ]----+------------------------------
pid | 18175
usesysid | 10
usename | postgres
application_name | server2
client_addr | 101.226.189.206
client_hostname |
client_port | 45862
backend_start | 2020-08-17 15:56:40.717087+02
backend_xmin |
state | streaming
sent_location | 0/7005430
write_location | 0/7005430
flush_location | 0/7005430
replay_location | 0/7005430
sync_priority | 0
sync_state | async
I found answer for my question. I miss one step in pacemaker resources.
The pgsqld defines the properties of a PostgreSQL instance: where it is located, where are its binaries, its configuration files, how to montor it, and so on.
The pgsql-ha resource controls all the PostgreSQL instances pgsqld in your cluster, decides where the primary is promoted and where the standbys are started.
pcs resource create pgsqld ocf:heartbeat:pgsqlms \
bindir="/usr/lib/postgresql/9.6/bin" \
pgdata="/etc/postgresql/9.6/main" \
datadir="/var/lib/postgresql/9.6/main" \
pghost="/var/run/postgresql" \
recovery_template="/etc/postgresql/9.6/main/recovery.conf.pcmk" \
op start timeout=60s \
op stop timeout=60s \
op promote timeout=30s \
op demote timeout=120s \
op monitor interval=15s timeout=10s role="Master" \
op monitor interval=16s timeout=10s role="Slave" \
op notify timeout=60s
pcs resource master pgsql-ha pgsqld notify=true
pcs resource cleanup
pcs status

SQLalchemy append dataframe to existing SQL Server table

I'm trying to append two columns from a dataframe to an existing SQL server table. The code runs but when I query the SQL table, the additional rows are not present. What am I missing?
import sqlalchemy
engine = sqlalchemy.create_engine("mssql+pyodbc://user:pw#host:port/dbname?driver=ODBC+Driver+13+for+SQL+Server")
df.to_sql(name='database.tablename', con=engine, if_exists='append', index=False)
You cannot use dot notation in the name= parameter. Just use name=tablename. The other parts are fine.
If you need to assign a non-default (dbo) schema, there is a schema= parameter for df.to_sql(). The prefix database. is redundant because you have already assigned dbname in the engine.
Tested with SQL Server 2017 (latest docker image on debian 10) and anaconda python 3.7.
Test code
SQL Server part (create an empty table)
use testdb;
go
if OBJECT_ID('testdb..test') is not null
drop table test;
create table test (
[Brand] varchar(max),
[Price] money
);
Python part
from pandas import DataFrame
import sqlalchemy
# check your driver string
# import pyodbc
# pyodbc.drivers() # ['ODBC Driver 17 for SQL Server']
# connect
eng = sqlalchemy.create_engine("mssql+pyodbc://myid:mypw#localhost:1433/testdb?driver=ODBC+Driver+17+for+SQL+Server")
df = DataFrame(
data={'Brand': ['A','B','C'],
'Price': [10.00, 20.00, 30.00]},
columns=['Brand', 'Price']
)
df.to_sql(name="test", schema="dbo", con=eng, if_exists="append", index=False)
Result
select * from [test]
| Brand | Price |
|-------|---------|
| A | 10.0000 |
| B | 20.0000 |
| C | 30.0000 |

Resources