Anybody that actually got this to work? To use a csv-file as a datasource with polybase? I've just get the error message below.
Msg 105082, Level 16, State 1, Line 12. See the rest in code block.
Got the latest updates both of SQL Server 2019 EE, OS and ODBC and Microsoft Access Text Driver. The user has the right credentials. Creating the external data source is no problem, it's when trying creating the external table the error occurs. Can anybody see the any obvious error
OPEN MASTER KEY DECRYPTION BY PASSWORD = 'TOPSECRET_PSW';
GO
CREATE EXTERNAL DATA SOURCE TestCSV
WITH
(
LOCATION = 'odbc://localhost',
CONNECTION_OPTIONS = 'Driver=Microsoft Access Text Driver (*.txt, *.csv);Dbq=D:\APA\',
PUSHDOWN = OFF
);
CREATE EXTERNAL TABLE dbo.testCSV
(
Header1 nvarchar(128)
,Header2 nvarchar(128)
)
WITH
(
LOCATION='[testCSV.csv]',
DATA_SOURCE = TestCSV
)
Msg 105082, Level 16, State 1, Line 12
105082;Generic ODBC error: [Microsoft][ODBC Text Driver]General error Unable to open registry key Temporary (volatile) Ace DSN for process 0x1608 Thread 0x3450 DBC 0x8d76d578 Text'. Additional error <2>: ErrorMsg: [Microsoft][ODBC Text Driver]Invalid connection string attribute SERVER, SqlState: 01S00, NativeError: 8 Additional error <3>: ErrorMsg: [Microsoft][ODBC Text Driver]Invalid connection string attribute SERVER, SqlState: 01S00, NativeError: 8 Additional error <4>: ErrorMsg: [Microsoft][ODBC Text Driver]Invalid connection string attribute SERVER, SqlState: 01S00, NativeError: 8 Additional error <5>: ErrorMsg: [Microsoft][ODBC Text Driver]Invalid connection string attribute SERVER, SqlState: 01S00, NativeError: 8 .
You need to apply latest SQL Server 2019 CU to get rid of that message. However, once that error disappears, I couldn't make it work on Docker, are you using a physical machine?
Related
New to SQL-Server. I'm attempting to load a pdf to a SQL-Server table (data type varbinary(max)) via PERL/MS ODBC driver/DBD::ODBC using the following (simplified) code:
use DBI qw(:sql_types);
open my $pdfFH, "test.pdf";
my #pdf = <$pdfFH>; close $pdfFH;
my $pdfStr = join('', #pdf);
my $dbh = <...valid db-handle ...>;
my $sth = $dbh->prepare(qq(
insert into
TestTable(Report)
values
(?)));
$sth->bind_param(1,$pdfStr,DBI::SQL_VARBINARY);
$sth->execute;
Error:
DBD::ODBC::st bind_param failed: [Microsoft][ODBC Driver 17 for SQL Server]Invalid precision value (SQL-HY104) at ./t_sqlserver.pl line 37.
DBD::ODBC::st execute failed: [Microsoft][ODBC Driver 17 for SQL Server]COUNT field incorrect or syntax error (SQL-07002) at ./t_sqlserver.pl line 38.
I am able to successfully load other data types. An alternative is to load the pdf locally from the file system using OPENROWSET(BULK...) but I would prefer to load directly to avoid moving the file from Linux to Windows.
The driver should be clever enough to guess the correct type most of the times. Try binding the parameter without specifying the type at all.
I am running into the error when running a python script pushing data from PostgreSQL to MSSQL.
2020/09/17 09:55:26:mssql:ERROR:Something went very wrong inserting data into mssql - bailing
Traceback (most recent call last):
File "./my_file.py", line 40, in importToMSSQLReal
dataToInsert)
pyodbc.ProgrammingError: ('42000', "[42000] [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Incorrect syntax near the keyword 'key'. (156) (SQLExecute)")
Here is the line of code:
query = """SELECT vi.asset_id ,vi.vulnerability_id,a.last_assessed_for_vulnerabilities as "last_detected",vi.status,vi.proof::varchar(10485760),vi.key::varchar(10485760),vi.service,vi.port,vi.protocol
FROM public.fact_asset_vulnerability_instance vi
INNER JOIN public.dim_asset a
ON vi.asset_id = a.asset_id"""
Insert
def importToMSSQLReal(mssql,mssqlstring,dataToInsert):
mssql = pyodbc.connect(mssqlstring)
log = logging.getLogger('mssql')
curs = mssql.cursor()
curs.fast_executemany = True
try:
curs.executemany(
"INSERT INTO {table} (asset_id,vulnerability_id,last_detected,status,proof,key) VALUES (?,?,?,?,?,?)".format(table=mscfg['table']),
dataToInsert)
mssql.commit()
except:
print("bad")
log.exception("Something went very wrong interting data into mssql - bailing")
exit(2)
curs.close()
mssql.close()
del mssql
Key is a reserved keyword in SQL Server. You have to make sure it is between brackets:
SELECT 1 as [key]
I am trying to insert data into a table which is Column Encrypted using ODBC Driver 17 for SQL Server with column master key in AZURE KEY VAULT but it throws an error. When I try the same using SQL SERVER MANAGEMENT STUDIO it works fine. SSMS asks me my azure credentials and it inserts values into the table. I could do this without any errors when the CMK was created using windows certificate store. I exported the same certificate to azure key vault.
The below link shows that Azure Key vault with AD authentication can be used using the ODBC connection options.
https://learn.microsoft.com/en-us/sql/connect/odbc/using-always-encrypted-with-the-odbc-driver?view=sql-server-2017.
I get the below error trying to insert and get no rows when retrieving using the odbc driver with KeyStoreAuthentication as KeyVaultPassword . However KeyVaultClientSecret auth method works . Am I missing something here ?
Connection String :
"DSN=SQLSERVERNativeAE;Database=test_usr;ColumnEncryption=Enabled;KeyStoreAuthentication=KeyVaultPassword;KeyStorePrincipalId=<my mail id>;KeyStoreSecret=<my password>".
Insert into Customers (CustName, SSN, Age) values (?, ?, 10)
SQLSTATE = CE275
NATIVE ERROR = 0
MSG = [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error requesting access token, HTTP status 400, expected 200
SQLSTATE = CE275
NATIVE ERROR = 0
MSG = [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error requesting access token, HTTP status 41360, expected 200
SQLSTATE = CE258
NATIVE ERROR = 0
MSG = [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]Error retrieving key information for https://aesveeramakeyvault.vault.azure.net:443/keys/AEWCSPFX/35241fa9559c4e04a5e04ae21a123e0f
SQLSTATE = CE202
NATIVE ERROR = 0
MSG = [Microsoft][ODBC Driver 17 for SQL Server][SQL Server]The keystore provider AZURE_KEY_VAULT failed to decrypt the ECEK https://aesveeramakeyvault.vault.azure.net:443/keys/AEWCSPFX/35241fa9559c4e04a5e04ae21a123e0f with RSA_OAEP.
CREATE TABLE [dbo].[Customers](
[CustName] [nvarchar](60) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Randomized, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
[SSN] [varchar](12) COLLATE Latin1_General_BIN2 ENCRYPTED WITH (COLUMN_ENCRYPTION_KEY = [CEK_Auto1], ENCRYPTION_TYPE = Deterministic, ALGORITHM = 'AEAD_AES_256_CBC_HMAC_SHA_256') NULL,
[Age] [int] NULL
) ON [PRIMARY]
CREATE COLUMN MASTER KEY [CMK_Auto1]
WITH
(
KEY_STORE_PROVIDER_NAME = N'AZURE_KEY_VAULT',
KEY_PATH = N'https://aesveeramakeyvault.vault.azure.net:443/keys/AEWCSPFX/35241fa9559c4e04a5e04ae21a123e0f'
)
CREATE COLUMN ENCRYPTION KEY [CEK_Auto1]
WITH VALUES
(
COLUMN_MASTER_KEY = [CMK_Auto1],
ALGORITHM = 'RSA_OAEP',
ENCRYPTED_VALUE = 0x016E000001630075007200720065006E00740075007300650072002F006D0079002F003000650066003400360038003400300063006300610039003700320063006200380032003400390062003200660033003900610039003200300063006600640033003700630061003800310036006300332CAAB1F90FEF11C204297F156833E3E1E4462F03E82C1EABCD717CF3E15C3D1F36FF71AEFFCCAF0C47762E21FD1537E3F65E28F311679FE4E27778F7C19958693D931007F2656DBDA7C37BF352979637CA50F3EB682F69366353F9FA2D0EC906F023784261E8C285A02A034A5B3E189939BE1F7F3D46B0BDD0437A4A1340DB93CF55D6493020CAFF9C3FD9BCFBE8CA466FEAF5C88D949200CC815CC2519E9ECB77FEA7FEB1F7644B52AF424E43D670DC3C7EB0C55DE358489AD30C7EF52CA12C606A0C149052D1085C029EDA7150DF2559CC92A3502ED094AD94C992AE3B00A97B935294F6A68C8A3A79692F8420E369C2AA90BA5C2100288136CAB0620241AAC931F67DD0441E57F3340B5B0D2F8E3668DBBFADC3C01AEFB714C27B37DBDA51F0F7E5EDA65758D172F7B01F87946FEBCAD92D9F55A0A8A0975DAD4F95F1BE9E95B40391E73C8DCB3E45BEF13E767AA240F69AE30A00696584570BB66D8261C81DA401C177513FC55CE137504DCB49561AD5623A55C3166CD8EA8812F0E1B9EB72D456705EC1A32751DA0DF894B4861C001845D0E9D8FBB5E0ED80F6FE95CC44E3571239BC16442EEDFE8F6334F2078677B087742A35C784CEC07E77C5EFACF635E6424C97DAC42363C4445DC1DFDEA9DB46CFCDB44FE611F2BFF54A7DBBD0ACE1F925C73C9F53A90E813B220FFE2491809A3D8A7B90E590CC9D83884CC07B
)
These are commands which I am running:-
bin/zookeeper-server-start etc/kafka/zookeeper.properties &
bin/kafka-server-start etc/kafka/server.properties &
bin/schema-registry-start etc/schema-registry/schema-registry.properties &
bin/connect-standalone etc/schema-registry/connect-avro-standalone.properties etc/kafka-connect-jdbc/quickstart-sqlserver.properties &
bin/kafka-avro-console-consumer --new-consumer --bootstrap-server localhost:9094 --topic test3-sqlserver-jdbc-ErrorLog --from-beginning
I am trying to connect sqlserver using confluent platform(kafka-connect) and facing following issues:
When I am trying to connect to default schema i.e. dbo , connection is built but it is not able to fetch data into the kafka consumer. The connection details that I am using are:
name=test-sqlserver-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlserver://********:1433;database=AdventureWorks2012;user=****;password=****
mode=incrementing
incrementing.column.name=ErrorLogID
topic.prefix=test3-sqlserver-jdbc-
table.whitelist=ErrorLog
schema.registry=dbo
When I am trying to connect to any other schema, the producer is throwing error, connection details that i am using are :
name=test-sqlserver-jdbc-autoincrement
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
tasks.max=1
connection.url=jdbc:sqlserver://********:1433;database=AdventureWorks2012;user=****;password=****
mode=incrementing
incrementing.column.name=AddressID
topic.prefix=test3-sqlserver-jdbc-
table.whitelist=Address
schema.registry=Person
Error :
INFO Source task WorkerSourceTask{id=test-sqlserver-jdbc-autoincrement-0} finished
initialization and start (org.apache.kafka.connect.runtime.WorkerSourceTask:138)
[2017-03-07 17:55:47,041] ERROR Failed to run query for table
TimestampIncrementingTableQuerier{name='Address', query='null',
topicPrefix='test3-sqlserver-jdbc-', timestampColumn='null',
incrementingColumn='AddressID'}:
com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'Address'.
io.confluent.connect.jdbc.JdbcSourceTask:239)
[2017-03-07 17:55:52,124] ERROR Failed to run query for table
TimestampIncrementingTableQuerier{name='Address', query='null',
topicPrefix='test3-sqlserver-jdbc-', timestampColumn='null',
incrementingColumn='AddressID'}: com.microsoft.sqlserver.jdbc.SQLServerException:
Invalid object name 'Address'. (io.confluent.connect.jdbc.JdbcSourceTask:239)
[2017-03-07 17:55:53,684] INFO Reflections took 9299 ms to scan
262 urls, producing 12112 keys and 79402 values
(org.reflections.Reflections:229)
[2017-03-07 17:55:57,181] ERROR Failed to run query for table
TimestampIncrementingTableQuerier{name='Address', query='null',
topicPrefix='test3-sqlserver-jdbc-', timestampColumn='null',
incrementingColumn='AddressID'}:
com.microsoft.sqlserver.jdbc.SQLServerException: Invalid object name 'Address'.
(io.confluent.connect.jdbc.JdbcSourceTask:239)
I am successfully querying what I need from the first Server with the following:
library(RODBC)
library(data.table)
library(sqldf)
library(zoo)
library(forecast)
library(plyr)
library(parallel)
myconn <- odbcDriverConnect("SERVER=SQL12p4;DRIVER=SQL Server")
ManagementHours <- sqlQuery(myconn, " SELECT .......")
However when I try to connect to a different server SQL12P4\VSQL1 I am unable to connect. I know that the \ is an escape character and I believe that is my problem. I am wondering if there is a way to get around this? Below is the R command I am currently using.
myconn1 <- odbcDriverConnect("SERVER=SQL12P4\VSQL1;DRIVER= SQL SERVER")
> myconn1A <-odbcDriverConnect("SERVER=SQL12P4\VSQL1;DRIVER= SQL SERVER")
Error: '\V' is an unrecognized escape in character string starting ""SERVER=SQL12P4\V"
> myconn1B <-odbcDriverConnect("SERVER=SQL12P4\\VSQL1;DRIVER= SQL SERVER")
Warning messages:
1: In odbcDriverConnect("SERVER=SQL12P4\\VSQL1;DRIVER= SQL SERVER") :
[RODBC] ERROR: state IM002, code 0, message [Microsoft][ODBC Driver Manager] Data source name not found and no default driver specified
2: In odbcDriverConnect("SERVER=SQL12P4\\VSQL1;DRIVER= SQL SERVER") :
ODBC connection failed