I'm setting up a new VM on a server to offload SQL Server database loading from my laptop. In doing so, I'd like to be able to execute stored procedures (no params, just 'exec storedprocedure') in my database via Python, but it's not working.
Stored procedure call worked when using sqlcmd via a batch file and in SSMS, but i'd like to make it all python based.
The stored procedure is appending fact tables follows the below general format:
--staging tbl drop and creation
if object_id(stagingtbl) is not null drop tabl stagingtbl
create table stagingtbl
(fields datatypes nullable
)
--staging tbl load
bulk insert stagingtbl
from 'c:\\filepath\\filename.csv'
with (
firstrow = 2
, rowterminator = '\n'
,fieldterminator = ','
, tablock /*don't know what tablock does but it works...*/
)
--staging table transformation
; with cte as (
/*ETL process to transform csv file into my tbl structure*/
)
--final table load
insert final_tbl
select * from cte
/*
T-SQL update the final table's effect to date, based on subsequent effect from date.
eg:
id, effectfromdate, effecttodate
1,1/1/19, 1/1/3000
1,1/10/19, 1/1/3000
becomes
id, effectfromdate, effecttodate
1,1/1/19, 1/10/19
1,1/10/19, 1/1/3000
*/
The stored procedure works fine with sqlcmd and in ssms but in python (pyodbc) executing the query 'exec storedprocedure', I get the error message:
pyodbc.ProgrammingError: ('42000', '[42000] [Microsoft][SQL Server Native Client 11.0][SQL Server]
Cannot bulk load because the file "c:\filepath\filename.csv" could not be opened.
Operating system error code 3(The system cannot find the path specified.). (4861) (SQLExecDirectW)')
When the csv file is there, no misspellings in path or filename, and I can open the csv when double clicking on it, and no one has the csv open.
With continued experimentation I've established the problem is not with python or pyodbc. In SSMS on my laptop (host machine of db) the stored procedures work just fine, but in SSMS on the VM the stored procedures cause the same error. This tells me that my question isn't the root problem and I had more digging to do. The error (in SSMS) is below.
Msg 4861, Level 16, State 1, Procedure Append_People, Line 71 [Batch Start Line 0]
Cannot bulk load because the file "N:\path\filename.csv" could not be opened. Operating system error code 3(The system cannot find the path specified.).
Once I established the problem is in SSMS, I broadened my search and discovered the issue is the path for the bulk insert command has to be relative to the machine hosting the database. So in the VM (client machine until I migrate the db) when I use the path c:\ thinking it's the VM's c:\ drive the stored procedure is looking at the c:\ of my laptop since it's the host machine. With that I also learned that on a shared drive (N:\) access is delegated and that is its own issue (https://dba.stackexchange.com/questions/44524/bulk-insert-through-network).
So I'm going to focus on migrating the database first, then that'll solve my problem. Thanks to all who tried to help
Related
Can't seem to get this bulk insert to work. I know the SQL Server windows account has appropriate permissions to this network folder - that has been verified.
The error I get is: Msg 12704, Level 16, State 1, Line 1 Bad or inaccessible location specified in external data source "(null)".
I can copy and paste the path into my windows finder and it opens the file but maybe I am using the wrong syntax in the SQL statement?
Here is the SQL statement:
BULK INSERT Employee_be.dbo.Table_Cur_Test
FROM '\\SERVERPATH\Data\Test\CurTest.txt'
WITH
(
FIELDTERMINATOR = ',',
ROWTERMINATOR = ';\n'
)
GO
Any suggestions?
I have a table stored in SQL Server Management Studio version 15.0.18369.0 and I have a working and established ODBC connection to SAS language program World Programming Software version 3.4.
Previous import/reading of this data has been successful but the operator of the SQL server may have recently converted their data type to NVARCHAR(max). It may also be due to a driver change (I got a new laptop and reinstalled what I thought was the exact same OBDC for SQl driver as I had before but who knows) I have 64-bit ODBC Driver 17 for SQL server.
The VARCHAR(max) type in SQL causes the data to only be 1 character long in every column.
I have tried to fix it by:
Adding the DB max text option to libname
libname odbclib odbc dsn="xxx" user="xxx" password="xxx" DBMAX_TEXT=8000;
This did nothing so I also tried to add DB type option:
data mydata (dbtype=(mycol='char(25)')) ;
set odbclib.'sql data'n
run;
And I get ERROR:
The option dbtype is not a valid output data set option.
I have also tried DBSASTYPE, and putting both options in the set statement and this yields the same error.
I also tried with proc SQL:
proc sql noprint;
CONNECT TO ODBC(dsn="xxx" user="xxx" password="xxx"); create table
extract(compress=no dbsastype=(mycol='CHAR(20)')) as select * from
connection to odbc ( select * from dbo.'sql data'n
); disconnect from odbc; quit;
And I get
NOTE: Connected to DB: BB64 (Microsoft SQL Server version 12.00.2148)
NOTE: Successfully connected to database ODBC as alias ODBC. 3915
create table extract(compress=no
dbsastype=(mycol='CHAR(20)')) as 3916 select *
from connection to odbc 3917 ( 3918 select *
from dbo.'sql data'n ERROR: The option dbsastype is not a valid
output data set option 3919 3920 ); 3921
disconnect from odbc; NOTE: ERRORSTOP was specified. The statement
will not be executed due to an earlier error NOTE: Statements not
executed because of errors detected 3922 quit;
One thing to try would be putting the DBTYPE or DBSASTYPE in the right place.
data mydata;
set odbclib.'sql data'n(dbtype=(mycol='char(25)'));
run;
It is an option that would go on the "set" not the "data" line.
I would also encourage you to contact the developers, as it seems like this is a bug in their implementation; or perhaps try a different ODBC SQL driver.
Maybe a workaround in the meanwhile is to use CONVERT in the passthrough?
proc sql noprint;
CONNECT TO ODBC(dsn="xxx" user="xxx" password="xxx"); create table
extract as select * from
connection to odbc ( select a,b,c,convert(NCHAR(20),mycol) from dbo.'sql data'n
); disconnect from odbc; quit;
I have a server with an alias which I'm referencing in a stored procedure when trying to bulk load the contents of a text file into a temp table (as below). I get the following error message when the stored procedure is called:
Cannot bulk load because the file "\\ServerAlias\SubFolder\FileName.txt" could not be opened. Operating system error code 3 (The system cannot find the path specified.).
Yet when I swap out the alias for the real server name, the stored procedure runs fine. The alias works perfectly well in every other context in our system and I haven't found any answers online. Why won't it work here?
BULK INSERT #Temp
FROM '\\ServerAlias\SubFolder\FileName.txt'
WITH
(
FIRSTROW = 1,
ROWTERMINATOR = '\n'
)
BULK INSERT dbo.bulkins
FROM "C:\BulkDataFile.csv"
WITH
(FIELDTERMINATOR =',',
ROWTERMINATOR = '\n')
Error:
Msg 4860, Level 16, State 1, Line 2
Cannot bulk load. The file "C:\BulkDataFile.csv" does not exist.
How to fix it?
The error message seems crystal clear: "the file ... does not exist" ...
So it seems that this file you're trying to use to do your BULK INSERT just isn't there.
How to fix it? Simple: just put the file where you expect it to be, and run your code again.
And if this is a remote SQL Server, the file must be on the remote machine's C:\ drive - not your local PC's C:\ drive ...
I have a big file which I have loaded in a table in a netezza database using an ETL tool, lets call this database Staging_DB. Now, post some verifications, the content of this table needs to be inserted into similar structured table residing in another netezza DB, lets call this one PROD_DB. What is the fastest way to transfer data from staging_DB to PROD_DB?
Should I be using the ETL tool to load the data into PROD_DB? Or,
Should the transfer be done using external tables concept?
If there is no transformation need to be done, then better way to transfer is cross database data transfer. As described in Netezza documentation that Netezza support cross database support where the user has object level permission on both databases.
You can check permission with following command -
dbname.schemaname(loggenin_username)=> \dpu username
Please find below working example -
INSERT INTO Staging_DB..TBL1 SELECT * FROM PROD_DB..TBL1
If you want to do some transformation and than after you need to insert in another database then you can write UDT procedures (also called as resultset procedures).
Hope this will help.
One way you could move the data is by using Transient External Tables. Start by creating a flat file from your source table/db. Because you are moving from Netezza to Netezza you can save time and space by turning on compression and using internal formatting.
CREATE EXTERNAL TABLE 'C:\FileName.dat'
USING (
delim 167
datestyle 'MDY'
datedelim '/'
maxerrors 2
encoding 'internal'
Compress True
REMOTESOURCE 'ODBC'
logDir 'c:\' ) AS
SELECT * FROM source_table;
Then create the table in your target database using the same DDL in the source and just load it up.
INSERT INTO target SELECT * FROM external 'C:\FileName.dat'
USING (
delim 167
datestyle 'MDY'
datedelim '/'
maxerrors 2
encoding 'internal'
Compress True
REMOTESOURCE 'ODBC'
logDir 'c:\' );
I would write a SP on production db and do a CTAS from stage to production database. The beauty of SP is you can add transformations as well.
One other option is NZ migrate utility provided by Netezza and that is the fastest route I believe.
A simple SQL query like
INSERT INTO Staging_DB..TBL1 SELECT * FROM PROD_DB..TBL1
works great if you just need to do that.
Just be aware that you have to be connected to the destination database when executing the query, otherwise you will get an error code
HY0000: "Cross Database Access not supported for this type of command"
even if you have read/write access to both databases and tables.
In most cases you can simply change the catalog using a "Set Catalog" command
https://www-304.ibm.com/support/knowledgecenter/SSULQD_7.0.3/com.ibm.nz.dbu.doc/r_dbuser_set_catalog.html
set catalog='database_name';
insert into target_db.target_schema.target_table select source_db.source_schema.source_table;