I have an ogr2ogr batch file that reprojects SQL data into a new SQL Server table.
It works fine when I run the bat file manually but it fails if I run the bat file via a SQL Server stored procedure. I have given the gdal folders SQL Service permissions and xp_CommandShell is also enabled. I'm using
EXECUTE xp_CMDShell 'blah'
in the T-SQL script.
For some reason the ogr_MSSQLSpatial.dll causes it to fail.
ERROR 1: Can't load requested DLL: Z:\BroadSpectrumSQLTreeExtract\ogr2ogr\gdalplugins\ogr_MSSQLSpatial.dll
If I remove this dll the script runs via SQL but it means I need to add extra commands that the dll must take care of, such as setting source coordinate system. I haven't managed to get it working 100%. The furthest I got to was producing the reprojected table but the geometry field is empty.
The DLL does contain SQL commands to the system tables. Could this be a SQL Server security issue stopping it from working?
I again had this problem with another ogr2ogr bat while executing with SQL. If I put the bat in the same folder as the dll's it works fine.
Related
System Error: Cannot bulk load because the file "XYZ.txt" could not be opened. Operating system error code 1311(There are currently no logon servers available to service the logon request.)
I have a stored procedure in SQL Server 2008 R2 which is using Bulk Insert command to load data from txt files into SQL Server tables. These files are on a shared folder on a drive located on different domain. I have full access to the drive. I tried copying files to different directory on that drive, moving files, deleting files and everything works.
When I execute the stored procedure from a ssms session from local computer it works like a champ. It is able to open the files on shared drive, read it and load the data into the SQL Server tables without any issue. When I call the stored procedure from a SQL Server Agent job, it throws this error.
SQL Server Agent is using the account which is very powerful with lot more permissions than mine. But the job fails.
To find a workaround I created a SSIS package which calls the stored procedure from "Execute SQL Task". It uses Windows authentication to connect to the database. I tried executing the package and it ran successfully. It is able to upload the data from txt to table.
So, then I created a SQL Server user with my account details, and then used that credentials to create a proxy with ssis sub system. I then scheduled the job to execute the step with newly created proxy to see if it can upload the data. But it failed with the same error.
I am confused what am I doing wrong..? I even added myself to bulkdmin role and ran the job again with no success.
I'll appreciate if someone can help.
Thanks.
Just out of curiosity I tried replacing Bulk Insert command with BCP. For some reason BCP worked. It is able to Open the files on network drive and read through it to insert the data in sql server tables. I can even call the same stored proc from sql agent job and it works perfectly fine. I didn't need to use SSIS package to solve this.
I have to weekly upload text files from a server location to Microsoft SQL Server Management Studio .I wish to automate the task so that files are automatically uploaded .Can somebody suggest me the way?
Methods I know of:-
Via SQL:
Use OPENROWSET to open the file and obtain the records to write into
a table.
Use BULK INSERT to open the file and insert directly into a table (you may need to pair with XP_CMDSHELL to get a directory listing to loop through)
VIa SSMS:
Create a DataFlow to import from file
SSMS makes it easier to do clever things with the import process. But it can be very finnicky.
With both of those you can set up an Agent job to run the script / package automatically.
I'm new to SSIS and stuck with a problem and hope some of them would have already gone through any of this.
Task:
Copying files from a remote server to a local machine folder using File System task and For each loop container.
Problem:
The job executes i.e. files are getting copied successfully when I execute from the SSIS designer but when deployed the project on the SQL server instance it isn't copying any files in fact the target folder is totally empty.
I'm not understanding this strange behavior. Any inputs would be of great help!
Regards-
Santosh G.
The For each loop will not error out if it doesn't find any files.
The SQL Agent account may not have access to read the directory contents.
Check is your path a variable - is it being set by a config or /SET statement?
Can you log the path before starting the for loop?
Can you copy a dummy file and see can SSIS see this file?
How are you running the job - cmd_exec() can give spurious results with file I/O tasks
The issue was related to the user authorizarions of the SQL Server agent service.
When I execute the job from SQL Server it uses agent service and for that agent service you need to assign a service user who has access rights to the desired file path.
I am working on a Windows Server 2012 64-bit. I want to be able to import data from a .dbf file into a SQL Server table. I used the import wizard and it worked correctly. However, I have SQL Server Express and can't schedule this insertion.
Is there another way to schedule the insertion of the .dbf data to the SQL Server tables, without the use of the SSIS package loader?
Update
I ended up using Python and writing a script to import from XML. However, I believe the answer by #Oleg was the most accurate, given the circumstances.
Thank you all!
You can also use DBF Commander Pro for this task:
Create command line for your insertion - choose 'File -> Export to DBMS'. Specify transfer options in the window appears, then copy the command line from the bottom of the window:
Create text .BAT file and insert the copied command line, e.g.:
"c:\Program Files\DBFCommander\DBFCommander.exe" -edb "D:\Data\customer.dbf" customer_table "Provider=SQLOLEDB.1;User ID=user1;Initial Catalog=test_db;Data Source=test_server"
Make a schedule using Windows Scheduler that will execute this .BAT file.
Additional info that may be useful for you:
Using DBF in batch mode
Export DBF file to SQL database
I suggest you the next approach:
Create C# script which will use the OleDbConnection (to fetch) and SqlConnection (to upload) objects to import data from the .DBF file to SQL Server database table.
By using LinqPad, LinqPad command-line utility (lprun.exe) and windows Scheduled Task service automate the execution of the mentioned script file
Useful links:
How to get data from DBF file using C#
How to load data into datadase using C#
About LINQPad command-line utility
Another way is create a SQL linked server an ODBC that is pointing at the DBF. Use Windows scheduler to call SQLCMD.EXE to run some SQL to copy the data in.
I have a folder with a .sql files; 1 file per query. I want to execute all queries/ sql files and save them as csv.
Is there a way to do that automated without using the windows cli (disabled in my environment). I do have the SQL Server Management Studio.
I would approach this task using SSIS, providing you have Business Intelligence Development Studio (BIDS) installed.
First create a 'Foreach Loop Container' pointed to the folder with the SQL files, then use a variable to retreive each file name.
Next, create a flat file connection and set the 'Connection String' property to the variable that contains the file location.
Next, using the 'Execute SQL Task' component set the 'SQLSourceType' to 'File Connection' and the 'FileConnection' to the one created in the previous step.
Finally, depending on how the data is returned you have a couple of options, if the result set is small, only a row or a single column, then you can save the results to a variable and using a 'Dataflow' task create a 'Derived Column' component and export the contents of that variable to a CSV file. Or, if the dataset is larger you could dump the results to a temp table and then using an 'OLE DB Source' and 'OLE DB Destination' you could push the full result set straight into a CSV.
Hopefully this isn't too convoluted of a solution, this approach has the advantage of being able be run from either a remote machine or from the server itself, plus you can automate its execution with a SQL Agent Job.
Create a VB.NET console application.
Generate a list of files that end in .SQL from the folder in question.
Load the contents of each file into individual SQL Commands
Execute the SQL Command for each, storing the results in DataSets.
For each table in each dataset, create a new .csv file
For each .csv file, you will need to iterate over each cell in the datatable, and utilize proper escaping for .csv files.
Use 'for' in combination with either sqlcmd or bcp command for each file in the script folder.