I have to weekly upload text files from a server location to Microsoft SQL Server Management Studio .I wish to automate the task so that files are automatically uploaded .Can somebody suggest me the way?
Methods I know of:-
Via SQL:
Use OPENROWSET to open the file and obtain the records to write into
a table.
Use BULK INSERT to open the file and insert directly into a table (you may need to pair with XP_CMDSHELL to get a directory listing to loop through)
VIa SSMS:
Create a DataFlow to import from file
SSMS makes it easier to do clever things with the import process. But it can be very finnicky.
With both of those you can set up an Agent job to run the script / package automatically.
Related
My goal is to script out a database with data like you used to with the sql publishing tool that older versions of SQL Server had
I am using SQL Server Express 2017, and I know how to perform this task manually by right clicking on the database->tasks->generate scripts
Is there a .exe being called here, in older versions of SQL Server this would call sqlpubwiz.exe, and you could call this .exe from a cli, and put into a batch file. I am looking to replace this behavior. Can someone let me know the exe that is being called, and let me know how to script against it.
you can use SQL Dumper, easiest way to export the script data (with Insert query)
for more information :
https://support.microsoft.com/en-in/help/917825/use-the-sqldumper-exe-utility-to-generate-a-dump-file-in-sql-server
to download the utility
https://download.cnet.com/SQL-Dumper/3000-10254_4-10514574.html
System Error: Cannot bulk load because the file "XYZ.txt" could not be opened. Operating system error code 1311(There are currently no logon servers available to service the logon request.)
I have a stored procedure in SQL Server 2008 R2 which is using Bulk Insert command to load data from txt files into SQL Server tables. These files are on a shared folder on a drive located on different domain. I have full access to the drive. I tried copying files to different directory on that drive, moving files, deleting files and everything works.
When I execute the stored procedure from a ssms session from local computer it works like a champ. It is able to open the files on shared drive, read it and load the data into the SQL Server tables without any issue. When I call the stored procedure from a SQL Server Agent job, it throws this error.
SQL Server Agent is using the account which is very powerful with lot more permissions than mine. But the job fails.
To find a workaround I created a SSIS package which calls the stored procedure from "Execute SQL Task". It uses Windows authentication to connect to the database. I tried executing the package and it ran successfully. It is able to upload the data from txt to table.
So, then I created a SQL Server user with my account details, and then used that credentials to create a proxy with ssis sub system. I then scheduled the job to execute the step with newly created proxy to see if it can upload the data. But it failed with the same error.
I am confused what am I doing wrong..? I even added myself to bulkdmin role and ran the job again with no success.
I'll appreciate if someone can help.
Thanks.
Just out of curiosity I tried replacing Bulk Insert command with BCP. For some reason BCP worked. It is able to Open the files on network drive and read through it to insert the data in sql server tables. I can even call the same stored proc from sql agent job and it works perfectly fine. I didn't need to use SSIS package to solve this.
I am working on a Windows Server 2012 64-bit. I want to be able to import data from a .dbf file into a SQL Server table. I used the import wizard and it worked correctly. However, I have SQL Server Express and can't schedule this insertion.
Is there another way to schedule the insertion of the .dbf data to the SQL Server tables, without the use of the SSIS package loader?
Update
I ended up using Python and writing a script to import from XML. However, I believe the answer by #Oleg was the most accurate, given the circumstances.
Thank you all!
You can also use DBF Commander Pro for this task:
Create command line for your insertion - choose 'File -> Export to DBMS'. Specify transfer options in the window appears, then copy the command line from the bottom of the window:
Create text .BAT file and insert the copied command line, e.g.:
"c:\Program Files\DBFCommander\DBFCommander.exe" -edb "D:\Data\customer.dbf" customer_table "Provider=SQLOLEDB.1;User ID=user1;Initial Catalog=test_db;Data Source=test_server"
Make a schedule using Windows Scheduler that will execute this .BAT file.
Additional info that may be useful for you:
Using DBF in batch mode
Export DBF file to SQL database
I suggest you the next approach:
Create C# script which will use the OleDbConnection (to fetch) and SqlConnection (to upload) objects to import data from the .DBF file to SQL Server database table.
By using LinqPad, LinqPad command-line utility (lprun.exe) and windows Scheduled Task service automate the execution of the mentioned script file
Useful links:
How to get data from DBF file using C#
How to load data into datadase using C#
About LINQPad command-line utility
Another way is create a SQL linked server an ODBC that is pointing at the DBF. Use Windows scheduler to call SQLCMD.EXE to run some SQL to copy the data in.
I am using the SQL Server Import/Export Wizard to import data from an Excel file into a table. I will need to do this exact import many times so I selected the option "Save SSIS Package."
I tried saving to SQL Server and I tried saving to File System.
Either way, I cannot figure out how to use the package in SQL Server. I read that I need to use Agent Job Task to do it but I cannot find that anywhere in the SQL Server program.
I also looked for Integrated Services in SQL Server and I found an icon that doesn't seem to do anything.
How do I go from having a .dtsx somewhere on my computer to executing the import (in a query preferably) without having to go through the whole wizard process again and again?
Thank you
The Import/Export Wizard creates a file called .DTSX. This is your SSIS package. An SSIS package is a text file filled with a bunch of XML. Contrast this with something that ends in .EXE/.COM which is an executable that any windows machine will be able to run.
We need something to consume that .dtsx file and perform the ETL operations described therein. There are three options open to you: dtexec.exe, dtexecui.exe, and custom .NET code. dtexecui is just a graphical wrapper to the command line so really, your choices for running packages are the prebuilt command line utility or custom code. Running a package via dtexec can be as simple as dtexec.exe /file C:\MyPackage.dtsx
Since you want to run this from the context of SSMS, then you'll need to either Create a SQL Agent Job or create a stored procedure that calls dtexec. Since you're new to the whole business, the Job will provide the lowest barrier of entry.
In your SSMS, connect to an instance and navigate to the bottom. You should see "SQL Server Agent" Expand that node and Right-Click on "Jobs" and choose New Job. In the Steps tab, click New.... Change the Type to "SQL Server Integration Services Package"
Pick your source (file system or SQL Server) Fill out the server name or where the package lives and you should be good to go.
One very important thing you will need to know is that Excel drivers are only going to work in 32 bit mode. This means you will need to use the dtexec.exe that exists at C:\Program Files (x86)\Microsoft SQL Server\XXX\DTS\Binn\dtexec.exe To get that behaviour from the SQL Agent job, then you will need to add the /X86 option to the command line like
The x86 flag only works for SQL Agent job calls. From the command line, you must use the correct version of dtexec.exe
I have a folder with a .sql files; 1 file per query. I want to execute all queries/ sql files and save them as csv.
Is there a way to do that automated without using the windows cli (disabled in my environment). I do have the SQL Server Management Studio.
I would approach this task using SSIS, providing you have Business Intelligence Development Studio (BIDS) installed.
First create a 'Foreach Loop Container' pointed to the folder with the SQL files, then use a variable to retreive each file name.
Next, create a flat file connection and set the 'Connection String' property to the variable that contains the file location.
Next, using the 'Execute SQL Task' component set the 'SQLSourceType' to 'File Connection' and the 'FileConnection' to the one created in the previous step.
Finally, depending on how the data is returned you have a couple of options, if the result set is small, only a row or a single column, then you can save the results to a variable and using a 'Dataflow' task create a 'Derived Column' component and export the contents of that variable to a CSV file. Or, if the dataset is larger you could dump the results to a temp table and then using an 'OLE DB Source' and 'OLE DB Destination' you could push the full result set straight into a CSV.
Hopefully this isn't too convoluted of a solution, this approach has the advantage of being able be run from either a remote machine or from the server itself, plus you can automate its execution with a SQL Agent Job.
Create a VB.NET console application.
Generate a list of files that end in .SQL from the folder in question.
Load the contents of each file into individual SQL Commands
Execute the SQL Command for each, storing the results in DataSets.
For each table in each dataset, create a new .csv file
For each .csv file, you will need to iterate over each cell in the datatable, and utilize proper escaping for .csv files.
Use 'for' in combination with either sqlcmd or bcp command for each file in the script folder.