I have CSV files stored in SFTP Server which I am accessing using winscp.
I have to import these CSVs into SQL Server tables.
Currently I am downloading files from SFTP and saving them in O Drive and created SSIS packages to import in SQL Server(Using O Drive as source)
How can I import data directly from SFTP server to SQL Server?
I would like to use SSIS for the same but would like to know other methods as well.
Thanks,
AP
There's a free tool called WinSCP which provides a command line utility to communicate with SFTP sites. You just need to understand the scripting language for the command line and execute the executable from SSIS.
Here you can find about this method: https://www.codeproject.com/Articles/23740/SFTP-with-SSIS-Packages
Another method to load data from SFTP to SQL Server is to use ETL tools, like Skyvia, Alooma, etc.
I use a Powershell script to copy files to an accessible location & then copy contents into a SQL table, not sure if you can skip a step & read the contents directly from the secure location (seem to remember I couldn't get it to work though).
It's a modified version of the powershell example here, followed by Import-Csv combined with Out-Datatable and Write-DataTable
There's further WinSCP code examples here
Related
I have to weekly upload text files from a server location to Microsoft SQL Server Management Studio .I wish to automate the task so that files are automatically uploaded .Can somebody suggest me the way?
Methods I know of:-
Via SQL:
Use OPENROWSET to open the file and obtain the records to write into
a table.
Use BULK INSERT to open the file and insert directly into a table (you may need to pair with XP_CMDSHELL to get a directory listing to loop through)
VIa SSMS:
Create a DataFlow to import from file
SSMS makes it easier to do clever things with the import process. But it can be very finnicky.
With both of those you can set up an Agent job to run the script / package automatically.
Given the DAT file and the DDL file for each table in a DB2 database, can I import this data to SQL Server? I have no access to the original server or any copy of a DB2 server so connecting to a live instance isn't an option.
Can I do this without a live instance of DB2 or should I go back to the client and ask for CSV files? Is there a procedure or tool that makes this process smoother? I've tried to find a file-based connection string to use to connect to a set of DB2 files with no luck. I've also tried SwissSQLDB2ToSQLServer and SqlLinesData to see if they have a file-based option built in.
OK, given the comment above, you can't import DB2's container files (DAT, LRG, or anything else) directly. You need a CSV or equivalent. Yes, one way to get this is run the EXPORT utility on a live DB2 database. HTH!
I am working on a Windows Server 2012 64-bit. I want to be able to import data from a .dbf file into a SQL Server table. I used the import wizard and it worked correctly. However, I have SQL Server Express and can't schedule this insertion.
Is there another way to schedule the insertion of the .dbf data to the SQL Server tables, without the use of the SSIS package loader?
Update
I ended up using Python and writing a script to import from XML. However, I believe the answer by #Oleg was the most accurate, given the circumstances.
Thank you all!
You can also use DBF Commander Pro for this task:
Create command line for your insertion - choose 'File -> Export to DBMS'. Specify transfer options in the window appears, then copy the command line from the bottom of the window:
Create text .BAT file and insert the copied command line, e.g.:
"c:\Program Files\DBFCommander\DBFCommander.exe" -edb "D:\Data\customer.dbf" customer_table "Provider=SQLOLEDB.1;User ID=user1;Initial Catalog=test_db;Data Source=test_server"
Make a schedule using Windows Scheduler that will execute this .BAT file.
Additional info that may be useful for you:
Using DBF in batch mode
Export DBF file to SQL database
I suggest you the next approach:
Create C# script which will use the OleDbConnection (to fetch) and SqlConnection (to upload) objects to import data from the .DBF file to SQL Server database table.
By using LinqPad, LinqPad command-line utility (lprun.exe) and windows Scheduled Task service automate the execution of the mentioned script file
Useful links:
How to get data from DBF file using C#
How to load data into datadase using C#
About LINQPad command-line utility
Another way is create a SQL linked server an ODBC that is pointing at the DBF. Use Windows scheduler to call SQLCMD.EXE to run some SQL to copy the data in.
Can I read a XML file from remote server such as
http://dealer.somedomain.com/data/inventory.xml
I need read and get data from this remote XML file and update some local tables.
I use SQL Server 2005.
Thanks in advance.
Your task can be split into two sub-tasks.
Downloading the file over HTTP to your local PC (to a temporary folder).
Importing the XML file into SQL Server.
You can use SSIS for both. Once you look for "SSIS downloading over HTTP" or "SSIS downloading from website" you will find many tutorials for file download.
For the second step you will need data import, there are plenty tutorials as well, here is just an interesting pick.
You can find ready solutions including data download and import all-in-one, like here.
Is it possible to somehow use a .bat file to script the schema and/or content of a SQL Server database?
I can do this via the wizard, but would like to streamline the creation of this file for source control purposes.
I would like to avoid the use of 3rd party tools, just limiting myself to the tools that come with SQL Server.
There is a free tool called SubCommander that is a part of the open source SubSonic software. I have successfully used this tool myself to create both schema and data "dumps" each night.
You can script out your schema and
data (and then version it in your
favorite source control system) using
SubCommander. Simply use the command
"version" and tell SubCommander where
to put the data:
sonic.exe version /out Scripts
This will output a script file (.sql)
to the local scripts directory of your
project
You can also try using the Microsoft SQL Server Database Publishing wizard, although i am not sure that you can use it in a bat file.