Basically what the title is saying.
Today when I use bulk insert with T-SQL on Microsoft SQL Server 2008 to get data from a local drive I use the following query,
BULK INSERT tmp_table
FROM 'c:\data\x.csv'
WITH ( FIELDTERMINATOR = ',', ROWTERMINATOR = '0x0a' );
Which works ok. But now I want to be able to read .csv data files from a folder on a FTP server.
As every other FTP server, I need username\password to log in first, and then, I somehow need to fetch the files. Is this possible with T-SQL?
I do know that with C# this would be a piece of cake for me, but want to learn this by using T-SQL.
I will also need to know how I can dynamically get the names of files from a given folder on the FTP server, but since I'm taking this one step at a time, you don't need to answer this right away.
When the files are on local drive i am able to use xp_dirtree to get the names of all files in a folder.
An excellent guide can be found here http://www.patrickkeisler.com/2012/11/how-to-use-xpdirtree-to-list-all-files.html
Related
I have some .xml files in a local folder (for example, C:\xml). I also have a table in SQL Server with columns Name (varchar) and XMLContent (xml).
How can I upload all my files from local folder to SQL Server, compare them and update in case they are different or insert a new entry containing filename without .xml and xml content if file is absent on server?
Is it solvable? Maybe there are a ready solution for this action? I'm not very good at SQL.
Looks like I've found the perfect solution. It does exactly what I wanted.
Upload multiple XML files
I recently had to move my files to a new SSIS server. Everything seems to be working except when I try to execute a bulk insert it tells me
(Cannot bulk load because the file "E:\FlatFiles\SSG\apmast.txt" could
not be opened. Operating system error code 21(The device is not
ready.).".
It does this for all my flat files. I found an article saying you need to give the MSSQLSERVER user full control of the files, which I did but this does not seem to fix it. Any other ideas? Do I need to give other files the same permissions? I really don't want to just throw full control around if I don't have to. Thanks
I figured it out, turns out that a bulk insert tells the server to look locally for the text file. I was trying to get SSIS to do a bulk insert of flat files from one server into another sql server on the network. As soon as I put the flatfiles on the remote server it grabbed and used them. This seems like a very odd way for it to work. I would expect it to push the files to the sql server instead of asking the sql server to look for the files locally via a hard path.
I'm trying to use Bulk Insert to insert some data into my database on local computer. I am using a SQL Server Express database and executing query using Microsoft SQL Server Management Studio. When I try to execute a query I get this error.
Cannot bulk load because the file "D:\Countries.xml" could not be opened. Operating system error code 123(The filename, directory name, or volume label syntax is incorrect.).
I have been moving this file over my HDD by everywhere and still got the same error. To be honest I have no idea what is going on. Any tips?
BULK INSERT Research.dbo.Countries
FROM 'C:\Users\someuser\Desktop\Localization DB\countryInfo.txt' WITH
(
FORMATFILE='D:\Countries.xml',
FIRSTROW=2,
CHECK_CONSTRAINTS
);
GO
when you copy a path for example from properties of file
copied text have some extra thrash bytes, you need to view pasted text as ANSI and will be revealed
Are you 100% sure you are connecting to you locally installed SQL Server - not a networked one?
That's my favorite thing to do, rdp into a machine, or on a VM, and forget where I am physically connected. If you copy and paste from Word, sometimes it puts in the wrong type of ' back that out and replace.
I have no idea why but the problem was with ' mark being copied. When i deleted it and put it from keyboard it started working.
I'm fairly new to SQL Server.
I'm trying to bulk insert into a table, using the command in SQL Server Management Studio (2005):
BULK INSERT Table1 FROM 'c:\text.txt' WITH (FIELDTERMINATOR = '|')
I get the error:
Msg 4860, Level 16, State 1, Line 1
Cannot bulk load. The file "c:\text.txt" does not exist.
I'm positive the file actually exists.
I get the feeling that it is looking for the file on the local hard drive for where ever the server is. Is that the case? If so, how do you generally solve this problem? (to note, I've tried specifying the network address of my PC when entering the location of the text file, but I get a permission error. Also, I know in advance that my company doesn't allow files to placed on a server).
SQL Server does not have an SQL statement that reads data from the client end (as the other posters have pointed out). Other RDBMS products do implement this (eg. the Postgres COPY statement lets you specify a file on the server or a file on the client that is read by the db connectivity library on the client side).
You can achieve moving data from a file on the client to a table on SQL Server using the bcp command line program.
bcp lets you copy data from a local file to a table on the server, or from a table (or select query) on the server into a local file. For example:
bcp servername.dbname.tablename in c:\temp.txt -T -c
will copy a tabbed delimited file (temp.txt) into the specified table (assuming the file contains the right number of columns).
I am not sure if this helps, but it is the only way to move data from a client file to a server table without giving the server some sort of access across the network to the data file on client.
I'd agree that it's a problem with the file being on your C drive, and not the server's drive.
If it's a permissions issue, have you tried creating a file share on your workstation that the server does have permissions to read from? Maybe something like \YourWorkstation\SQLFile, and then granting everyone (or Guest, depending on how your network permissions are set up) read access on it?
If you can't create the share on your laptop, or you can't grant rights to it for some reason, is there a file share somewhere in the office that you do have rights to, and that SQL can also read from? Maybe a NAS or a "Common" network folder?
Have you created a share drive on your machine that the server can see? If so then you just need to refer to the path including your machine name instead of C:
Yes, it will look for the file on the SQL server itself.
If you can map a network drive to the C drive of your SQL server, then you can just copy the file over before running the bulk insert.
If you absolutely can't get any access to the server's file system, then you can look at doing something like this:
write a program that reads your text file and inserts the contents into single record in a temporary table that has a Text field, perhaps using a stored procedure
have the program execute the bcp command to export the data from the temporary table into a text file on the SQL server's local file system, to a folder that the account under which the SQL service is running has write permission
have the program run a bulk insert command specifying the path to the text file on the server
delete the text file and the temporary table
A while back I needed to parse a bunch of Serve-U FTP log files and store them in a database so people could report on them. I ended up developing a small C# app to do the following:
Look for all files in a dir that have not been loaded into the db (there is a table of previously loaded files).
Open a file and load all the lines into a list.
Loop through that list and use RegEx to identify the kind of row (CONNECT, LOGIN, DISCONNECT, UPLOAD, DOWNLOAD, etc), parse it into a specific kind of object corresponding to the kind of row and add that obj to another List.
Loop through each of the different object lists and write each one to the associated database table.
Record that the file was successfully imported.
Wash, rinse, repeat.
It's ugly but it got the job done for the deadline we had.
The problem is that I'm in a DBA role and I'm not happy with running a compiled app as the solution to this problem. I'd prefer something more open and more DBA-oriented.
I could rewrite this in PowerShell but I'd prefer to develop an SSIS package. I couldn't find a good way to split input based on RegEx within SSIS the first time around and I wasn't familiar enough with SSIS. I'm digging into SSIS more now but still not finding what I need.
Does anybody have any suggestions about how I might approach a rewrite in SSIS?
I have to do something similar with Exchange logs. I have yet to find an easier solution utilizing an all SSIS solution. Having said that, here is what I do:
First I use logparser from Microsoft and the bulk copy functionality of sql2005
I copy the log files to a directory that I can work with them in.
I created a sql file that will parse the logs. It looks similar to this:
SELECT TO_Timestamp(REPLACE_STR(STRCAT(STRCAT(date,' '), time),' GMT',''),'yyyy-M-d h:m:s') as DateTime, [client-ip], [Client-hostname], [Partner-name], [Server-hostname], [server-IP], [Recipient-Address], [Event-ID], [MSGID], [Priority], [Recipient-Report-Status], [total-bytes], [Number-Recipients], TO_Timestamp(REPLACE_STR([Origination-time], ' GMT',''),'yyyy-M-d h:m:s') as [Origination Time], Encryption, [service-Version], [Linked-MSGID], [Message-Subject], [Sender-Address] INTO '%outfile%' FROM '%infile%' WHERE [Event-ID] IN (1027;1028)
I then run the previous sql with logparser:
logparser.exe file:c:\exchange\info\name_of_file_goes_here.sql?infile=c:\exchange\info\logs\*.log+outfile=c:\exchange\info\logs\name_of_file_goes_here.bcp -i:W3C -o:TSV
Which outputs a bcp file.
Then I bulk copy that bcp file into a premade database table in SQL server with this command:
bcp databasename.dbo.table in c:\exchange\info\logs\name_of_file_goes_here.bcp -c -t"\t" -T -F 2 -S server\instance -U userid -P password
Then I run queries against the table. If you can figure out how to automate this with SSIS, I'd be glad to hear what you did.