I wish to write a query that inserts a file that resides on the client (C# web server) into a column in the database server (SQL Server), something like INSERT … SELECT * FROM OPENROWSET(BULK…), but without having to save the file on the server machine first.
Is this even possible in SQL?
Although your context is unstated, I'm assuming that you're intending to run this from SSMS rather than from OSQL, a PowerShell script, or through some other means.
The file doesn't need to reside on the physical box running SQL Server, but SQL Server does need access to it. The typical approach, I believe, would be for an application server to copy the file to a shared repository and then pass it off to SQL Server through a UNC reference. The syntax to do so is relatively trivial and can be found in Importing Bulk Data by Using BULK INSERT or OPENROWSET(BULK...).
If instead you're interested in providing a mechanism for the SQL Server to save a file from some type of stream operation where the client is directly transmitting a file and there is no shared repository, I'm not aware of a way to do that. Even if you use an SQL FILESTREAM object you still need an accessible NTFS location to stream from. See Saving and Retrieving File Using FileStream SQL Server 2008.
At some point, the server will have to have a hand on the file. That does not mean that the server has to keep the file, but the file has to get to the server in order to be read and inserted into the db. Typically, this is achieved with a form and a file-type input. On the server, you can use the uploaded file to create your query, then delete it.
That said, storing files in a database is a debatable practice. Depending on the type and size of files you're storing, your database can quickly balloon in size. For starters, this makes backups slower and more prone to failure, along with a laundry list of other potential pitfalls. Check out this question on SO: Storing Images in DB - Yea or Nay? As you can see from the answers, there are a number of considerations to be made, but a good rule of thumb is to not do this unless you have compelling reasons to do so.
In SQL Server BLOB Data in .NET: Tutorial, Mohammad Elsheimy explains how it can be done:
using(SqlConnection con = new SqlConnection(conStr) )
using(SqlCommand command = new SqlCommand("INSERT INTO MyFiles VALUES (#Filename, #Data)", con) )
{
command.Parameters.AddWithValue("#Filename", Path.GetFileName(filename));
command.Parameters.AddWithValue("#Data", File.ReadAllBytes(filename));
connection.Open();
}
Basically, this way the file is read on the client and sent to the database server without a need for a temporary file on the server machine.
Related
I'm looking to create a text file, which will contain SQL code to create a database and its tables, and to later on, modify the same database.
The text file will be read via an application the user installs, and when it runs, it should read the text file and create, or modify the database if any changes have been applied.
The SQL text file should of course, be somewhat validated in order to not duplicate tables and such.
I'm not asking for any code, just a specific pathway I should follow in order to make this happen.
Thanks for your input.
I'd do database creation via a SQL script which checks for the existence of tables/views/SPs/etc. before creating them, then I'd execute it in the VB application via ADO.NET. I'd ship it with the application in a subdirectory. It's not a big deal to read text files, or to execute a SQL string via ADO.NET.
I'd have a VERSION table in the database that identifies what DB schema version is installed, and when I shipped upgrade scripts which modified the DB, I would have them update the VERSION table. The first version you ship is 1.0, increment as appropriate thereafter.
All the SQL object creation/detection/versioning logic would be in SQL. That's by far the simplest way to do it on the client, it's the simplest thing to develop and to test before shipping (MS SQL Management Studio is a godsend), it's the simplest thing to diff against the previous version, store in source control, etc.
Incidentally, I would also have my application interact with the database strictly via stored procedures, and I would absolutely never, ever feed SQL any concatenated strings. All parameters going to SQL should be delivered via ADO.NET's SqlParameter mechanism, which is very cool because it makes for clean, readable code, and sanitizes all of your parameters for you. Ever use a DB application that crashed on apostrophes? They didn't sanitize their parameters.
If what you are asking is How do I read a text file and make the results execute in SQL
I would use a StreamReader to read the text file into a string variable.
Once you have read it in, go ahead and open a connection to the database and do a ExecuteNonQuery with the value of the string variable.
I would post in the comments but I can't. I think this may be what you are looking for.
Is it possible to execute a text file from SQL query?
Use MS SQL Server Management Studio to perfect your scripts:
http://technet.microsoft.com/en-us/library/ms174173.aspx
SSMS comes with the server installs and is available for the SQL Express versions. (It isn't needed on the client PCs but it may be useful for debugging.
This will most likely be a low security environment and each user will have full control of the DB.
For there it is pretty straight forward to read the text file and run it against the DB. Just get a connection and send the script:
Dim cmd As New Data.SqlClient.SqlCommand
con.Open()
cmd.CommandText = SQL
cmd.Connection = con
cmd.ExecuteNonQuery()
You might want to use a virtual machine on your development PC as it will allow you to quickly do testing of your scripts and code, and return to baseline state.
My question is about where I should put a file I need to use in a BULK INSERT command in MS SQL.
I have a database running on a server. I run queries on this through an ODBC connection from my machine on the same network. I want to create a stored procedure that will use Bulk Insert to import data from a .txt file and then execute this stored procedure from my machine (from clicking a button in an Excel sheet).
I'm no expert on how SQL Server actually works to say the least so I have what I imagine is a very basic question for someone who does. Does the .txt file used in the Bulk Insert need to be in a location that can be read by:
a) my machine e.g. on it's local hard disk
or
b) on a location that can be read by the database server e.g. somewhere on the network that it can access
I'm not sure if my local machine or the server is actually opening the file. I would assume it's the server, but I'd like to be sure!
Many thanks in advance for your help!
If the file is not on the server computer, then you will need to make sure SQL Server has access to the file. This becomes a permissions issue. In particular, the permission you need to look at is the log on account for the SQL Server service. Open the services control panel, locate the SQL Server service and check the log on account in the properties section.
As a general rule, it's not a good idea to give too many network permissions to the SQL Server service account because this can allow hackers access to resources outside the server computer.
If you mean the BULK INSERT command from T-SQL then the name of the 'data_file' can be a local filename or an UNC path, local that is to the SQL Server machine running the query. You can put the file to import from on a share hosted on the SQL Server machine or on any other share in the network the SQL Server has access to.
I use a .mdf file as my database. I also used Linq-to-SQL connection.
At first, my application can add and display the data in the .mdf file correctly. But when I try to see the data by clicking the .mdf file directly and from the table, the database is empty.
I've set the .mdf file and the .dbml file Copy if newer, but still nothing.
Please help me and thank you for your help.
I bet your connection string contains
AttachDbFileName=somefilename.mdf
The whole User Instance and AttachDbFileName= approach is flawed - at best! Visual Studio will be copying around the .mdf file and most likely, your INSERT works just fine - but you're just looking at the wrong .mdf file in the end!
If you want to stick with this approach, then try putting a breakpoint on the myConnection.Close() call - and then inspect the .mdf file with SQL Server Mgmt Studio Express - I'm almost certain your data is there.
The real solution in my opinion would be to
install SQL Server Express (and you've already done that anyway)
install SQL Server Management Studio Express
create your database in SSMS Express, give it a logical name (e.g. YourDatabase)
connect to it using its logical database name (given when you create it on the server) - and don't mess around with physical database files and user instances. In that case, your connection string would be something like:
Data Source=.\\SQLEXPRESS;Database=YourDatabase;Integrated Security=True
and everything else is exactly the same as before...
I am developing a win32 windows application with Delphi and MS SQL Server. it works fine in LAN but I am trying to add the support for SQL Server remote connections (= working with a DB that can be accessed with an external IP, as described in this article: http://support.microsoft.com/default.aspx?scid=kb;EN-US;914277).
Basically I have a Table in DB where I keep the DocumentID, the document description and the Document path (like \\FILESERVER\MyApplicationDocuments\45.zip).
Of course \\FILESERVER is a local (LAN) path for the server but not for the client (as I am now trying to add the support for remote connections).
So I need a way to access \\FILESERVER even if of course I cannot see it in LAN.
I found the following T-SQL code snippet that is perfect for the "download trick":
SELECT BulkColumn as MyFile FROM OPENROWSET(BULK '\FILESERVER\MyApplicationDocuments\45.zip' , SINGLE_BLOB) AS X
With the code above I can download a file on the client.
But how to upload it? I need an "Uppload trick" to be able to insert new files, but also to delete or replace existing files.
Can anyone suggest? If a trick is not available could you suggest an alternative? Like an extended stored procedure or calling some .net assembly from the server.
If you have sql 2008, then you can use FILESTREAM, then sql server will automatically throw it out to disk.
If you have sql 2005, I'd consider just moving the data into a varbinary(max) column and deal with it that way (also pretty simple).
If neither of those apply OR you can't shove it into a varbinary column, then I would not use sql server to handle the actual file contents and instead just have a web service which stored the file on the file system or a SAN that the web service can easily access. (same as IMHO)
UPDATE:
One other idea that crossed my mind. If you are using SQL 2005/08 then you can write a CLR Stored procedure in .Net. This could handle transferring the blob data to / from the local file system.
In ideal world I would create simple:
- ASP.NET Web Service
- or .Net Remoting Service (faster than web service)
- or new .Net 4.0 RIA service.
Deploy it to the SQL Server on custom TCP/IP port
This service would listen to the port and client would request the file via the service. The service would get the file via local LAN and communicate with the DB via local OLE DB connection.
I would not use any SQl Server "web service" support - this is security and performance issues.
UPDATE:
Since this is Delphi app - you can do the same using Delphi, even though above solution still valid, but more work to integrate different technologies. Delphi has its own tools to build remote applications
If you are on 2005, you could try to store file in temp blob field of some temp table, and then call stored procedure which should put the file where you want it, and update path field as you want it.
In that stored procedure you must use extended stored procedures (xp_something), which allow access to file system. That means that those should be enabled for sql server.
BTW You are trying to use relational DB as Document database. That will, sooner or later, backfire.
I'm fairly new to SQL Server.
I'm trying to bulk insert into a table, using the command in SQL Server Management Studio (2005):
BULK INSERT Table1 FROM 'c:\text.txt' WITH (FIELDTERMINATOR = '|')
I get the error:
Msg 4860, Level 16, State 1, Line 1
Cannot bulk load. The file "c:\text.txt" does not exist.
I'm positive the file actually exists.
I get the feeling that it is looking for the file on the local hard drive for where ever the server is. Is that the case? If so, how do you generally solve this problem? (to note, I've tried specifying the network address of my PC when entering the location of the text file, but I get a permission error. Also, I know in advance that my company doesn't allow files to placed on a server).
SQL Server does not have an SQL statement that reads data from the client end (as the other posters have pointed out). Other RDBMS products do implement this (eg. the Postgres COPY statement lets you specify a file on the server or a file on the client that is read by the db connectivity library on the client side).
You can achieve moving data from a file on the client to a table on SQL Server using the bcp command line program.
bcp lets you copy data from a local file to a table on the server, or from a table (or select query) on the server into a local file. For example:
bcp servername.dbname.tablename in c:\temp.txt -T -c
will copy a tabbed delimited file (temp.txt) into the specified table (assuming the file contains the right number of columns).
I am not sure if this helps, but it is the only way to move data from a client file to a server table without giving the server some sort of access across the network to the data file on client.
I'd agree that it's a problem with the file being on your C drive, and not the server's drive.
If it's a permissions issue, have you tried creating a file share on your workstation that the server does have permissions to read from? Maybe something like \YourWorkstation\SQLFile, and then granting everyone (or Guest, depending on how your network permissions are set up) read access on it?
If you can't create the share on your laptop, or you can't grant rights to it for some reason, is there a file share somewhere in the office that you do have rights to, and that SQL can also read from? Maybe a NAS or a "Common" network folder?
Have you created a share drive on your machine that the server can see? If so then you just need to refer to the path including your machine name instead of C:
Yes, it will look for the file on the SQL server itself.
If you can map a network drive to the C drive of your SQL server, then you can just copy the file over before running the bulk insert.
If you absolutely can't get any access to the server's file system, then you can look at doing something like this:
write a program that reads your text file and inserts the contents into single record in a temporary table that has a Text field, perhaps using a stored procedure
have the program execute the bcp command to export the data from the temporary table into a text file on the SQL server's local file system, to a folder that the account under which the SQL service is running has write permission
have the program run a bulk insert command specifying the path to the text file on the server
delete the text file and the temporary table