I use RedGate SQL data compare and generated a .sql file, so I could run it on my local machine. But the problem is that the file is over 300mb, which means I can't do copy and paste because the clipboard won't be able to handle it, and when I try to open the file in SQL Server Management Studio I get an error about the file being too large.
Is there a way to run a large .sql file? The file basically contains data for two new tables.
From the command prompt, start up sqlcmd:
sqlcmd -S <server> -i C:\<your file here>.sql
Just replace <server> with the location of your SQL box and <your file here> with the name of your script. Don't forget, if you're using a SQL instance the syntax is:
sqlcmd -S <server>\instance.
Here is the list of all arguments you can pass sqlcmd:
Sqlcmd [-U login id] [-P password]
[-S server] [-H hostname] [-E trusted connection]
[-d use database name] [-l login timeout] [-t query timeout]
[-h headers] [-s colseparator] [-w screen width]
[-a packetsize] [-e echo input] [-I Enable Quoted Identifiers]
[-c cmdend] [-L[c] list servers[clean output]]
[-q "cmdline query"] [-Q "cmdline query" and exit]
[-m errorlevel] [-V severitylevel] [-W remove trailing spaces]
[-u unicode output] [-r[0|1] msgs to stderr]
[-i inputfile] [-o outputfile] [-z new password]
[-f | i:[,o:]] [-Z new password and exit]
[-k[1|2] remove[replace] control characters]
[-y variable length type display width]
[-Y fixed length type display width]
[-p[1] print statistics[colon format]]
[-R use client regional setting]
[-b On error batch abort]
[-v var = "value"...] [-A dedicated admin connection]
[-X[1] disable commands, startup script, environment variables [and exit]]
[-x disable variable substitution]
[-? show syntax summary]
I had exactly the same issue and had been struggling for a while then finally found the solution which is to set -a parameter to the sqlcmd in order to change its default packet size:
sqlcmd -S [servername] -d [databasename] -i [scriptfilename] -a 32767
You can use this tool as well. It is really useful.
BigSqlRunner
NB: Broken link, so have updated it.
Take command prompt with administrator privilege
Change directory to where the .sql file stored
Execute the following command
sqlcmd -S 'your server name' -U 'user name of server' -P 'password of server' -d 'db name'-i script.sql
I am using MSSQL Express 2014 and none of the solutions worked for me. They all just crashed SQL. As I only needed to run a one off script with many simple insert statements I got around it by writing a little console app as a very last resort:
class Program
{
static void Main(string[] args)
{
RunScript();
}
private static void RunScript()
{
My_DataEntities db = new My_DataEntities();
string line;
System.IO.StreamReader file =
new System.IO.StreamReader("c:\\ukpostcodesmssql.sql");
while ((line = file.ReadLine()) != null)
{
db.Database.ExecuteSqlCommand(line);
}
file.Close();
}
}
Run it at the command line with osql, see here:
http://metrix.fcny.org/wiki/display/dev/How+to+execute+a+.SQL+script+using+OSQL
Hope this help you!
sqlcmd -u UserName -s <ServerName\InstanceName> -i U:\<Path>\script.sql
I had similar problem. My file with sql script was over 150MB of size (with almost 900k of very simple INSERTs). I used solution advised by Takuro (as the answer in this question) but I still got error with message saying that there was not enough memory ("There is insufficient system memory in resource pool 'internal' to run this query").
What helped me was that I put GO command after every 50k INSERTs.
(It's not directly addressing the question (file size) but I believe it resolves problem that is indirectly connected with large size of sql script itself. In my case many insert commands)
==> sqlcmd -S [servername] -d [databasename] -i [scriptfilename] -a 32767
I have successfully done with this command with 365mb sql file.
this syntax runs in about 15 minutes.
it helped me solve a problem that took me a long time to figure out
Run the script file
Open a command prompt window.
In the Command Prompt window, type: sqlcmd -S <ServerName\InstanceName> -i C:\yourScript.sql
Press ENTER.
Your question is quite similar to this one
You can save your file/script as .txt or .sql and run it from Sql Server Management Studio (I think the menu is Open/Query, then just run the query in the SSMS interface). You migh have to update the first line, indicating the database to be created or selected on your local machine.
If you have to do this data transfer very often, you could then go for replication. Depending on your needs, snapshot replication could be ok. If you have to synch the data between your two servers, you could go for a more complex model such as merge replication.
EDIT: I didn't notice that you had problems with SSMS linked to file size. Then you can go for command-line, as proposed by others, snapshot replication (publish on your main server, subscribe on your local one, replicate, then unsubscribe) or even backup/restore
The file basically contain data for two new tables.
Then you may find it simpler to just DTS (or SSIS, if this is SQL Server 2005+) the data over, if the two servers are on the same network.
If the two servers are not on the same network, you can backup the source database and restore it to a new database on the destination server. Then you can use DTS/SSIS, or even a simple INSERT INTO SELECT, to transfer the two tables to the destination database.
There is probably another way for all the fellows still encountering problems importing really large SQL dumps.
What also be considered when possible: If you have access to the server you could export the database in multiple parts, like first the structure, then per table (or related objects) an export of the data in smaller pieces, instead of one big file.
When you don't have access to server and/or required to use the existing big file, you could try to split them into parts with SQLDumpSplitter: https://philiplb.de/sqldumpsplitter3/.
Then import the pieces to get a full copy of the database.
Good luck, guys.
Related
I have an Azure SQL Server database and a linux box. I have a csv file on the linux machine that I want to import into SQL Server. I have a table already created where I am going to import this file. I have the following questions -
1) Why does this command return an Unknown argument: -S
bcp table in ~/test.csv -S databaseServerName -d dbName -U myUsername -q -c -t
2) How do I import only part of the csv file? It has 20 columns, but I only want to import 2.
3) My table has these two columns - State, Province. My csv file has these two columns that I want to import - State, Region. How do I get Province to map to region.
For #2 and #3, you need to use a BCP format file. This allows you column-level control over which fields from the file go to which columns in the destination and which are left behind (not given a destination).
Use the -f option of BCP and specify the location and name of the format file you want to use. Sorry, no help yet with #1. Have a few questions/suggestions. But im not that familiar with Linux environments.
For part 2 of your question, you can use the Linux cut command to extract just the columns you want. A short awk script can do the same thing (see this SO answer). For both of these, you'll have to identify the "State" and "Region" columns by number. A non-native solution is [querycsv.py][1], which can also rename the "Region" column (disclaimer: I wrote querycsv.py).
For part 3 of your question, you can use the Linux sed command to change the column name on the first line of the CSV file, e.g., sed -e "1s/Region/Province/" file.csv >file2.csv.
I have created a BCP utility and I have wrapped it in a bat file. I have then created a daily task using Task Scheduler in Windows Server 2012.
The function of the BCP utility is to rename a file called 'myfile.csv' (located in C:) by adding a date stamp to it and updating the file with the result of a SQL query.
The codes currently stand as follows:
cd:\Program Files\ Microsoft SQL Server\Client SDK\ODBC\110\Tools\Binn
set vardate=%DATE:~4,10%
set varDateWithoutSlashes=%vardate:/=-%
ren C:\myfile.csv myfile_%varDateWithoutSlashes%.csv
bcp "SELECT TOP 100 ReservationStayID,NameTitle,FirstName,LastName,ArrivalDate,DepartureDate FROM MyDatabase.dbo.GuestNameInfo" queryout C:\myfile.csv -t, -c -S [ipaddress] -U sa -P 1234
My problem is that when the task runs, it renames the file correctly with a the date stamp but it seems that the SELECT query does not run as the file is empty (except the headers, which have been pre-loaded by the way).
What is wrong with my codes?
I should also add the following:
Are the double quotes in the select statement above correct? Or should they be single quotes?
Should the ipaddress in my codes above be in square brackets or should I remove them?
I have left the "Location" filed 'as is' in the Task Scheduler (please see screenshot below). Should that be filled? If yes, by what?
Thanks for helping out!
I was wondering if anyone can help.
I have a number of queries in SQL (all in separate *.sql files). I wanted to know if there is a way to run these queries automatically or mass run them to be saved to either a csv or txt file?
Also, I have come variables within these queries which will need to be amended on a weekly bases before the queries are run.
Thanks.
KJ
Could you please provide some additional help in relation to the variables? Previously I would declare and set variables as:
DECLARE #TW_FROM DATETIME
DECLARE #TW_TO DATETIME
SET #TW_FROM = '2015-11-16 00:00:00';
SET #TW_TO = '2015-11-22 23:00:00';
How do I do this using sqlcmd?
Yes, you can use sqlcmd to do this.
First of all - variables. You can refer to your variables in the .sql files using $(variablename) wherever you want to substitue the variable. For example,
use $(dbname);
select $(columnname) from table1 where column= '$(var1)'
You then call sqlcmd with the following command (note the argument -v variables)
sqlcmd -S servername -d database -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred"
In order to output this to a file, you tag > filename.txt on the end
sqlcmd -S servername -d database -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" > filename.txt
If you want to output to a csv, you can also specify the delimiter using the argument -s (note the idfference with the capital S for server). So now we have
sqlcmd -S servername -d database -s "," -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" > filename.csv
If you want to output several commands to the same csv or txt file, use >> instead of > as it add to teh bottom of the file, rather than replacing it.
sqlcmd -S servername -d database -s "," -i "yoursqlfile.sql" -v dbname="database" columnname="column" var1="Fred" >> filename.csv
To run this for several scripts, you can put the statements in a batch file, and then change the variables every week.
You could write a batch file that uses sqlcmd:
MSDN sqlcmd
That will allow you to call script files in a loop and output the results to a file.
Convert your current scrips to a Stored Procedure.
You can then pass your variables to that and run the query.
If you have SQL Server agent available (SQL standard or better) you can use this to automate the running of the stored procedures.
Otherwise the same can be achieved with Task Scheduler in windows.
As for exporting to CSV this will be useful.
It depends on where your SQL Server is acutally running. It might be quite tricky to write anything to the location you want.
You could read about BCP.
My suggestion is:
Create an UDF (best is inline-UDF!) from all of your queries within your database. Than call them from EXCEL or any other fitting product. You might want to set up an Excel where all your queries are filled one on each Sheet automatically
I have a huge database which I want to dump out using BCP and then load it up elsewhere. I have done quite a bit of research on the Sybase version of BCP (being more familiar with the MSSQL one) and I see how to USE an Import file but I can't figure out for the life of me how to create one.
I am currently making my Sybase bcp out files of data like this:
bcp mytester.dbo.XTABLE out XTABLE.bcp -U sa -P mypass -T -n
and trying to import them back in like this:
bcp mytester.dbo.XTABLE in XTABLE.bcp -E -n -S Sybase_157 -U sa -P SyAdmin
Right now, the IN part gives me an error about IDENTITY_INSERT regardless of if the table has an identity or not:
Server Message: Sybase157 - Msg 7756, Level 16, State 1: Cannot use
'SET IDENTITY_INSERT' for table 'mytester.dbo.XTABLE' because the
table does not have the identity property.
I have often used the great info on this page for help, but this is the first time i've put in a question, so i humbly request any guidance you all can provide :)
In your BCP in, the -E flag tells bcp to take identity column values from the input file. I would try running it without that flag. fmt files in Sybase are a bit finicky, and I would try to avoid if possible. So as long as your schemas are the same between your systems the following command should work:
bcp mytester.dbo.XTABLE in XTABLE.bcp -n -S Sybase_157 -U sa -P SyAdmin
Also, the -T flag on your bcp out seems odd. I know SQLServer -T is a security setting, but in Sybase it indicates the max size of a text or image column, and is followed by a number..e.g -T 32000 (would be 32Kbytes)
But to answer the question in your title, if you run bcp out interactively (without specifying -c,-n, or -f) it will step through each column, prompting for information. At the end it will ask if you want to create a format file, and allow you to specify the name of the file.
For reference, here is the syntax and available flags:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc30191.1550/html/utility/X14951.htm
And the chapter in the Utility Guide:
http://infocenter.sybase.com/help/index.jsp?topic=/com.sybase.infocenter.dc30191.1550/html/utility/BABGCCIC.htm
I thought it is a very simple task to export data in a view from SQL Server 2005 to a fixed width text file. But the wizard is a pain. The format is not correct. Does anybody know how to deal with it? or any better way to do that?
Use bcp with queryout option
http://msdn.microsoft.com/en-us/library/ms162802(SQL.90).aspx
bcp "SELECT * FROM AdventureWorks.Person.Contact" queryout Contacts.txt -c -T
Use a format file if you want fixed-width output
http://weblogs.sqlteam.com/brettk/archive/2006/07/06/10504.aspx
I just tried exporting in AdventureWorks, Fixed Width gave me a lot of issues too (compared to column delimiters)
I had to ignore the GUID columns, not include column names in the first row, etc.. finally exported Sales.Customer table
Without BCP, you may have to look into SSIS or SQLCMD
One option is to use OpenDataSource to write the rows out to a text file. This requires that the text file already exists, but is relatively easy.
This article (for disclosure, I wrote it, and it focuses on reading from, but touches on writing too) explains the basics of reading and writing from text files with OpenDataSource. : http://www.sqlservercentral.com/articles/OpenDataSource/61552/
If the format from the wizard doesn't suit your needs, you will need to develop your own SSIS package. Are you sure you really need a fixed width file? A delimited file would probably be easier to get right as they are much more common.
I have found SQL2005/SSIS to be anything but simple to do seemingly simple tasks. Things always take me 5x longer than they should. I know I'm not alone on this.
BCP should work.
Don't know about sqlcmd - seems like it doesn't do fixed width, unless you did the padding in your sql statement - which would also be a pain.
usage: Sqlcmd [-U login id] [-P password]
[-S server] [-H hostname] [-E trusted connection]
[-d use database name] [-l login timeout] [-t query timeout]
[-h headers] [-s colseparator] [-w screen width]
[-a packetsize] [-e echo input] [-I Enable Quoted Identifiers]
[-c cmdend] [-L[c] list servers[clean output]]
[-q "cmdline query"] [-Q "cmdline query" and exit]
[-m errorlevel] [-V severitylevel] [-W remove trailing spaces]
[-u unicode output] [-r[0|1] msgs to stderr]
[-i inputfile] [-o outputfile] [-z new password]
[-f <codepage> | i:<codepage>[,o:<codepage>]] [-Z new password and exit]
[-k[1|2] remove[replace] control characters]
[-y variable length type display width]
[-Y fixed length type display width]
[-p[1] print statistics[colon format]]
[-R use client regional setting]
[-b On error batch abort]
[-v var = "value"...] [-A dedicated admin connection]
[-X[1] disable commands, startup script, enviroment variables [and exit]]
[-x disable variable substitution]
[-? show syntax summary]