A project I'm working on at work involves modifying one of the subsystems to store/pull data that is currently stored in files into the database. Each of the files is a single, sometimes-large, chunk of custom (xml-based) script generated by another custom tool.
Conceptually, I'm looking for an easy way to do something like:
For Each file in folder_and_subfolders
INSERT INTO table
(script_name, version_num, script )
VALUES
({file_name}, 1, {file_contents})
;
Next
Preferably on an entire directory tree at once.
If there's no easy way to do this via T-SQL, I can write a utility to do the job, but I'd prefer something that didn't require having to write another custom tool that will only be used once.
So, I don't have SQL Server installed and therefore can't test this, but if you are looking for a simple batch file that could do what you're after, I'd suggest something like the following might well help;
#echo off
SET xmldir=./myxmlfiles/live/here/
echo --- Processing files
for %%f in ("%xmldir%*.xml") do (echo Running %%f.... && #sqlcmd -I -U %1 -P %2 -S %3 -d %4 -v filename="%xmldir%%%f" -i ProcessFile.sql)
I'm not sure how much you know about sqlcmd, but it is a command line tool that is generally provided by SQL Server. It will allow you to run SQL commands, or in the case above, run a script which is indicated by the -i parameter. I am assuming that you'd place your SQL statement in there to perform your additions to the table.
The other parameters to sqlcmd are described below;
-I sets QUOTED_IDENTIFIER on (you may or may not need this. I did for an earlier issue I faced with sqlcmd and QUOTED_IDENTIFIER)
-U sets the database username
-P sets the database password
-S sets the database server
-d sets the database to connect to
-v is the interesting one here as it lets you pass parameters to your script. Note that on the MSDN page describing this, it states that if your path or filename contains spaces, then you'll need to enclose it in quotes, so check that out. Basically though, you'd be able to refer to the parameter inside your sql script (ProcessFile.sql) like INSERT INTO mytable (file_name) VALUES ('$(filename)')
You'd have to use the logic described in the answer from my previous comment to ensure
Related
I was wondering if there is any way to copy a data from csv file by using command line (CMD) to a table from Vertica Data Base. I've searching for a while and I don't find something that works for me. I want to create something similar to this I've made, is a batch file to copy a data from CSV file to a table in postgreSQL Data Base:
C:
cd "C:\pgsql\bin"
psql -h suggestedorder.postgres.database.azure.com -d DataAnalytics -U dev_ext#suggestedorder -c "\copy planning.sap_mx_env_me2n_zaaptr_zpaitr_envase_transito (planta, folio, des_proveedor, fecha_carga, hora)from 'C:\Users\geradiaz.MODELO\Desktop\Envase\Transitos\Outputs\Transitos.csv' with (format csv, header)
(To avoid CMD asked for my password I created The password file)
Do you know guys, if there is any way to make something similiar? do I need to create The Password file like in posgreSQL first? I hope you can help me or tell me where I can find info about how to create this properly.
Have a nice day, Best regards!
So you are in a DOS box of a Windows system.
Add an environment variable to your account, by hitting the magnifying glass next to the Windows button at the far low left corner of your Windows screen, and type "Environment variables". You will see a list of hits, among which "Edit Environment Variables for your Account" and "Edit the system environment variables". Click on either of the two, depending on whether you want it for yourself or for the whole Windows system.
Add 4 variables with the [New ...] button, for example:
Variable Value
VSQL_USER dbadmin
VSQL_PASSWORD dbadmin_s_password
VSQL_HOST <ip address or host name>
VSQL_DATABASE <database name, for example vmart>
and edit the variable:
Variable Value
PATH <what is already there>;C:\Program Files\Vertica Systems\VSQL64
then go:
cd /d <the directory where your data file resides, not necessarily vsql.exe>
vsql -c "COPY planning.sap_mx_env_me2n_zaaptr_zpaitr_envase_transito ^
(planta, folio, des_proveedor, fecha_carga, hora) ^
FROM LOCAL 'C:\Users\geradiaz.MODELO\Desktop\Envase\Transitos\Outputs\Transitos.csv' ^
DELIMITER ',' SKIP 1 EXCEPTIONS 'con' REJECTMAX 3"
I developped the following T-SQL script with a bit of SQLCMD to call separate stored procedure script for their creations. The stored procedures are located in a child Store Procedure\ folder (I want to keep the space).
They are probably better ways to perform it, but I wan to a unique script (but using files) which creates the database, the tables, the stored procedures and so on.
I detail neither the the database, nor the table scripting.
/* Create database (I still have some issues with dropping it (existing connections) */
*/ Create tables with constraints... */
-- Is it possible to declare a relative path (here MY_PARENT_PATH is replaced by "c:\...")
:setvar STORE_PROCEUDRE_PATH "[MY_PARENT_PATH]\Stored Procedures"
-- Calls CRUD stored procedures from separate scripts via SQLCMD - is it possible to batch it with all spXXX_XXXX.sql files included in the sub-folder?
:r $(STORE_PROCEUDRE_PATH)\spFields_Create.sql
:r $(STORE_PROCEUDRE_PATH)\spFields_Read.sql
:r $(STORE_PROCEUDRE_PATH)\spFields_Update.sql
:r $(STORE_PROCEUDRE_PATH)\spFields_Delete.sql
-- Idem for other table, such as Features, Values, etc.
Is it possible to loop all files spXXX_XXXX.sql files within the Store Procedure\ folder and execute the :r $(STORE_PROCEUDRE_PATH)\spXXX_XXXX.sql scripts?
I came accross several articles which show example using a FOR construct, but I got confused.
Thanks for any insights :-)
yes there is a way ,
SQLCMD is not able to loop through files but you can use Windows Batch Scripting to accomplish that.
here is a good blog about how to do it :
SQLCMD and Batch File magic
so basically you need to make a .bat file ex: CRUDE.bat and edit and paste below code in it and save it ,
##echo off
cd "[MY_PARENT_PATH]\Stored Procedures"
FOR %%A IN (*.SQL) DO ( sqlcmd -S [SERVERNAME] -d [DATABASE] -U [][username -P password] -i "%%A")
you need to replace these values with your values :
MY_PARENT_PATH : Your path or Directory where your suborder is.
SERVERNAME : Your Database Server Name.
DATABASE1 : Your Database name.
Username : Your SQL Username.
Password : Your SQL Password.
now you can run the batch file and it does the magic.
also here you can find out more about SQLCMD utility
I'm trying to set up a batch script that basically runs a SQL statement against a database, and if the script returns results it will follow some logic.
Is there a way to have SQLCMD actually return the number of rows it found, or something similar?
I see that I can have the output displayed on the screen or a file, but is there a way to have it put it into a variable so I can have the script evaluate the variable? For example:
SQLCMD -q "select count(*) from active_connections" -r #varactive
IF #varactive > 0 THEN
<do things>
ELSE END
Or would I need to switch to Powershell to handle this sort of logic?
While #Gary is technically correct that the only thing returned is the ERRORLEVEL, sqlcmd does also display its results to STDOUT. Armed with that, you could do something like this in a batch file:
set SERVERNAME=yoursqlserver
for /f "skip=2" %%x in ('sqlcmd -S %SERVERNAME% -Q "select count(*) from active_connections" ^| findstr /v /c:"rows affected"') do set COUNT=%%x
echo There are %COUNT% records in the active_connections table.
See Docs for sqlcmd and you will see quite a few options you probably never paid attention to.
The only thing an executatble "returns" to the batch script environment is the ERRORLEVEL. For SqlCmd you need the -b option to set this (based on the sql server error level)
If you use the -m option, you can control the error messages send to stdout -- I can't test at the moment, but I think this include the rows affected message (a level 0 error perhaps). You would then have to parse this too (ugly in batch scripts)
This sounds like a real kludge at best to be, you are likely better off to use a better scripting environment. PowerShell, Perl, Python, etc. all more powerful and you can find plenty of examples on-line.
Batch is best when you have a "no-deployment" requirement or you needs are simple. Easy to hit the wall as needs change.
In Oracle you can use &&VAR_NAME in a script and then the script will ask you for that value when you run it.
In SQLSERVER you can use $(VAR_NAME) and reference a property file using:
:r c:/TEMP/sqlserver.properties
And in the property file you have something like:
:setvar VAR_NAME_some_value
Can you do the equivalent of &&VAR_NAME so the script asks you for the value when you run it instead of having the value predefined in a script.
If I've understood correctly, you're talking about variable substitution with the SQLCMD utility.
I don't see that SQLCMD supports the behaviour you describe.
An alternative would be to exploit the fact that SQLCMD will substitute the values of system or user environment variables (see the link above), and create a wrapper CMD script which prompts the user for the variable value(s) using SET with the /P flag.
There is nothing in sql server like this, you should predefine all parameters values before using them, like this:
DECLARE #i SMALLINT
SET #i = 1
The problem with having a form pop up and ask you for the parameter is that you normally want rather more control over the nature of the form, even for an admin script. I'd use the variable substitution in SQLCMD, from within a Powershell or Python script so you can provide the guy running the script a better and more helpful form. That would make a very powerful combination.
You can do quite a lot with template variable substitution in SSMS, but that would only go so far as formulating the correct SQL to execute. you'd then have to bang the button. It would be a bit clunky outside the development environment!
How to get the relative path in t sql? Take for example a .sql file is located in the folder D:\temp, I want to get path of the file hello.txt in the folder D:\temp\App_Data. How to use the relative path reference?
Let's say I am executing the sql file inside the SQL server management studio.
I had a similiar problem, and solved it using sqlcmd variables in conjunction with the %CD% pseudo-variable. Took a bit of trial and error to combine all the pieces. But eventually got it all working. This example expects the script.sql file to be in the same directory as the runscript.bat.
runscript.bat
sqlcmd -S .\SQLINSTANCE -v FullScriptDir="%CD%" -i script.sql -b
script.sql
BULK INSERT [dbo].[ValuesFromCSV]
FROM '$(FullScriptDir)\values.csv'
with
(
fieldterminator = ',',
rowterminator = '\n'
)
go
The .sql file is just.... a file. It doesn't have any sense of its own location. It's the thing that excutes it (which you didn't specify) that would have a sense of its location, the file's location.
I notice that you mentioned an App_Data folder, so I guess that ASP.NET is involved. If you want to use relative paths in your web app, see MapPath
http://msdn.microsoft.com/en-us/library/system.web.httpserverutility.mappath.aspx
The server is executing the t-sql. It doesn't know where the client loaded the file from. You'll have to have the path embedded within the script.
DECLARE #RelDir varchar(1000)
SET #RelDir = 'D:\temp\'
...
Perhaps you can programmatically place the path into the SET command within the .sql script file, or perhaps you can use sqlcmd and pass the relative directory in as a variable.
When T-SQL is executing, it is running in a batch on the server, not on the client machine running Management Studio (or any other SQL client). The client just sends the text contents of the .sql file to the server to be executed. So, unless that file is located on the database server, I highly doubt you're going to be able to interact with it from a SQL script.
The t-sql script is first preprocessed by QueryAnalyzer, SSMS or sqlcmd on the client side. These programs are aware of the file localcation and could easily handle relative pathes similar To Oeacle sqlplus.
Obviously this is just a design decision from Microsoft and I dare say a rather stupid one.
I tried method from mateuscb's comments.
I found it can not work ,i do not know why,then I managed after several test.
It can work with the script below:
runscript.bat
#set FullScriptDir=%CD%
sqlcmd -S .\SQLINSTANCE -i script.sql
script.sql
BULK INSERT [dbo].[ValuesFromCSV]
FROM '$(FullScriptDir)\values.csv'
with
(
fieldterminator = ',',
rowterminator = '\n'
)
go
Just for your information for further discussion.
well it's not a Microsoft thing first off... it's an industry standard thing.
second your solution for running T-SQL with a relative path is to use a batch script or something to inject your path statement IE:
#echo OFF
SETLOCAL DisableDelayedExpansion
FOR /F "usebackq delims=" %%a in (`"findstr /n ^^ t-SQL.SQL"`) do (
set "var=%%a"
SETLOCAL EnableDelayedExpansion
set "var=!var:*:=!"
set RunLocation=%~dp0
echo(%~dp0!var! > newsql.sql
ENDLOCAL
)
sqlcmd newsql.sql
or something like that anyway