I have a SQL Server 2005 SSIS package with an Execute SQL task. I edit the SQL statement, go through the OKs to make the change. I go back in to the Execute SQL task- and the changes have taken hold. Great.
I now go to save the package. I click save. I look back in the Execute SQL task, and the changes have been lost.
What is going on here and how can I stop it?
Click the Execute SQL Task
Click F4 to see the list of properties
Click the Expressions line and then the little [...] box to see the expressions
Change the expressions (or remove them if you don't want your name/sqlstatementsource/etc. to be set dynamically).
This is a nice feature for reusability and template based development of SSIS packages. We use it allt he time.
I had the same issue. The reason was that I used a config file with all the executables included, and sql code is just a property of one of the executable which is stored in the config file. When you change your code the config file does NOT get updated, but when you close/open your project the values are getting pulled from the config file which reverse your change to the initial state!
In order to fix this you need either to exclude your executable from the config file (what I did) or re-create the config file every time you change the package.
I hope this helps.
I found I was unable to change the "To" in the Send Mail Task. Or rather, I would press OK on the new value and then bring it up again and it would be the old value. What I did -- was to edit the dtsx file in Notepad by searching for the old value and changing it.
Related
I am trying to automate the following steps:
We currently try to keep our database project (.sqlproj) up to date with out database, but this is currently a manual procedure.
We open Visual Studio, put our database as source, put our sqlproj as target, select the pieces we want (we only need the tables, views, indexes and stored procedures) and then run a comparison. When that is done, we hit the 'update' button and our sqlproj file gets updated with the files generated or updated. We check this in and done.
Now I'm trying to automate this process. I'm able to generate the source dacpac file with sqlpackage and I can run a comparison script with the dacpac generated using msbuild OR update a target database, but I can't find how to update the sqlproj like the Update button on the above screenshot does.
I've found DACExtensions to try to simulate the behaviour, but it seems the Update button works in its own specific way.
Long story short: what command do I need to run in order to simulate the 'Update' button?
Any and all help would be appreciated :)
My SSIS projects tend to run queries that require changes as they move between environments, like the table schema might change or a value in the Where clause. I've always either put my SQL into a Project Parameter, which is hard to edit since formatting is lost, or just put it directly into the Execute SQL Task/Data Flow Source then manually edited it between migrations which is also not ideal.
I was wonder though if I added my SQL scripts to files within the project, can these be read back in? Example if I put a query like this:
select id, name from %schema%.tablename
I'd like to read this into a variable then it's easy to use an expression as I do with Project Parameters to replace %schema% with the appropriate value. Then the .sql files within the project can be edited with little effort or even tested through an Execute SQL Task that's disabled/removed before the project goes into the deployment flow. But I've been unable to find how to read in a file using a relative path within the project. Also I'm not even sure these get deployed to the SSIS Server.
Thanks for any insight.
I've added a text file query.sql to an SSIS (SQL 2017) Project in Visual Studio, bit I've found no way to pull the contents of query.sql into a variable.
Native tooling approach
For an Execute SQL Task, there's an option to source your query directly from a file.
Set your SQLSourceType to File Connection and then specify a file connection manager in the FileConnection section.
Do be aware that while this is handy, it's also ripe for someone escalating their permissions. If I had access to the file the SSIS package is looking for, I can add a drop database, create a new user and give them SA rights, etc - anything the account that runs the SSIS package can do, a nefarious person could exploit.
Roll your own approach
If you're adamant about reading the file yourself, add two Variables to your SSIS package and supply values like the following
User::QueryPath -> String -> C:\path\to\file.sql
User::QueryActual -> String -> SELECT 1;
Add a Script Task to the package. Specify as a ReadOnly variable User::QueryPath and specify as a ReadWrite variable User::QueryActual
Within the Main you'd need code like the following
string filePath = this.Dts.Variables["User::QueryPath"].Value.ToString();
this.Dts.Variables["User::QueryActual"].Value = System.IO.File.ReadAllText(filePath);
The meat of the matter is System.IO.File.ReadAllText. Note that this doesn't handle checking whether the file exists, you have permission to access, etc. It's just a barebones read of a file (and also open to the same injection challenges as the above method - just this way you own maintaining it versus the fine engineers at Microsoft)
You can build your query by using both Variable and Parameter.
For example:
Parameter A: dbo
Build your variable A (string type) as : "Select * FROM server.DB." + ParameterA + ".Table"
So if you need to change the schema, just change the parameter A will give you the corresponding query in variable A.
I'm learning to develop ETL using Pentaho Spoon, I'm pretty noob yet.
Instead of storing SQL operations inside its file, I'd like to have them on their own .sql files. It makes easier to track changes on Subversion, and in case of need I can just open the sql file on DB manager and execute it directly.
How could I do that? I suppose I could use some component to read a txt file into a variable, and another component to take that variable and execute it on DB.
How's the simplest way to achieve that?
In the standard SQL Table input, you can define the query to be a parameter ${my_query} and this parameter has to be defined (without ${...} decoration) in the transformation properties: right-click anywhere, select Properties on the popup menu, the Parameter tab.
Each time you run the transformation, you'll presented the list of parameters, among which my_query which you can overwrite.
To automatize, follow the example which was shipped with the installation zip. In the same directory as you spoon.bat/spoon.sh, there is a folder named sample, in which you will find a job to read_all_files or read all_tables. Basically this job list the files in a directory, and for each one puts it in a variable and use it as a parameter to run the transformation. Much more easier to do than to explain.
I am using SSIS to insert data from flat file to database.
I have created Data Flow Task for that. I am using Flat File as Source and ADO NET Destination to insert data.
Below is how my setting looks like for Flat File Source.
Below is how my "Columns" tab look like
THIS WORKS FINE WHEN I RUN THAT USING BIDS AND DATA IS INSERTED PROPERLY INTO DATABASE. IT EVEN WORKS WITH DTEXEC.EXE WHEN RUN LOCALLY.
Now, The problem is with executing the package on the server using dtexec.exe. On the server, data is inserted properly but the text qualifier (double quotes) given is totally ignored while inserting data to database. THE SAME WORKS TOTALLY FINE WHEN RUN LOCALLY. I have attached image below for how its stored in database.
I have checked SQL SERVER version and SSIS version locally and on remote server and both are same.
What can be the problem? Can anyone help?
So I found a solution for this problem. Thanks to LukeBI answer here
Create a string variable called TextQualifier and assign the value " (double quotes)
Select the connection manager, and in the Properties window select 'Expressions'. See below.
Click ..., add the property 'TextQualifier' and assign the variable #[User::TextQualifier]. See below image
Now its working fine. It will even work on 64 bit OS now.
In the flat file source, click "Columns". Make sure that when you preview the data there are no quotes in the preview. Otherwise you may have to look back at your file and make sure that BOTH the text qualifier and delimiter are correct.
If this does not work, then please take a screenshot of the "Columns" screen as well and post it. A screenshot of the actual file layout would help as well. Hope this helps!
Within your flat file connection Manager, within the "Advanced" option, you should be given a view of different parameters for each field. You will see for each field it will have a Name, ColumnDelimiter, a bunch of faded out fields and the DataType and a choice if it is text qualified or not.
In there, you should specify that the column(s) which you wish to be determined as text qualified by setting the TextQualified option to true.
I'm trying to replace a scheduled job which runs a .dtsx file with the same .dtsx file, modified.
I want to use the same .dtsxonfig file as the old package, but I don't know where it is located. How can I find the location of the config file? How can I ensure the new package will reference it?
If the current sql agent job provides an explicit path for the configuration, the job step will list that (grab the details from the Command Line tab).
Otherwise, look inside the package. SSIS, Package Configurations... and look for a Configuration Type of XML configuration file
Update
In your control flow, right click on the background and in the context sensitive menu, select Package Configurations... If they have turned on configurations, I would expect the "Enable package configurations" check box to be checked and the section below that to not be greyed out.
With XML configuration, you get two options with it. The first (Sample) is what is referred to as direct configuration. You can see the Configuration String specifies a specific location on disk that the package will examine when the package runs. The second (XmlEnv) option demonstrates indirect configuration. This works by using an environment variable, SSIS_CONFIG in my example, to determine the location of the .dtsConfig file
If you don't see anything listed in the Package Configurations section, then you can assume they are applying configuration at run-time. There are a few different ways of accomplishing this so I would ask your DBA to provide the details of the particular Agent's Integration Services job step.