I have done a couple projects using the LoopEach construct. This the first where I'm generating an output file inside a Data Flow in the loop.
I have a script that extracts the path and basename of the input file. I have looked at that is extracted (using FileInfo) and it is correct.
A few steps later, I am outputing to a flat file. I have both the input and output connection managers set to DelayValidation. The output keeps failing. Although I have the output set to a variable that includes the input file name (so it will continue to change, the output fails and shows the filename without the info from the variable.
The variable is set to a scope of the package, so it should be covered.
What am I missing???
Related
I have a program flow where a database command button writes a file with data from the current record, then executes a batch file (windows) to knit a markdown file using the output from the database as inputs.
"C:\Program Files\R\R-4.1.1\bin\i386\Rscript.exe" -e "library('knitr'); rmarkdown::render('MyMarkdownFile.Rmd', output_file='MyOutput.html')"
The final step is that this file is opened in a browser.
start "" "MyOutput.html"
I do not use a unique file name, the same html file (MyOutput.html in the example above) is over-written each time. Sometimes the markdown process throws an error and halts execution during the knit. In these cases the previous version of the html file is then opened by the next batch command and, to the users, this may be confusing: they may assume they are seeing the current report when in fact they are not. (Note there are clear labels to distinguish, but still ...). I am wondering if there is a way to somehow "know" within the batch file that there has been an error in the knit process and thereby halt execution of the batch so that the html is not opened in the final step.
See Mofi's comment. This is the answer. Simply inserting && between batch commands halts execution when the knit returns an error. Thank you.
I have a dataflow that is used to do transformation of multiple flat files from given folder using for each loop container. I have a flat file again as output file. The problem is that every time I execute the the job only the last file that got transformed will be stored in destination file.
Is there a way in SSIS I can create individual transformed output file instead on overwriting on same one over and over again?
For. eg. I have 5 flat files ,test_1.txt,test_2.txt,test_3.txt ,test4_.txt
and test_5.txt in a folder.
After the job ran I can only see the data from last file test_5.txt being
transformed in my destination file.
Here's steps on a working example I tested.
Variables
I have 3 variables defined:
FileName - To be used in the foreach loop
DestinationDir - where are the files going
SourceDir - where are the files I want to process
Foreach Loop Setup
I have a foreach loop configured as:
Expression for "Directory" set to #[User::SourceDir]
Retrieve file name set to "Name and extension"
Then under the "Variable Mappings":
That means as the foreach loop is iterating over the files in the directory it will be setting the "Name and extension" of the file its on to the variable #[User:FileName]
Data Flow Task
The I add a Data Flow Task inside the foreach loop:
Then inside the DFT I have a simple Flat File Source to Flat File Destination. We'll just pass the contents of each file to new files:
During initial development I'll manually pick one file to walk through setting each of the source and destinations. Then come back and change the connection managers and set an expression on the ConnectionString.
Connection Manager Expressions
SourceFile Connection Manager:
ConnectionString gets an expression as: #[User::SourceDir] + #[User::FileName]
DestinationFile Connection Manager:
ConnectionString gets an expression as: #[User::DestinationDir] + #[User::FileName]
Testing
I have 2 test files in my source directory and no files in my destination:
After I execute my package I get success and also get new files in my destination:
There are ways to do what you are asking in SSIS with variables and expressions but there is an easier way to accomplish it using command line.
Since you are just consolidating a text files into 1 you can use a command prompt to better handle your issue:
copy *.txt output.txt
I've got a batch file which needs to perform operations on a specific file each day. So far, the file names have followed the pattern EX_2017-08-30.DAT which means I could use the following to get the exact filename for the day:
set today=%date:~-4,4%-%date:~-10,2%-%date:~-7,2%
set filename=EZ_%today%.DAT
Now I'm being told the filenames will change to include a timestamp, such as EX_2017-08-30-231859.DAT. However, the exact time won't be known beforehand (it gets set when a certain process completes).
I can't use a wildcard throughout the batch file because the filename is being written to an external file for another application to use, so I have to know the exact filename. Is there anyway that I can do a search with a wildcard and store the resulting complete filename into a variable?
If you can list the files in the directory your EX_* file is in, you can do:
for %%i in (EX_%today%-*.DAT) do (
set filename=%%i
)
The first line lists all files in the directory matching the date and the extension, and then it sets the last file to the filename variable. Be careful as this does not throw any warnings should there be more than one file matching the expression.
If you cannot list the directory, your only chance is bruteforce. There are only 24*60*60 possibilities of the filename, and if you go backwards in time, you should reach the desired file in just a couple of thousands of iterations, providing the task is usually completed close to midnight.
I need to extract the object name from a sql text file. All of my sql files have as their 1st line "CREATE some type [schema name].[object name]. Sometimes the brackets are there, other times not. In either case, I need to be able to discern the object name affected so I can determine if it actually exists before updating the server with the new changes. I need to do this from a Windows 7 command line batch file. Not powershell, please.
Doing this in a batch file is a bit like working without your hands tied behind your back, but if you insist, I would suggest the following:
Get the first line of the file (you said in the comments that you can already do this).
Split the line on spaces and get the x-th value.
Split the resulting value on the dot.
Strip the backets from the value.
Voila. It won't be easy, it won't be readable, but it will do what you need and it will be a Windows cmd batch file.
I am trying to make a batch file that will copy the contents of one file (with problem characters) that sends an email into another file (also vbs) so as to then run it and send a customizable email (customized by various batch things, my program needs to use batch) So I want to use this command:
type mailersample.vbs>> mailerfinal.vbs
BUT I want to edit certain things. Would I need to go through a variable (problem characters), or would I use the set command, in which case, under which format?