I have a scenario where a Logic App will be scheduled to run at 11 am everyday and will move file from one SFTP to another SFTP, which I have done.
I want to add a condition if the first time the file is not present on SFTP then it should check again after 5 min for 3 retry attempts.
Thanks in advance.
I want to add a condition if the first time the file is not present on SFTP then it should check again
In order to check if the file is present or not, you can use List files in folder action.
Then you can check for file's existance by looping through files inside that folder using DisplayName Variable.
I want to add a condition if the first time the file is not present on SFTP then it should check again after 5 min for 3 retry attempts.
Whereas For the above requirement, when you want to retry you can use until Action of Control Connector.
Then set its count to 3 from Change Limits. Below is the flow in my Logic app
In the next step you can use Delay action and set its limits to 5 minutes.
So that for every 5 minutes the flow is going to check whether the file is present or not in sftp for 3 times.
Below is the whole flow that satisfies your requirement.
Related
I am currently using JMeter 5.5 and I have the below data set stored in a .csv file
BatchId
DocID_Start
DocID_End
1
1
93
2
94
1266
...
...
...
7
25313
45700
Each BatchID has 'n' number of Document ID ranging between DocID_Start and DocID_End.
I am trying to create a sampler that sends a POST Drop loop Call to the API ${url}/drop?documentIds=${DocID}
where DocID is any number between the DocID_Start and DocID_End.
I Created a the following testplan
GET Batch is working as expected and it is looping through each of the BatchID within the CSV file one by one but I could not get POST Drop Loop to go through the CSV and grab a different DocID_Start value. It will just keep reusing the first line for every user.
I've tried using Loop Counter and ticked off different checkboxes but i couldn't get it to work
With the default Sharing Mode of All threads CSV Data Set Config reads the next line from the CSV file on each iteration of each virtual user.
If you want to get a random number between DocID_Start and DocID_End - go for __Random() function:
${__Random(${DocID_Start},${DocID_End},)}
you can call this function as many times as you want under the Loop Controller.
If I misunderstood your question and you want to proceed to the next "Batch" under the Loop Controller - you need to move your CSV Data Set Config inside the Loop Controller as currently it proceeds to the next line on the new iteration of the Thread Group. JMeter Scoping Rules documentation chapter can shed some light on this.
I need to iterate a request multiple times in JMeter for different users. The iteration number differs for each users. For example: I need to iterate a request 5 times for first user, 3 times for second user and only 1 time for 3rd user and so on. How can we do this and how the input can be fed to the same. Is it possible to say read first 5 line for first user and next 3 lines for second user in csv data set config.
You can define the desired loops count for each user in the CSV file or via JMeter Properties
You can read the desired line from the CSV file using __groovy() function like:
${__groovy(new File('test.csv').readLines().get(0),)} - read 1st line
${__groovy(new File('test.csv').readLines().get(1),)} - read 2nd line
etc.
Here is a solution for your problem. There could be many other solutions available in JMeter.
You can define the loop count associated with the users (threads) in a (User Parameters)enter link description here Pre-processor.
This panel allows you to specify a series of values for any User Variable. For each thread, the variable will be assigned one of the values from the series in sequence. If there are more threads than values, the values get re-used.
Then the requests can be placed within a Loop Controller
The User Parameters component shall be placed within a JSR223 Sampler to ensure the values are initiated before the loop controller is reached. Also the JSR223 Sampler shall be placed within a Once Only Controller to ensure the Sampler is executed only once.
Add SampleResult.setIgnore() in the JSR223 Sampler to avoid the sampler getting recorded into the test results.
Test plan structure
CSV Data Set Config more suitable for large numbers of parameters
I have a list of time stamped S3 objects
eg: s3://01-02-20:10:00:00 , s3://01-02-20:10:00:01
and so on.
I want to consume all files which are within the last 5 minutes from S3 into flink as a DataSource and have checkpointing work as expected.
Can we do this with current File based source / Is this even possible ?
There is no source function available for your requirement, you need to implement a RichSourceFunction yourself and filter out the file path you need
I am working on my first SSIS package. I have a view with data that looks something like:
Loc Data
1 asd
1 qwe
2 zxc
3 jkl
And I need all of the rows to go to different files based on the Loc value. So all of the data rows where Loc = 1 should end up in the file named Loc1.txt, and the same for each other Loc.
It seems like this can be accomplished with a conditional split to flat file, but that would require a destination for each Location. I have a lot of Locations, and they all will be handled the same way other than being split in to different files.
Is there a built in way to do this without creating a bunch of destination components? Or can I at least use the script component to act as a way?
You should be able to set an expression using a variable. Define your path up to the directory and then set the variable equal to that column.
You'll need an Execute SQL task to return a Single Row result set, and loop that in a container for every row in your original result set.
I don't have access at the moment to post screenshots, but this link should help outline the steps.
So when your package runs the expression will look like:
'C:\Documents\MyPath\location' + #User::LocationColumn + '.txt'
It should end up feeding your directory with files according to location.
Set the User::LocationColumn equal to the Location Column in your result set. Write your result set to group by Location, so all your records write to a single file per Location.
I spent some time try to complete this task using the method #Phoenix suggest, but stumbled upon this video along the way.
I ended up going with the method shown in the video. I was hoping I wouldn't have to separate it in to multiple select statements for each location and an extra one to grab the distinct locations, but I thought the SSIS implementation in the video was much cleaner than the alternative.
Change the connection manager's connection string, in which you have to use variable which should be changed.
By varying the variable, destination file also changes
and connection string is :
'C:\Documents\ABC\Files\' + #User::data + '.txt'
vote this if it helps you
Looked through the docs for a way to use Camel for ETL just as in the site's examples, except with these additional conditionals based on an md5 match.
Like the camel example, myetl/myinputdir would be monitored for any new file, and if found, file of ${filename} would be processed.
Except it would first wait for ${filename}.md5 to show up, which would contain the correct md5. If ${filename}.md5 never showed up, it would simply ignore the file until it did.
And if ${filename}.md5 did show up but the md5 didn't match, it would be processed but with an error condition.
Found suggestions to use crypto for matching, but have not figured out how to ignore the file until the matching .md5 file shows up. Really, these two files need to be processed as a matched pair for everything to work properly, and they may not arrive in the input directory at the exact same millisecond. Or alternately, the md5 file might show up a few milliseconds before the data file.
You could use an aggregator to combine the two files based on their file name. If your files are suitably named, then you can use the file name (without extension) as the correlation ID. Continue the route once completionSize equals 2. If you set groupExchanges to true then in your next route step you have access to both the file to compute the hash value for and the contents of the md5 file to compare the hash value against. Or if the md5 or content file never arrived within completionTimeout you can trigger whatever action is appropriate for your scenario.