In Azure Data Factory v2 (ADFv2) I am having trouble passing a parameter whose value is an expression that needs evaluated at runtime.
For example, I have a set of extracts I want to download daily from the same LinkedService/Connection. I want a pipeline with a Foreach to be able to input a JSON pipeline parameter with a list of configuration for each report type (this I can do). "but" when I have one of those configuration KVPairs with value that is an expression, the expression does not seem to be evaluated.
here is an example of a Foreach parameter set that works for an SFTP LinkedService :
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_works_fine_20180307*.txt" } ]
here is an example of a Foreach parameter set that does not match the files I need to get.
(assume utcnow('yyyyMMdd') returns 20180307
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_does_NOT_work_#{utcnow('yyyyMMdd')}*.txt" } ]
This assumes that in the underlying Copy activity I am passing the dataset parameter fileFilter as
#item().fileFilter
...and in the dataset, the value of the fileFilter is an expression with value
#dataset().fileFilter
...I have also tried to wrap the argument completely as:
[ { "dirPath" : "/dirPath" ,"fileFilter" : "#toLower(concat(string('this_does_NOT_work_'),string(utcnow('yyyyMMdd')),string('*.txt') )))" } ]
...If you have suggestions/guidance, please let me know.
Thanks,
J
Try to put the fileFilter parameter directly in pipeline parameter.
Something like this will work:
[ { "dirPath" : "/dirPath" ,"fileFilter" : "this_works_fine_#{formatDateTime(utcnow(), 'yyyy')}#{formatDateTime(utcnow(), 'MM')}#{formatDateTime(utcnow(), 'dd')}*.txt" } ]
Related
I have a json payload that looks like this
{
"data":{
"methods":[
[
{
"p_id":"01",
"description":"Test01",
"offline":true
}
],
[
{
"p_id":"02",
"description":"Test02",
"offline":false
}
],
[
{
"p_id":"03",
"description":"Test03",
"offline":true
}
]
]
}
}
How can I write a JSONPath expression to get the "p_id" where "offline"= false?
You can use a filter expression which selects all elements in an object or array that match the specified filter. For example, [?(#.offline === false)] will match any object if its offline property is strictly false.
So, if the object is always in the same place, you could do:
$.data.methods.*[?(#.offline === false)].p_id
Or, if you want to look for any object where offline is false and fetch p_id, you could use a recursive descent with the filter expression:
$..[?(#.offline === false)].p_id
Note: I used strict equality in my examples so it will only match with a boolean false. If you don't need/want that you could instead simply use a ! to negate the filter. E.g. [?(!#.offline)]
I am trying to convert the below Array in to the JSON, trying by iterating through the Array and transforming. The array looks like below
Quote_Sent_To__r = [
{
Quote__c=0Q02D05XGQSA2,
Id=a1H2D0m94QUAQ,
type=QuoteSentTo__c
},
{
Quote__c=0Q02D00XGQSA2,
Id=a1H2D00000AQ,
type=QuoteSentTo__c
}
]
I have stored the array in to the variable quoteSentToList and iterating through the for loop
Within each iteration I need to get the JSON like
{
"Quote__c": "0Q02D05XGQSA2"
}
So this can be passed to the Salesforce Update operation. I tried like
%dw 2.0
output application/json
var item = vars.quoteSentToList[counter]
---
{
"Quote__c" :payload.Id
}
It errors saying
Reason: Unable to resolve reference of: counter..
Scripting language error on expression 'payload'. Reason: Unable to resolve reference of: payload..
This is my first project and any help is greatly appreciated
Error
""Unexpected character 'v' at quoteSentToList#[1:1] (line:column), expected false or true or null or {...} or [...] or number but was , while reading quoteSentToList as Json.
1| vars.existingQuote[0].Quote_Sent_To__r ^" evaluating expression: "%dw 2.0 output application/json
---
vars.quoteSentToList map { Quote__c: payload.Id, Id: $.Id }"."
counter is a Mule variable, not a DataWeave variable. You need to use the prefix vars. to reference it inside DataWeave scripts: vars.counter.
Alternatively, instead of using a <foreach> scope, you can transform the entire array at once and then use each element as needed:
%dw 2.0
output application/json
---
vars.quoteSentToList map { Quote__c: $.Id }
Output:
[
{
"Quote__c": "a1H2D0m94QUAQ"
},
{
"Quote__c": "a1H2D00000AQ"
}
]
Consider the following MongoDB collection of a few thousand Objects:
{
_id: ObjectId("xxx")
FM_ID: "123"
Meter_Readings: Array
0: Object
Date: 2011-10-07
Begin_Read: true
Reading: 652
1: Object
Date: 2018-10-01
Begin_Reading: true
Reading: 851
}
The wrong key was entered for 2018 into the array and needs to be renamed to "Begin_Read". I have a list using another aggregate of all the objects that have the incorrect key. The objects within the array don't have an _id value, so are hard to select. I was thinking I could iterate through the collection and find the array index of the errored Readings and using the _id of the object to perform the $rename on the key.
I am trying to get the index of the array, but cannot seem to select it correctly. The following aggregate is what I have:
[
{
'$match': {
'_id': ObjectId('xxx')
}
}, {
'$project': {
'index': {
'$indexOfArray': [
'$Meter_Readings', {
'$eq': [
'$Meter_Readings.Begin_Reading', True
]
}
]
}
}
}
]
Its result is always -1 which I think means my expression must be wrong as the expected result would be 1.
I'm using Python for this script (can use javascript as well), if there is a better way to do this (maybe a filter?), I'm open to alternatives, just what I've come up with.
I fixed this myself. I was close with the aggregate but needed to look at a different field for some reason that one did not work:
{
'$project': {
'index': {
'$indexOfArray': [
'$Meter_Readings.Water_Year', 2018
]
}
}
}
What I did learn was the to find an object within an array you can just reference it in the array identifier in the $indexOfArray method. I hope that might help someone else.
I'm using elasticsearch (v2.0.0) for search in Rails and want to add to our custom script for scoring, but I'm either messing up the syntax or just missing something else entirely. It all works without the check in the script for the array, so that's the only part that's not working.
So for the index, recipe_user_ids is an array of integers:
indexes :recipe_user_ids, type: 'integer'
Then in the search query I specify the parameter for the script file and which script file:
functions: [{
script_score: {
params: { current_user_id: user.id },
script_file: 'ownership_script'
}
}]
And the ownership-script.groovy file:
if (current_user_id == doc['user_id'].value) { owner_modifier = 1.0 } else { owner_modifier = 0.0 }
if (doc['recipe_user_ids'].values.contains(current_user_id)) { recipe_user_modifier = 50.0 } else { recipe_user_modifier = 0.0 }
(_score + (doc['score_for_sort'].value + owner_modifier + recipe_user_modifier)*5)/_score
I'm not getting any errors, but the results don't seem to match what I'd expect when the recipe_user_ids array does contain current_user_id, so everything is falling into the else statement. Is it a type issue, syntax? Any tips greatly appreciated.
This seems to occur due to mismatch in type caused by autoboxing.
The doc['field_name].values for field mapping short, integer, long types seems to be returning a collection always of type 'Long' and the argument to contains is autoboxed to Integercausing contains to fail.
You could probably explictly cast current_user_id to the type of Long:
Example:
doc['recipe_user_ids'].values.contains(new Long(current_user_id))
Or better to use the 'find' method
doc['recipe_user_ids'].values.find {it == current_user_id}
I have a list of checkboxes. I am trying to pass the list of selected checkboxes to a perl script. I am obtaining the list of checkboxes using the folliwng code :
function exec(){
var checkedValue = "";
var inputElements = document.getElementsByTagName('input');
for(var i=0; inputElements[i]; i++){
if(inputElements[i].className==="chk" &&
inputElements[i].checked){
checkedValue += inputElements[i].value;
if (inputElements[i+1])
checkedValue += ", ";
else
checkedValue += "";
}
}
I am then passing "checkedValue" to a perl script as follows :
self.location='/cgi-bin/ATMRunJob.pl?tcs='+checkedValue;
In the perl script, I read the array as follows :
our #testCasesToRun = $var->param("tcs");
This is then assigned to a hash as follows :
my $runSpec = {
TestCasesToRun => #testCasesToRun
};
However, I get the following error when I load the page in the browser :
Failed TestLimits() with error: [hash: k=TestCasesToRun, v=1,]:[array]:Value is not an array ref
In check against following TLS:
[
'hr',
{
'OptDefaults' => {
'JobRunningGroupName' => 'astbluetooth',
'RunMode' => 'Queue',
'CountTowardsReporting' => 1,
'JobOwnerGroupName' => 'astbluetooth',
'SelectSetupTeardown' => 1
},
'Optional' => {
'TestCasesToRun' => [
'ar',
undef,
undef,
[
'r',
1,
undef
]
],
I am new to perl as well as CGI scripting. How could I get around this error?
NOTE : All the code snippets have been shortened for brevity, but still portray the essence of the problem.
EDIT : What I want to do is this. The user selects a list of test cases from a checkboxed list that he wants to execute. I take the test case ids of all the selected test cases and pass it to a perl script. In the perl script, I just need to assign these selected testcase ids to the TestCasesToRun element in the runspec hash.
What would be the correct way to do that?
You are assigning an array as a hashkey value. That doesn't work; you need to assign an array ref:
my $runSpec = {
TestCasesToRun => \#testCasesToRun
};
Given that the code compiles, I have a feeling you just messed up your examples in the Q - please fix them to accurately reflect your code, even if they will be slightly less brief.
Your 'tcs' parameter is a single string (assigned via JS). Why are you then assigning results of param('tcs') to an array in the first place? Do you have a split somewhere in your code that you didn't include into the example?
Your dump contains an array reference within an array reference. You need to elaborate on what the expected structure of TestCasesToRun arrayref is, and show the code which processes it in the test runner.
As per your last comment:
Change your JavaScript code to join using simple comma: checkedValue += ",";
Change your Perl assignment to: our #testCasesToRun = split(/,/, $var->param("tcs"));