I tried to instantiate multiple routes from the same camel route template which resulted in a misbehaviour in the evaluation of several simple expressions.
I am using camel 3.20.1 in a spring boot application and i am having problems creating routes from the following route template. I am using constants for the specific template parameter keys - these are also used in several expressions (simple expression etc). At one point in the route template / instantiation of a route based on the following route template, especially at the aggregate / completionSize definition, an expression (templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE)) evaluates to a value specified for a second route which is using this template.
Route template:
#Override
public void configure() throws Exception {
routeTemplate("generic-data-file-based-template")
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX)
.from(templateParameterString(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE))
.filter(FILENAME_FILTER_PREDICATE)
.aggregate(templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX), new GroupedMessageAggregationStrategy())
.completionSize(templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE))
.log("correlation completed by ${header." + Exchange.AGGREGATED_COMPLETED_BY + "} with ${header." + Exchange.AGGREGATED_SIZE + "} files")
.setHeader(INTERNAL_HEADER_REFERENCE_DATE, headerSubstring2(header(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX), Exchange.FILE_NAME))
.to(templateParameterString(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI));
}
private Expression templateParameterExpression(String value) {
return simple("{{"+value+"}}");
}
Route I based on this template:
public void configure() throws Exception {
templatedRoute("generic-data-file-based-template")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI, "sftp:localhost:22/test/application/cashflows?username=tester&password=password")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI, "mock:cashflow-watch-mock")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE, "CASHFLOW")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX, "[0-9]{8}(Flow_tot|Head_tot|IntPeriod_tot){1}.csv")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX, "[0-9]{8}")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX, "[0-9]{8}")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE, "3")
.routeId("file-watch-1");
}
Route II based on this template:
public void configure() throws Exception {
templatedRoute("generic-data-file-based-template")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI, "sftp:localhost:22/test/application/bookvalues?username=tester&password=password")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI, "mock:bookvalue-watch-mock")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE, "BOOKVALUE")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX, "BW_BVA_[0-9]{8}.csv")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX, "BW_BVA_[0-9]{8}.csv")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX, "[0-9]{8}")
.parameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE, "1")
.routeId("file-watch-2");
}
It depends on the order in which these two routes are added to the camel context, to which value the expression templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE) evaluates for both routes. For example, if file-watch-1 was added first, then the expression in file-watch-2 will evaluate to the value 3 instead of 1.
I debugged my code and saw that camel uses an expressions cache which returns the upper value for the second route - but only for the expression used in the completionSize definition. Other expressions have the right value.
I already took a look on the official documentation which isn't exactly stating that this is common behaviour for the way i specified the templateParameter.
So, am i doing something wrong? Is this an error in the framework? Common behaviour? Should i use templateBeans instead?
Thanks in advance!
I investigated my problem further and found out that the two routes i am instantiating are using the same instance of AggregationDefinition which evaluates the simple expression in completionSize always to the same value once set (expressions cache !?).
I fixed this behaviour using another signature of the method completionSize in the template itself (completionSize(String completionSize)) and used the property method to specify the size:
#Override
public void configure() throws Exception {
routeTemplate("generic-data-file-based-template")
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX)
.templateParameter(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX)
.from(property(RouteTemplateConstants.TEMPLATE_PARAMETER_FROM_URI))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILENAME_FILTER_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_GENERIC_DATA_TYPE))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_REFERENCE_DATE_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX))
.setHeader(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE, templateParameterExpression(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE))
.filter(FILENAME_FILTER_PREDICATE)
.aggregate(simple(property(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX)), new GroupedMessageAggregationStrategy())
.completionSize(property(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_COMPLETION_SIZE))
.log("correlation completed by ${header." + Exchange.AGGREGATED_COMPLETED_BY + "} with ${header." + Exchange.AGGREGATED_SIZE + "} files")
.setHeader(INTERNAL_HEADER_REFERENCE_DATE, headerSubstring2(header(RouteTemplateConstants.TEMPLATE_PARAMETER_FILE_CORRELATION_REGEX), Exchange.FILE_NAME))
.to(property(RouteTemplateConstants.TEMPLATE_PARAMETER_TO_URI));
}
Now my code works as expected.
Related
I'm trying to add search options on my search form that would allow the user to ensure that all the words he searched for are in the results, or at least one, or an "exact match".
I've found MinimumMatch and it's perfect for that.
I've made a custom viewhelper that I placed in my Result.html, it takes as parameter the type of search the user wants (atLeastOne, AllWords, etc...) and I've dug a bit in the source code and it seems I can override values of the current Solr Typoscript by passing an array to a mergeSolrConfiguration function.
So I tried something like that as a draft to see how it works :
public static function renderStatic(array $arguments, Closure $renderChildrenClosure, RenderingContextInterface $renderingContext)
{
$previousRequest = static::getUsedSearchRequestFromRenderingContext($renderingContext);
$usedQuery = $previousRequest->getRawUserQuery();
$contextConfiguration = $previousRequest->getContextTypoScriptConfiguration();
$override['plugin.']['tx_solr.']['search.']['query.']['minimumMatch'] = '100%';
$contextConfiguration->mergeSolrConfiguration($override, true, true);
return self::getSearchUriBuilder($renderingContext)->getNewSearchUri($previousRequest, $usedQuery);
}
But it simply doesn't work. Solr keeps using the site's Typoscript config instead of my overriden one.
I saw there was a way to also override the Typoscript of the filters, with setSearchQueryFilterConfiguration so I gave it a try with :
public static function renderStatic(array $arguments, Closure $renderChildrenClosure, RenderingContextInterface $renderingContext)
{
$previousRequest = static::getUsedSearchRequestFromRenderingContext($renderingContext);
$usedQuery = $previousRequest->getRawUserQuery();
$previousRequest->getContextTypoScriptConfiguration()
->setSearchQueryFilterConfiguration(['sourceId_intS' => 'sourceId_intS:7']);
return self::getSearchUriBuilder($renderingContext)->getNewSearchUri($previousRequest, $usedQuery);
}
But nope, it keeps using the configuration set in the website's template, completely ignoring my overrides.
Am I going the wrong way with this ? Is the thing I'm trying to do not possible technically ?
I am trying to build a Get request as follows and I would like CaseReference value to be populated via feeder .feed(CaseProviderSeq) but for some reason it's not picking CaseReference value and printing following for my println statement in .sign statement bellow
PATH KJ: /caseworkers/554355/jurisdictions/EMPLOYMENT/case-types/Manchester_Multiples/cases/$%7BCaseReference%7D/event-triggers/updateBulkAction_v2/token
My feeder CSV got following rows currently
1574761472170530
1574622770056940
so I am expecting this amended URL would be like
/caseworkers/554355/jurisdictions/EMPLOYMENT/case-types/Manchester_Multiples/cases/1574761472170530/event-triggers/updateBulkAction_v2/token
any idea what wrong I am doing here ??
.get(session => SaveEventUrl.replace(":case_reference","${CaseReference}").replaceAll("events", "") + s"event-triggers/${EventId}/token")
.header("ServiceAuthorization", s2sToken)
.header("Authorization", userToken)
.header("Content-Type","application/json")
.sign(new SignatureCalculator {
override def sign(request: Request): Unit = {
val path = request.getUri.getPath
println("PATH KJ: " + path)
request.getHeaders.add("uri", path)
}
})
This is not related to .sign, but your session attribute CaseReference not being interpreted. If you look closely you can see the braces %-encoded in $%7BCaseReference%7D.
Interpretation of the Gatling Expression Language strings happens only when a String is present when an Expression[Something] is needed1.
This bug you wrote is shown exactly in the warning in the documentation above.
I believe you can simply remove session => in your .get, so you are passing in a String rather than a Session => String2. That string will be implicitly converted to Expression[String]. That way Gatling will put the session attribute into the URL.
This happens because of the Scala implicit conversion.
In fact it is Session => Validation[String], because, again, of implicit conversions.
I have implemented MapFunction for my Apache Flink flow. It is parsing incoming elements and convert them to other format but sometimes error can appear (i.e. incoming data is not valid).
I see two possible ways how to handle it:
Ignore invalid elements but seems like I can't ignore errors because for any incoming element I must provide outgoing element.
Split incoming elements to valid and invalid but seems like I should use other function for this.
So, I have two questions:
How to handle errors correctly in my MapFunction?
How to implement such transformation functions correctly?
You could use a FlatMapFunction instead of a MapFunction. This would allow you to only emit an element if it is valid. The following shows an example implementation:
input.flatMap(new FlatMapFunction<String, Long>() {
#Override
public void flatMap(String input, Collector<Long> collector) throws Exception {
try {
Long value = Long.parseLong(input);
collector.collect(value);
} catch (NumberFormatException e) {
// ignore invalid data
}
}
});
This is to build on #Till Rohrmann's idea above. Adding this as an answer instead of a comment for better formatting.
I think one way to implement "split + select" could be to use a ProcessFunction with a SideOutput. My graph would look something like this:
Source --> ValidateProcessFunction ---good data--> UDF--->SinkToOutput
\
\---bad data----->SinkToErrorChannel
Would this work? Is there a better way?
Tried to find a solution, but I got always stuck a the docs or at answers include other bundles. In the documentation of the dynamic router you can find the hint:
"Of course you can also have several parameters, as with normal Symfony routes. The semantics and rules for patterns, defaults and requirements are exactly the same as in core routes."
Thats it.
...
/foo/{id}/bar
I tried (seems not) everything to get it done.
Same for all tries:
I tried it to apply a variable pattern and a child route.
use Symfony\Cmf\Bundle\RoutingBundle\Doctrine\Phpcr\Route as PhpcrRoute;
$dm = $this->get('cmf_routing.route_provider');
$route = new PhpcrRoute();
$route->setPosition( $dm->find( null, '/cms/routes' ), 'foo' );
$route->setVariablePattern('/{id}');
$dm->persist( $route );
$child = new PhpcrRoute();
$child->setPosition( $route, 'bar' );
$dm->persist( $child );
$dm->flush();
With or without default value and requirement only '/foo/bar' and '/foo/*' return matches, but '/foo/1/bar' prompts me with a 'No route found for "GET /foo/1/bar"'.
...
Just now I nearly got it done.
use Symfony\Cmf\Bundle\RoutingBundle\Doctrine\Phpcr\Route as PhpcrRoute;
$dm = $this->get('cmf_routing.route_provider');
$route = new PhpcrRoute();
$route->setPosition( $dm->find( null, '/cms/routes' ), 'example_route' );
$dm->persist( $route );
$route->setPrefix( '/cms/routes/example_route' );
$route->setPath( '/foo/{id}/bar' );
$dm->flush();
If prefix is '/cms/routes' and name is 'foo' everything works fine. But now that I got this far, assigning a speaking name would round it up.
Thanks in advice!
You got quite close to the solution, actually!
When using PHPCR-ODM, the route document id is its path in the repository. PHPCR stores all content in a tree, so every document needs to be in a specific place in the tree. We then use the prefix to get a URL to match. If the prefix is configured as /cms/routes and the request is for /foo, the router looks in /cms/routes/foo. To allow parameters, you can use setVariablePattern as you correctly assumed. For the use case of /foo/{id}/bar, you need to do setVariablePattern('/{id}/bar'). You could also have setVariablePattern('/{context}/{id}') (this is what the doc paragraph you quoted meant - i will look into adding an example there as its indeed not helpful to say "you can do this" but not explain how to).
Calling setPath is not recommended as its just less explicit - but as you noticed, it would get the job done. See the phpdoc and implementation of Model\Route::setPattern:
/**
* It is recommended to use setVariablePattern to just set the part after
* the static part. If you use this method, it will ensure that the
* static part is not changed and only change the variable part.
*
* When using PHPCR-ODM, make sure to persist the route before calling this
* to have the id field initialized.
*/
public function setPath($pattern)
{
$len = strlen($this->getStaticPrefix());
if (strncmp($this->getStaticPrefix(), $pattern, $len)) {
throw new \InvalidArgumentException('You can not set a pattern for the route that does not start with its current static prefix. First update the static prefix or directly use setVariablePattern.');
}
return $this->setVariablePattern(substr($pattern, $len));
}
About explicit names: The repository path is also the name of the route, in the example /cms/routes/foo. But it is not a good idea to use a route name of a dynamic route in your code, as those routes are supposed to be editable (and deletable) by an admin. If you have a route that exists for sure and is at a specific path, use the configured symfony routes (the routing.yml file). If its dynamic routes, have a look at the CMF Resource Bundle. It allows to define a role for a document and a way to look up documents by role. If you have a route with a specific role that you want to link to from your controller / template, this is the way to go. If you have a content document that is linked with a route document and have that content document available, your third and best option is to generate the URL from the content document. The CMF dynamic router can do that, just use the content object where you normally specify the route name.
I have the following route DSL:
from("file:" + autoLoadBaseDir + "?move=.classified")
.loop(fileTypes.length)
.choice()
.when(header("CamelFileName").contains(fileTypes[Integer.valueOf("${CamelLoopIndex}")]))
.to("file:" + classesBaseDir + "/" + fileTypes[Integer.valueOf("${CamelLoopIndex}")]);
As shown, I wish to access the CamelLoopIndex and use it as an index in an array. The expression is not evaluated, hence the route is not created. What am I doing wrong? Thanks in advance.
Documentation on this is pretty scanty, and I've not been successful getting a solution after hours of searching.
UPDATE: I've posted the same question to the Camel Users Mailing List.
You should use ${property.CamelLoopIndex}
you can use a processor or directly access the property
${exchangeProperty.CamelLoopIndex}
...
.loop(4)
.log("${exchangeProperty.CamelLoopIndex}")
.process(exchange -> {
Integer index = (Integer) exchange.getProperty(Exchange.LOOP_INDEX);
})
.end()
...
https://camel.apache.org/components/latest/eips/loop-eip.html
https://camel.apache.org/manual/latest/processor.html
Hope this helps
I'm late to the party but maybe somebody will get help from this answer. In the example given above this works for me for accessing the loop index in the Java DSL.
property(Exchange.LOOP_INDEX)
So for the example in the first post from okello above I guess this will work
Integer.valueOf(property(Exchange.LOOP_INDEX).toString())
Having experimented around with a number of options, the following works for me:
from("file:" + autoLoadBaseDir + "?preMove=inprogress&move=.classified")
.routeId("Test-Route")
.loop(fileTypes.length)
.processRef("keFileTypeNameService")
.choice()
.when(header("CamelFileName").contains(header("MyFileType")))
.to("file:" + classesBaseDir + "/?autoCreate=true&fileName=${header[MyFileType]}/${header[CamelFileName]}");
The keFileTypeNameService retrieves the CamelLoopIndex property from the exchange. It then uses this to get the file type at that index. It then just simply set this file type name in the header. The keFileTypeNameService bean is shown below:
#Service( value = "keFileTypeNameService" )
public class FileTypeNameService implements Processor {
private #Value("${ke.file.types}") String[] fileTypes;
public void process(Exchange exchange) throws Exception {
Integer count = exchange.getProperty("CamelLoopIndex", Integer.class);
String fileType = fileTypes[count];
exchange.getIn().setHeader("MyFileType", fileType);
}
}
I hope this assists someone else.