I am using Apache Camel, Mina2 and HAPI to receive HL7 v2 messages. I noticed that its taking lot of time to unmarshal and create Message object. And this time increases when I have larger message.
My message has around 120 OBX segment and I am using OBX 3 and OBX 5 field only. I tested manually removing fields after OBX 5 and found some improvement in performance. Is there any way to tell HAPI not to parse any fields after OBX 5 ?
You could extend ca.uhn.hl7v2.parser.PipeParser and override the Segment parsing method.
#Override
public void parse(Segment destination, String segment, EncodingCharacters encodingChars, Integer theRepetition) throws HL7Exception {
if(!"OBX".equals(destination.getName()) || destination.getParent().getParent().getAll("OBSERVATION").length <= 5) {
super.parse(destination, segment, encodingChars, theRepetition);
}
}
Use this to parse your messages and it will only parse the first 5 OBSERVATIONS in the ORDER_DETAIL.
Related
http://camel.apache.org/splitter.html [1]
From [1] link i saw CamelSplitSize will be on the completed Exchange.
I am learning camel and i would like to know is there a way possible to split the xml file containing 100 rows (assuming 100 rows)
If the split failed while processing the 50th row and we need to show CamelSplitIndex as 50, CamelSplitSize as 100 and CamelSplitComplete as false
.bean(Splitter.class,"saveFile("${camelContext.properties[mySplitSize]}, ${camelContext.properties[mySplitIndex]}, ${camelContext.properties[mySplitComplete]})")
I could not find a way to accomplish this as the link [1] clearly states CamelSplitSize will be stored only on the completed Exchang. Any way to achieve this ??
If you need this properties you can catch exception that causes stopping splitter and get exchange which caused the exception. There you will find your properties.
public void show(Exchange exchange) {
CamelExchangeException camelExceptionCaught = (CamelExchangeException) exchange.getProperty("CamelExceptionCaught");
System.out.println(camelExceptionCaught.getExchange().getProperty("CamelSplitSize"));
System.out.println(camelExceptionCaught.getExchange().getProperty("CamelSplitComplete"));
System.out.println(camelExceptionCaught.getExchange().getProperty("CamelSplitIndex"));
}
I am just getting started on Apache Flink (Scala API), my issue is following:
I am trying to stream data from Kafka into Apache Flink based on one example from the Flink site:
val stream =
env.addSource(new FlinkKafkaConsumer09("testing", new SimpleStringSchema() , properties))
Everything works correctly, the stream.print() statement displays the following on the screen:
2018-05-16 10:22:44 AM|1|11|-71.16|40.27
I would like to use a case class in order to load the data, I've tried using
flatMap(p=>p.split("|"))
but it's only splitting the data one character at a time.
Basically the expected results is to be able to populate 5 fields of the case class as follows
field(0)=2018-05-16 10:22:44 AM
field(1)=1
field(2)=11
field(3)=-71.16
field(4)=40.27
but it's now doing:
field(0) = 2
field(1) = 0
field(3) = 1
field(4) = 8
etc...
Any advice would be greatly appreciated.
Thank you in advance
Frank
The problem is the usage of String.split. If you call it with a String, then the method expects it to be a regular expression. Thus, p.split("\\|") would be the correct regular expression for your input data. Alternatively, you can also call the split variant where you specify the separating character p.split('|'). Both solutions should give you the desired result.
I have this very weird error, "System.TypeException: Cannot have more than 10 chunks in a single operation", has anyone seen/encountered this before ? Please can you guide me if you know how to solve this.
I am trying to insert different types of sObjects together in a list of sObject. The list is never larger than 10 rows.
This post here:
https://developer.salesforce.com/forums/ForumsMain?id=906F000000090nUIAQ
suggests that it is not the number of different sObjects, but the order of the objects that causes this chunk limit to be exceeded. In other words, "1,1,1,2,2,2" has one chunk, the transition from "1" to "2". "1,2,3,4,5,6" has six chunks, even though the number of elements is the same. Putting the objects into the list sorted in object order is the suggested solution.
Is it possible for you to create a reasonable test case with only 2 or 3 rows?
There are two possible explanations for this issue:
As Jagular noted, you did not order the sobjects you tried to insert, so there are more than 10 'chunks' in the list.
You try to insert > 2000 records, and > 1 sobject type. This one seems like a Salesforce Bug, since the error message doesn't match the issue.
Scenario 1 and its solution
When you have a hybrid list, make sure that the objects are not scattered without any order. For example, A,B,A,B,A,B,A,B…. Salesforce has an inherent trouble in switching sObject types for more than 10 times. They call this switching limit as Chunking Limit. So, on this hybrid list, if you would have sorted it and then passed it for DML, Salesforce would have been much happier. For example. A,A,A,A,B,B,B,B… In this case, salesforce only has to switch one time (that is read all A objects –>switch –> read all B objects). The max chunk limit is 10. So, here we are safe.
listToUpdate.sort();
UPDATE listToUpdate;
Scenario 2 and its solution
Another point that we have to bear in our mind is that if the hybrid list contains more number of objects for one type, we can run into TypeException. As mentioned in the screenshot, if list contains 1001 objects of type A and 1001 objects of type B, then total objects is equal to 2002. The maximum chunks allowed is 10. So, if you do a simple math, the number of objects in each chunk would be 2002/10 = 200. Salesforce also enforces another governor limit that each chunk should not contain 200 or more than 200 objects. In this case, we will have to foresee how much objects are possible to enter this code and we have to write code to pass lists of safe size for DML every time.
Scenario 3 and its solution
Scenario 3 and its solution
Third scenario that can happen is if the hybrid list contains objects of more than 10 types, then even if the size of the list if very small, switching happens when salesforce reads different sObject. So, we have to make sure that in this case, we allot separate lists for each sObject type and then pass it on for DML. Doing this in an apex trigger or apex class would cause you some trouble as multiple DML’s are initiated in a context. Passing this kind of multiple sObject lists for DML operations in a different context would really ease up the load you pump into the platform. Consider doing this kind of logic in a Batch Apex Job rather than a apex trigger or apex class.
Hope this helps.
Below a code that should cover all 3 Scenario from Arpit Sethi.
It's a piece of code I took from this topic: https://developer.salesforce.com/forums/?id=906F000000090nUIAQ.
and modified to cover Scenario 2.
private static void saveSobjectSet(List <Sobject> listToUpdate) {
Integer SFDC_CHUNK_LIMIT = 10;
// Developed this part due to System.TypeException: Cannot have more than 10 chunks in a single operation
Map<String, List<Sobject>> sortedMapPerObjectType = new Map<String, List<Sobject>>();
Map<String, Integer> numberOf200ChunkPerObject = new Map<String, Integer>();
for (Sobject obj : listToUpdate) {
String objTypeREAL = String.valueOf(obj.getSObjectType());
if (! numberOf200ChunkPerObject.containsKey(objTypeREAL)){
numberOf200ChunkPerObject.put(objTypeREAL, 1);
}
// Number of 200 chunk for a given Object
Integer numnberOf200Record = numberOf200ChunkPerObject.get(objTypeREAL);
// Object type + number of 200 records chunk
String objTypeCURRENT = String.valueOf(obj.getSObjectType()) + String.valueOf(numnberOf200Record);
// CurrentList
List<sObject> currentList = sortedMapPerObjectType.get(objTypeCURRENT);
if (currentList == null || currentList.size() > 199) {
if(currentList != null && currentList.size() > 199){
numberOf200ChunkPerObject.put(objTypeREAL, numnberOf200Record + 1);
objTypeCURRENT = String.valueOf(obj.getSObjectType()) + String.valueOf(numnberOf200Record);
}
sortedMapPerObjectType.put(objTypeCURRENT, new List<Sobject>());
}
sortedMapPerObjectType.get(objTypeCURRENT).add(obj);
}
while(sortedMapPerObjectType.size() > 0) {
// Create a new list, which can contain a max of chunking limit, and sorted, so we don't get any errors
List<Sobject> safeListForChunking = new List<Sobject>();
List<String> keyListSobjectType = new List<String>(sortedMapPerObjectType.keySet());
for (Integer i = 0;i<SFDC_CHUNK_LIMIT && !sortedMapPerObjectType.isEmpty();i++) {
List<Sobject> listSobjectOfOneType = sortedMapPerObjectType.remove(keyListSobjectType.remove(0));
safeListForChunking.addAll(listSobjectOfOneType);
}
update safeListForChunking;
}
}
Hope it helps,
Bye
Hi i kind of deviced a simple way to sort a list of different sobject types
public List<Sobject> SortRecordsByType(List<Sobject> records){
List<Sobject> response = new List<Sobject>();
Map<string,List<Sobject>> sortDictionary = new Map<string,List<Sobject>>();
for(Sobject record : records){
string objectTypeName = record.getSobjectType().getDescribe().getName();
if(sortDictionary.containsKey(objectTypeName)){
sortDictionary.get(objectTypeName).add(record);
}else{
sortDictionary.put(objectTypeName , new List<Sobject>{record});
}
}
// arrange in order
for(string objectName : sortDictionary.keySet()){
response.addAll(sortDictionary.get(objectName));
}
return response;
}
hopefully this solves your problem .
I am trying to create my first NHibernate project, so it is possible I am doing something stupid here but have been Googling for a couple of days and not had any joy yet.
I have an Article object which has various properties:
public class Article {
public virtual string Title { get; set; }
public virtual string Body { get; set; }
}
I am using the fluent configuration to load the mappings:
configuration.Mappings(m => m.AutoMappings.Add((AutoMap.AssemblyOf<Article>())
.Conventions.Add(DefaultCascade.All())
.UseOverridesFromAssemblyOf<SchemaConfigurationController>())
The override is:
public class ArticleOverrideMapping : IAutoMappingOverride<Article>
{
public void Override(AutoMapping<Article> mapping)
{
//mapping.Map(x => x.Body).CustomSqlType("NVARCHAR(4000)");
//mapping.Map(article => article.Body).Length(10000);
//mapping.Map(article => article.Body).Length(Int32.MaxValue);
//mapping.Map(article => article.Body).CustomSqlType("NVARCHAR(max)");
//mapping.Map(article => article.Body).CustomSqlType("NVARCHAR(max)").Length(Int32.MaxValue);
mapping.Map(article => article.Body).CustomType("StringClob").CustomSqlType("NVARCHAR(max)");
}
}
I have tried each of the commented out lines (roughly in order of when I found a possible solution online). I can get SQL Server to create the nvarchar(max) column and if I use SQL management studio I can paste a LOT (185,602 words was largest test) into the Body column. My issue is trying to get it to save from the MVC site using NHibernate.
There are two main errors I get:
String or binary data would be truncated. The statement has been terminated.
This would occur if I didn't set the ".Length(Int32.MaxValue)" override.
The second Error that occurs (when the length is set):
The length of the string value exceeds the length configured in the mapping/parameter.
I am pretty confused as to what I should be doing at this point. My goal is to be able to store a very large string (an whole article) in SQL Server (and SQLite for testing, nvarchar(max) wasn't liked by SQLite) and get that back out, (and edit it) in an MVC site.
UPDATE
As per #Cymen's link I tried
.CustomSqlType("nvarchar(max)").Length(Int32.MaxValue).Nullable();
but this lead to the error:
The length of the string value exceeds the length configured in the mapping/parameter.
when I only tried to save 1201 words (all the word "test"). When I added the length on the end of the above mapping ".Length(Int32.MaxValue)" I still the same error.
update
wanted to confirm which versions I am using:
FluentNHibernate.1.3.0.733
NHibernate.3.3.1.4000
Microsoft.AspNet.Mvc.4.0.20710.0
final update
Kieren had it correct, I had completely forgotten that I took that property and ran markdownsharp on it on the server and populated a second property on the server. So it was the second property that I hadn't mapped that was actually blowing up, sorry.
this is how i normally handle it.
mapping.Map(x => x.Problem).CustomType("varchar(MAX)");
not sure what CustomSqlType is, but i've never used it, and this works.
Ok, folks. Same stuff, so i've did some tests.
It seems that int.MaxValue cannot be used as LENGTH for fluent. If you do that, the resulting create SQL will be nvarchar(255). So if nhibernate (fluent) still generates such a script, then he is awaiting 255 chars at max.
If you hovewer use int.MaxValue / 2, then everything is ok. At least in the script.
Only reasonable explanation: unicode string, so for a single char fluent automatically takes 2 bytes. SO internally fluent hibernate may do a multiplication by 2. And if we get above 2GB space, who knows what fluent will do...
Please let me know if it works with data too.
I'm experiencing a really weird situation when passing on a POJO java object within the payload of a Pull Queue task using Gson. Without changing the code or the POJO being set within the payload of a task, this will randomly succeed or fail.
This is the code I'm using:
PullQueueTaskPayLoad tqp = new PullQueueTaskPayLoad("id","name");
tqp.uploadURL = taskPayLoad.uploadURL;
tqp.urls = taskPayLoad.urls;
tqp.sliceQueryParameter = taskPayLoad.sliceQueryParameter;
TaskOptions task = TaskOptions.Builder.withMethod(TaskOptions.Method.PULL);
task.payload(new Gson().toJson(tqp));
q.add(task);
Using an external queue consumer I then retrieve the POJO as follows:
Type GSON_TYPE = new TypeToken<PullQueueTaskPayLoad>() {}.getType();
byte[] b = new Base64().decodeBase64(leasedTask.getPayloadBase64().getBytes());
String payload = new String(b);
logger.info("About to convert payload: "+payload);
PullQueueTaskPayLoad taskpayload = new Gson().fromJson(payload, GSON_TYPE);
So from the debugging I did, the problem seems to be happening when I'm decoding the payload bytes. While encoding the same POJO (with different Ids) I randomly get 2 different decoded payload Strings as follows:
Correct decoding:
{"id":"1786024566","sliceQueryParameter": {"queryId":786024566,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?"],"http://www.slicepedia.org/ontology#hasroot":["?"]}}},"uploadURL":"http://3.linguabox0412.appspot.com/_ah/upload/AMmfu6YRjxX23Ks-yh-9AZs4-3I1p6hxrFd6d4ptxSQegUkQHN7y4hNZwX6u7PufIHJbwtsHLXFZJ5P-vs90mslZEOMw0T-amN2qhEOAj_6YdwuY50FXMi8/ALBNUaYAAAAAT7Towgs4M00M5RLI8xnEOMdIxouZzuGu/","status":"IN_PROGRESS","action":"SLICE_SEARCH_AND_CREATE"}
Incorrect decoding:
{"id":"1-1968382407","sliceQueryParameter":{"queryId":-1968382407,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?K??????˜?X?\YXK?ܙ?????H?\ܛ????Ȃ%?????'W??EU$?#?&?GG???2?Ɩ?wV&??C"?7?B?6??????W??B???gSe????'u?U'd?D??6?S??4UV?D?e7?%U?&%F%f?D?$???$&vu6?fF$????EG?v??6?6դvt?D???G??&D?fdֵ6%?甦??GD????F???$?V?CuF?$?F?F֤֧f?D??u?wt?4?C$?W?"?'7FGW2#?$???$?u$U52"?&7F???#?%4Ĕ4U?4T$4???E?5$TDR'
So the second string obviously fails when using Gson to convert it back to a POJO. But I dont' understand why this happens in only some cases and not others. For what I've seen, it seems to always happen after a ["?"] character string. I tried replacing and ? with other strings but it didn't change anything.
I think what is happening here is that the payload is webSafe-base64 encoded. In practice, this means swapping + and / and = for - and _ and .. Most base64 libraries have native support for decoding websafe base 64 strings.
Probably you are meeting one of these chars at a certain point, and that kills the decoding.
Here is some info on WebSafeBase64
Word of warning, though: the taskqueue implementation is actually sending padding equals (=) that you will have to convert manually before parsing.