I read the Finatra getting started guide and I was able to write the HelloWorld Service and its feature test.
Currently my feature test looks like
server.httpPost(
path = "/hi",
postBody = """{"name": "Foo", "dob": 136190040000}""",
andExpect = Ok,
withBody = """{"msg":"Hello Foo. You are 15780 days old today"}""")
This works fine and my tests pass. However my requirement is that I extract the json returned by the server and then manually perform asserts on the object returned.
I changed my code to
val response = server.httpPost(
path = "/hi",
postBody = """{"name": "Abhishek", "dob": 136190040000}""",
andExpect = Ok,
withBody = """{"msg":"Hello Abhishek. You are 15780 days old today"}""")
val json = response.contentString
This also works and I can see the json returned in side the variable json.
My question is that if I have to deserialize this json into an object. Should I just pull in any json library like circe? and then deserialize the object?
or can I use the jackson framework which comes inside of Finatra.
In all examples I could find, I see that Finatra "automatically" handles the json serialization and deserialization. But in my case I want to perform this manually.
You can use the FinatraObjectMapper by calling (using your example) server.mapper. That wraps a Jackson ObjectMapper that you could use if you wanted to use the Jackson library without any of the Finatra add ons.
Or you can import your a different JSON library. If you are using SBT, you can restrict libraries to certain areas of your code, so if you wanted to use circe only in the test code, you could add the following to your build.sbt
"org.scalatest" %% "scalatest" % "2.2.6" % "test"
Related
I am posting the following object
{
skillName : "Professional Skills"
_id : {$oid: "5adf23946ab671bf6cb36aff"}
}
to the DjangoService given below:
#csrf_exempt
#api_view(['GET','POST'])
def saveSubjectView(request): #this service will add & update Subject
if request.method == 'POST':
try:
stream = StringIO(request.body)
subject = JSONParser().parse(stream)
print("The subejct is ")
pp.pprint(subject)
serializedsubject = json.loads(json_util.dumps(subject))
print("serializedsubject")
pp.pprint(serializedsubject)
The output that I am getting is
'skillType': { u'_id': { }, u'skillName': u'Professional Skills'}
The ObjectId posted from the front end (AngularJS) is not printed in the service. I know that I can fix it by removing the $oid while posting from the AngularJS application. But I would like to know why this is not happening. I have searched the documents and I couldn't get a proper reply. May be the keywords I used are wrong. Keywords used are : "JSON serialisation of ObjectId", "$oid json serialization using Django".
The complete object I am posting to the Django service is given below:
Exactly. $oid or anything prefixed with $ is an internal format and reserved, so you cannot post field names. The convention is from MongoDB Extended JSON where such prefixes are used to identify the BSON Type for proper conversion, and used as a serializable transport since these "types" are not supported in basic JSON.
So the solution is to actually use the bson.json_util to "deserialize" the JSON string right from the start:
from bson import json_util
# serializedsubject = json.loads(json_util.dumps(subject))
serializedsubject = json_util.loads(request.body) # correct usage
Or more succinctly self contained:
input = '{ "skillName" : "Professional Skills" ,"_id" : { "$oid": "5adf23946ab671bf6cb36aff"} }'
json_util.loads(input)
Returns
{u'skillName': u'Professional Skills', u'_id': ObjectId('5adf23946ab671bf6cb36aff')}
This correctly casts objects from any keys notated with the Extended JSON Syntax to their correct BSON Type, as also supported in the driver functions. And naturally the driver will then convert back to BSON when sending to MongoDB.
If for some reason your request.body contains anything other than a "string" which is valid for input to the function, then it is up to your code to convert it to that point. But there should be no need to "parse to JSON" and then "stringify" again just to input to the function.
NOTE: If you have not already done so within your JavaScript client side of the application, there is also the bson package available. This would allow where such Extended JSON is "received" from the server the translation into the BSON Types as JavaScript Objects, and of course then the serialization of such objects back into the Extended JSON Format.
This would in fact be recommended where "type" information needs to be maintained with the data transmitted and kept between client and server.
I am dealing with json objects containing geo coordinate points. I would like to run these points against a postgis server I have locally to assess point in polygon matching.
I'm hoping to do this with preexisting processors - I am successfully extracting the lat/lon coordinates into attributes with an "EvaluateJsonPath" processor, and successfully issuing queries to my local postgis datastore with "ExecuteSQL". This leaves me with avro responses, which I can then convert to JSON with the "ConvertAvroToJSON" processor.
I'm having conceptual trouble with how to merge the results of the query back together with the original JSON object. As it is, I've got two flow files with the same fragment ID, which I could theoretically merge together with "mergecontent", but that gets me:
{"my":"original json", "coordinates":[47.38, 179.22]}{"polygon_match":"a123"}
Are there any suggested strategies for merging the results of the SQL query into the original json structure, so my result would be something like this instead:
{"my":"original json", "coordinates":[47.38, 179.22], "polygon_match":"a123"}
I am running nifi 6.0, postgres 9.5.2, and postgis 2.2.1.
I saw some reference to using replaceText processor in https://community.hortonworks.com/questions/22090/issue-merging-content-in-nifi.html - but this seems to be merging content from an attribute into the body of the content. I'm missing the point of merging the content of the original and either the content of the SQL response, or attributes extracted from the SQL response without the content.
Edit:
Groovy script following appears to do what is needed. I am not a groovy coder, so any improvements are welcome.
import org.apache.commons.io.IOUtils
import java.nio.charset.*
import groovy.json.JsonSlurper
def flowFile = session.get();
if (flowFile == null) {
return;
}
def slurper = new JsonSlurper()
flowFile = session.write(flowFile,
{ inputStream, outputStream ->
def text = IOUtils.toString(inputStream, StandardCharsets.UTF_8)
def obj = slurper.parseText(text)
def originaljsontext = flowFile.getAttribute('original.json')
def originaljson = slurper.parseText(originaljsontext)
originaljson.put("point_polygon_info", obj)
outputStream.write(groovy.json.JsonOutput.toJson(originaljson).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, ExecuteScript.REL_SUCCESS)
If your original JSON is relatively small, a possible approach might be the following...
Use ExtractText before getting to ExecuteSQL to copy the original JSON into an attribute.
After ExecuteSQL, and after ConvertAvroToJSON, use an ExecuteScript processor to create a new JSON document that combines the original from the attribute with the results in the content.
I'm not exactly sure what needs to be done in the script, but I know others have had success using Groovy and JsonSlurper through the ExecuteScript processor.
http://groovy-lang.org/json.html
http://docs.groovy-lang.org/latest/html/gapi/groovy/json/JsonSlurper.html
In JMeter I want to check the number of objects in a JSON array, which I receive from the server.
For example, on a certain request I expect an array with 5 objects.
[{...},{...},{...},{...},{...}]
After reading this: count members with jsonpath?, I tried using the following JSON Path Assertion:
JSON Path: $
Expected value: hasSize(5)
Validate against expected value = checked
However, this doesn't seem to work properly. When I actually do receive 5 objects in the array, the response assertion says it doesn't match.
What am I doing wrong?
Or how else can I do this?
Although JSONPath Extractor doesn't provide hasSize function it still can be done.
Given the example JSON from the answer by PMD UBIK-INGENIERIE, you can get matches number on book array in at least 2 ways:
1. Easiest (but fragile) way - using Regular Expression Extractor.
As you can see, there are 4 entries for category like:
{ "category": "reference",
{ \"category\": \"fiction\"
...
If you add a Regular Expression Extractor configured as follows:
It'll capture all the category entries and return matches number as below:
So you will be able to use this ${matches_matchNr} variable wherever required.
This approach is straightforward and easy to implement but it's very vulnerable to any changes in the response format. If you expect that JSON data may change in the foreseeable future continue with the next option.
2. Harder (but more stable) way - calling JsonPath methods from Beanshell PostProcessor
JMeter has a Beanshell scripting extension mechanism which has access to all variables/properties in scope as well as to the underlying JMeter and 3rd-party dependencies APIs. In this case you can call JsonPath library (which is under the hood of JsonPath Extractor) directly from Beanshell PostProcessor.
import com.jayway.jsonpath.Criteria;
import com.jayway.jsonpath.Filter;
import com.jayway.jsonpath.JsonPath;
Object json = new String(data);
List categories = new ArrayList();
categories.add("fiction");
categories.add("reference");
Filter filter = Filter.filter(Criteria.where("category").in(categories));
List books = JsonPath.read(json, "$.store.book[?]", new Filter[] {filter});
vars.put("JSON_ARRAY_SIZE", String.valueOf(books.size()));
The code above evaluates JSONPath expression of $.store.book[?] against parent sampler response, counts matches number and stores it into ${JSON_ARRAY_SIZE} JMeter Variable
which can later be reused in an if clause or an assertion.
References:
JMeter – Working with JSON – Extract JSON response
JMeter's User Manual Regular Expressions entry
JSON Path Documentation and Examples
How to use BeanShell: JMeter's favorite built-in component
This is not possible with the plugin you are using (JMeter-plugins).
But it can be done with JSON Extractor since JMeter 3.0, this plugin has been donated by UbikLoadPack (http://jmeter.apache.org/changes_history.html)
Example:
Say you have this JSON that contains an array of books:
{ "store": {"book": [
{ "category": "reference","author": "Nigel Rees","title": "Sayings of the Century","price": 8.95},
{ "category": "fiction","author": "Evelyn Waugh","title": "Sword of Honour","price": 12.99},
{ "category": "fiction","author": "Herman Melville","title": "Moby Dick","isbn": "0-553-21311-3","price": 8.99},
{ "category": "fiction","author": "J. R. R. Tolkien","title": "The Lord of the Rings","isbn": "0-395-19395-8","price": 22.99}
],
"bicycle": {"color": "red","price": 19.95}} }
To have this count:
1/ Add JSON Extractor:
The count will be then available bookTitle_matchNr which you can access through:
${bookTitle_matchNr}
Running this Test Plan would display this:
As you can see, Debug Sampler-${bookTitle_matchNr} shows Debug Sampler-4
I'm trying to figure out how to populate a table from a JSON array. So far, I can populate my table cells perfectly fine by using the following code:
self.countries = [[NSArray alloc]initWithObjects:#"Argentina",#"China",#"Russia",nil];
Concerning the JSON, I can successfully retrieve one line of text at a time and display it in a label. My goal is to populate an entire table view from a JSON array. I tried using the following code, but it still won't populate my table. Obviously I'm doing something wrong, but I searched everywhere and still can't figure it out:
NSURL *url = [NSURL URLWithString:#"http://BlahBlahBlah.com/CountryList"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];
AFJSONRequestOperation *operation = [AFJSONRequestOperation JSONRequestOperationWithRequest:request success:^(NSURLRequest *request, NSHTTPURLResponse *response, id JSON)
{
NSLog(#"%#",[JSON objectForKey:#"COUNTRIES"]);
self.countries = [JSON objectForKey:#"COUNTRIES"];
}
failure:nil];
[operation start];
I am positive that the data is being retrieved, because the NSLog outputs the text perfectly fine. But when I try setting my array equal to the JSON array, nothing happens. I know the code is probably wrong, but I think I'm on the right track. Your help would be much appreciated.
EDIT:
This is the text in the JSON file I'm using:
{
"COUNTRIES": ["Argentina", "China", "Russia",]
}
-Miles
It seems that you need some basic JSON parsing. If you only target iOS 5.0 and above devices, then you should use NSJSONSerialization. If you need to support earlier iOS versions, then I really recommend the open source JSONKit framework.
Having recommended the above, I myself almost always use the Sensible TableView framework to fetch all data from my web service and automatically display it on a table view. Saves me a ton of manual labor and makes app maintenance a breeze, so it's probably something to consider too. Good luck!
I'm working on a data migration task, where I have to export a somewhat large Lotus Notes application into a blogging platform. My first task was to export the articles from Lotus Notes into CSV files.
I created a Agent in LotusScript to export the data into CSV files. I use a modified version of this IBM DeveloperWorks forum post. And it basically does the job. But the contents of the Rich Text field is stripped of any formatting. And this is not what I want, I want the Rich Text field rendered as HTML.
The documentation for the GetItemValue method explicitly states that the text is rendered into plain text. So I began to research for something that would retrieve the HTML. I found the NotesMIMEEntity class and some sample code in the IBM article How To Access HTML in a Rich Text Field Using LotusScript.
But for the technique described in the above article to work, the Rich Text field need to have the property "Store Contents as HTML and MIME". And this is not the case with my Lotus Notes database. I tried to set the property on the fields in question, but it didn't do the trick.
Is it possible to use the NotesMIMEEntity and set the "Store Contents as HTML and MIME" property after the content has been added, to export the field rendered as HTML?
Or what are my options for exporting the Notes database Rich Text fields as HTML?
Bonus information: I'm using IBM Lotus Domino Designer version 8.5
There is this fairly unknown command that does exactly what you want: retrieve the URL using the command OpenField.
Example that converts only the Body-field:
http://SERVER/your%5Fdatabase%5Fpath.nsf/NEW%5FVIEW/docid/Body?OpenField
Here is how I did it, using the OpenField command, see D.Bugger's post above
Function GetHtmlFromField(doc As NotesDocument, fieldname As String) As String
Dim obj
Set obj = CreateObject("Microsoft.XMLHTTP")
obj.open "GET", "http://www.mydomain.dk/database.nsf/0/" + doc.Universalid + "/" + fieldname + "?openfield&charset=utf-8", False, "", ""
obj.send("")
Dim html As String
html = Trim$(obj.responseText)
GetHtmlFromField = html
End Function
I'd suggest looking at Midas' Rich Text LSX (http://www.geniisoft.com/showcase.nsf/MidasLSX)
I haven't used the personally, but I remember them from years ago being the best option for working with Rich Text. I'd bet it saves you a lot of headaches.
As for the NotesMIMEEntity class, I don't believe there is a way to convert RichText to MIME, only MIME to RichText (or retain the MIME within the document for emailing purposes).
If you upgrade to Notes Domino 8.5.1 then you can use the new ConvertToMIME method of the NotesDocument class. See the docs. This should do what you want.
Alternativly the easiest way to get the Domino server to render the RichText will be to actually retrieve it via a url call. Set up a simple form that just has the RichText field and then use your favourite HTTP api to pull in the page. It should then be pretty straight forward to pull out the body.
Keep it simple.
Change the BODY field to Store contents as HTML and MIME
Open the doc in editmode.
Save.
Close.
You can now use the NotesMIMEEntity to get what you need from script.
You can use the NotesDXLExporter class to export the Rich Text and use an XSLT to transform the output to what you need.
I know you mentioned using LotusScript, but if you don't mind writing a small Java agent (in the Notes client), this can be done fairly easily - and there is no need to modify the existing form design.
The basic idea is to have your Java code open a particular document through a localhost http request (which is simple in Java) and to have your code capture that html output and save it back to that document. You basically allow the Domino rendering engine to do the heavy lifting.
You would want do this:
Create a form which contains only the rich-text field you want to convert, and with Content Type of HTML
Create a view with a selection formula for all of the documents you want to convert, and with a form formula which computes to the new form
Create the Java agent which just walks your view, and for each document gets its docid, opens a URL in the form http://SERVER/your_database_path.nsf/NEW_VIEW/docid?openDocument, grabs the http response and saves it.
I put up some sample code in a similar SO post here:
How to convert text and rich text fields in a document to html using lotusscript?
Works in Domino 10 (have not tested with 9)
HTMLStrings$ = NotesRichTextItem .Converttohtml([options] ) As String
See documentation :
https://help.hcltechsw.com/dom_designer/10.0.1/basic/H_CONVERTOHTML_METHOD_NOTESRICHTEXTITEM.html
UPDATE (2022)
HCL no longer support this method since version 11. The documentation does not include any info about the method.
I have made some tests and it still works in v12 but HCL recommended to not use it.
Casper's recommendation above works well, but make sure the ACL is such to allow Anonymous Access otherwise your HTML will be the HTML from your login form
If you do not need to get the Richtext from the items specifically, you can use ?OpenDocument, which is documented (at least) here: https://www.ibm.com/developerworks/lotus/library/ls-Domino_URL_cheat_sheet/
https://www.ibm.com/support/knowledgecenter/SSVRGU_9.0.1/com.ibm.designer.domino.main.doc/H_ABOUT_URL_COMMANDS_FOR_OPENING_DOCUMENTS_BY_KEY.html
OpenDocument also allows you to expand sections (I am unsure if OpenField does)
Syntax is:
http://Host/Database/View/DocumentUniversalID?OpenDocument
But be sure to include the charset parameter as well - Japanese documents were unreadable without specifying utf-8 as the charset.
Here is the method I use that takes a NotesDocument and returns the HTML for the doc as a string.
private string ConvertDocumentToHml(Domino.NotesDocument doc, string sectionList = null)
{
var server = doc.ParentDatabase.Server.Split('/')[0];
var dbPath = doc.ParentDatabase.FilePath;
string viewName = "0";
string documentId = doc.UniversalID.ToUpper();
var ub = new UriBuilder();
ub.Host = server;
ub.Path = dbPath.Replace("\\", "/") + "/" + viewName + "/" + documentId;
if (string.IsNullOrEmpty(sectionList))
{
ub.Query = "OpenDocument&charset=utf-8";
}
else
{
ub.Query = "OpenDocument&charset=utf-8&ExpandSection=" + sectionList;
}
var url = ub.ToString();
var req = HttpWebRequest.CreateHttp(url);
try
{
var resp = req.GetResponse();
string respText = null;
using (var sr = new StreamReader(resp.GetResponseStream()))
{
respText = sr.ReadToEnd();
}
return respText;
}
catch (WebException ex)
{
return "";
}
}