Now I'm trying to convert Java List object to JSON array, and struggling to convert UTF-8 strings. I've tried all followings, but none of them works.
Settings.
response.setContentType("application/json");
PrintWriter out = response.getWriter();
ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();
final ObjectMapper mapper = new ObjectMapper();
Test#1.
// Using writeValueAsString
String json = ow.writeValueAsString(list2);
Test#2.
// Using Bytes
final byte[] data = mapper.writeValueAsBytes(list2);
String json = new String(data, "UTF-8");
Test#3.
// Using ByteArrayOutputStream with new String()
final OutputStream os = new ByteArrayOutputStream();
mapper.writeValue(os, list2);
final byte[] data = ((ByteArrayOutputStream) os).toByteArray();
String json = new String(data, "UTF-8");
Test#4.
// Using ByteArrayOutputStream
final OutputStream os = new ByteArrayOutputStream();
mapper.writeValue(os, list2);
String json = ((ByteArrayOutputStream) os).toString("UTF-8");
Test#5.
// Using writeValueAsString
String json = mapper.writeValueAsString(list2);
Test#6.
// Using writeValue
mapper.writeValue(out, list2);
Like I said, none of above works. All displays characters like "???". I appreciate your helps. I'm using Servlet to send JSON response to clients.
This problem only happens when I write java.util.List object. If I write single data object, e.g. customer object in below example, then there is no ??? characters, and UTF-8 is working with the following code.
PrintWriter out = response.getWriter();
ObjectWriter ow = new ObjectMapper().writer().withDefaultPrettyPrinter();
String json = ow.writeValueAsString(customer);
out.print(json);
The answer was very simple. You need to specify UTF-8 charset encoding in response.setContentType too.
response.setContentType("application/json;charset=UTF-8");
Then, many of above code will work correctly. I will leave my question as is, since it will show you several ways of writing JSON to clients.
On RequestMapping in Controller:
#RequestMapping(value = "/user/get/sth",
method = RequestMethod.GET,
produces = { "application/json;**charset=UTF-8**" })
Related
I have the following table :
CREATE TABLE DI_Simulation
(
[city] nvarchar(255),
[profession] nvarchar(255)
);
I load the data from an URL with a Script task where I created a class Simulation and added two string attributes. I then deserialize the downloaded JSON data and create output rows.
I specify that the output columns city and profession are of type DT_WSTR but the following characters [é,à,è,...] are always replaced...
I tried different collations on both columns but no changes were seen. I also tried forcing UTF8 conversion on the Script Task but that also didn't work.
Any suggestions ?
EDIT: I should also mention that I have other tables where the insertion is made correctly, but this one especially has this issue, which I'm thinking the Script Task has something to do with it.
ServicePointManager.Expect100Continue = true;
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12 | SecurityProtocolType.Ssl3;
// Convert json string to .net object using the old school JavaScriptSerializer class
string Uri = "https://....";
JavaScriptSerializer serialize = new JavaScriptSerializer
{
MaxJsonLength = Int32.MaxValue,
};
var simulation = serialize.Deserialize<Simulation[]>(DownloadJson(Uri));
EDIT 2:
WebClient client = new WebClient();
Stream stream = client.OpenRead(Url);
StreamReader streamreader = new StreamReader(stream, System.Text.Encoding.GetEncoding(1252));
var ags = streamreader.ReadToEnd();
/*System.IO.File.WriteAllText(#"C:\Users\hhamdani\Desktop\Data Integration Objetcs\simulation_data.json",
ags,
System.Text.Encoding.GetEncoding(1252));*/
var simulation = serialize.Deserialize<Simulation[]>(ags);
Instead of downloading with DownloadJson, I used streamreader to get the Json Data from the URL and forced the Encoding, when I save the data on a txt file it's good, but on the Database it's the same issue.
Works fine from a script source component based on my reproduction
Setup
Table creation
A trivial table with two columns
CREATE TABLE dbo.[SO_71842511] (
[TestCase] int,
[SomeText] nvarchar(50)
)
SCR Do Unicode
Proof that we can inject unicode characters into the data flow task from a script source.
Define the Script Task as a Source. Add 2 columns to the output, one int, one DT_WSTR
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
using Microsoft.SqlServer.Dts.Runtime.Wrapper;
[Microsoft.SqlServer.Dts.Pipeline.SSISScriptComponentEntryPointAttribute]
public class ScriptMain : UserComponent
{
public override void CreateNewOutputRows()
{
Output0Buffer.AddRow();
Output0Buffer.SomeText = "e e plain";
Output0Buffer.TestCase = 0;
Output0Buffer.AddRow();
Output0Buffer.SomeText = "é e forward";
Output0Buffer.TestCase = 1;
Output0Buffer.AddRow();
Output0Buffer.SomeText = "à a back";
Output0Buffer.TestCase = 2;
Output0Buffer.AddRow();
Output0Buffer.SomeText = "è e backward";
Output0Buffer.TestCase = 3;
}
}
Results
WebClient client = new WebClient();
Stream stream = client.OpenRead(Url);
StreamReader streamreader = new StreamReader(stream, System.Text.Encoding.UTF8);
var ags = streamreader.ReadToEnd();
This did the job for me.
Thanks #billinkc
In my Spring Integration pipeline I am getting a XML payload and depending on the value of the attributes in the XML I have to generate a key and publish it to kafka.
return IntegrationFlows.from(Kafka.messageDrivenChannelAdapter(kafkaListenerContainer))
.wireTap(ACARS_WIRE_TAP_CHNL) // Log the raw message
.enrichHeaders(h ->h.headerFunction(KafkaHeaders.MESSAGE_KEY, m -> {
StringBuilder header = new StringBuilder();
Expression expression = new SpelExpressionParser().parseExpression("payload.Body.toString()");
//Expression expression = new SpelExpressionParser().parseExpression("m.payload.Body.ACIFlight.fltNbr.toString()");
String flightNbr = expression.getValue(String.class);
header.append(flightNbr);
return header.toString();
}))
.get();
XMl is
<?xml version="1.0" encoding="UTF-8"?>
<ns0:Envelope xmlns:ns0="http://www.exmaple.com/FlightLeg">
<ns0:Header>
<ns1:eventHeader xmlns:ns1="http://www.exmaple.com/header" eventID="659" eventName="FlightLegEvent" version="1.0.0">
<ns1:eventSubType>FlightLeg</ns1:eventSubType>
</ns1:eventHeader>
</ns0:Header>
<ns0:Body>
<ns1:ACIFlight xmlns:ns1="http://ual.com/cep/aero/ACIFlight">
<flightKey>1267:07042020:UA</flightKey>
<fltNbr>1267</fltNbr>
<fltLastLegDepDt>07042020</fltLastLegDepDt>
<carrCd>UA</carrCd>
</ns1:ACIFlight>
</ns0:Body>
</ns0:Envelope>
I am trying to get the fltNbr from this xml payload using spel. Please suggest
Updated
String flight = XPathUtils.evaluate(message.getPayload(), "/*[local-name() = 'fltNbr']",XPathUtils.STRING);
String DepDate = XPathUtils.evaluate(message.getPayload(), "/*[local-name() = 'fltLastLegDepDt']",XPathUtils.STRING);
return MessageBuilder.fromMessage(message).setHeader("key", flight+DepDate).build();
You can use the XPath Header Enricher.
XPath is also available as a Spel function, but you'd be better off using the enricher in this case.
public class XPathHeaderEnricher extends HeaderEnricher {
Here's a test case...
#Test
public void convertedEvaluation() {
Map<String, XPathExpressionEvaluatingHeaderValueMessageProcessor> expressionMap =
new HashMap<String, XPathExpressionEvaluatingHeaderValueMessageProcessor>();
XPathExpressionEvaluatingHeaderValueMessageProcessor processor = new XPathExpressionEvaluatingHeaderValueMessageProcessor(
"/root/elementOne");
processor.setHeaderType(TimeZone.class);
expressionMap.put("one", processor);
String docAsString = "<root><elementOne>America/New_York</elementOne></root>";
XPathHeaderEnricher enricher = new XPathHeaderEnricher(expressionMap);
Message<?> result = enricher.transform(MessageBuilder.withPayload(docAsString).build());
MessageHeaders headers = result.getHeaders();
assertThat(headers.get("one")).as("Wrong value for element one expression")
.isEqualTo(TimeZone.getTimeZone("America/New_York"));
}
I am trying to use resteasy. While I am able to do send a mixed multipart as a request to a webservice, I am unable to do get a mixed multipart in the response.
For eg: Requesting for a file (byte[] or stream) and the file name in a single Response.
Following is what I have tested:
Service code:
#Path("/myfiles")
public class MyMultiPartWebService {
#POST
#Path("/filedetail")
#Consumes("multipart/form-data")
#Produces("multipart/mixed")
public MultipartOutput fileDetail(MultipartFormDataInput input) throws IOException {
MultipartOutput multipartOutput = new MultipartOutput();
//some logic based on input to locate a file(s)
File myFile = new File("samplefile.pdf");
multipartOutput.addPart("fileName:"+ myFile.getName(), MediaType.TEXT_PLAIN_TYPE);
multipartOutput.addPart(file, MediaType.APPLICATION_OCTET_STREAM_TYPE);
return multipartOutput;
}
}
Client code:
public void getFileDetails(/*input params*/){
HttpClient client = new DefaultHttpClient();
HttpPost postRequest = new HttpPost("urlString");
MultipartEntity multiPartEntity = new MultipartEntity();
//prepare the request details
postRequest.setEntity(multiPartEntity);
HttpResponse response = client.execute(postRequest);
HttpEntity returnEntity = response.getEntity();
//extracting data from the response
Header header = returnEntity.getContentType();
InputStream is = returnEntity.getContent();
if (is != null) {
byte[] bytes = IOUtils.toByteArray(is);
//Can we see the 2 parts that were added?
//Able to get a single InputStream only, and hence unable to differentiate two objects in the response
//Trying to see the contents - printing as string
System.out.println("Output from Response :: " + new String(bytes));
}
}
The output is as follows - able to see 2 different objects with different content types, but unable to extract them separately.
Output from Response ::
--af481055-4e4f-4860-9c0b-bb636d86d639
Content-Type: text/plain
fileName: samplefile.pdf
--af481055-4e4f-4860-9c0b-bb636d86d639
Content-Length: 1928
Content-Type: application/octet-stream
%PDF-1.4
<<pdf content printed as junk chars>>
How can I extract the 2 objects from the response?
UPDATE:
Tried the following approach to extract the different parts - use the 'boundary' to break the MultipartStream; use the content type string to extract approp object.
private void getResponeObject(HttpResponse response) throws IllegalStateException, IOException {
HttpEntity returnEntity = response.getEntity();
Header header = returnEntity.getContentType();
String boundary = header.getValue();
boundary = boundary.substring("multipart/mixed; boundary=".length(), boundary.length());
System.out.println("Boundary" + boundary); // --af481055-4e4f-4860-9c0b-bb636d86d639
InputStream is = returnEntity.getContent();
splitter(is, boundary);
}
//extract subsets from the input stream based on content type
private void splitter(InputStream is, String boundary) throws IOException {
ByteArrayOutputStream boas = null;
FileOutputStream fos = null;
MultipartStream multipartStream = new MultipartStream(is, boundary.getBytes());
boolean nextPart = multipartStream.skipPreamble();
System.out.println("NEXT PART :: " + nextPart);
while (nextPart) {
String header = multipartStream.readHeaders();
if (header.contains("Content-Type: "+MediaType.APPLICATION_OCTET_STREAM_TYPE)) {
fos = new FileOutputStream(new File("myfilename.pdf"));
multipartStream.readBodyData(fos);
} else if (header.contains("Content-Type: "+MediaType.TEXT_PLAIN_TYPE)) {
boas = new ByteArrayOutputStream();
multipartStream.readBodyData(boas);
String newString = new String( boas.toByteArray());
} else if (header.contains("Content-Type: "+ MediaType.APPLICATION_JSON_TYPE)) {
//extract string and create JSONObject from it
} else if (header.contains("Content-Type: "+MediaType.APPLICATION_XML_TYPE)) {
//extract string and create XML object from it
}
nextPart = multipartStream.readBoundary();
}
}
Is this the right approach?
UPDATE 2:
The logic above seems to work. But got another block, when receiving the RESPONSE from the webservice. I could not find any references to handle such issues in the Response.
The logic assumes that there is ONE part for a part type. If there are, say, 2 JSON parts in the response, it would be difficult to identify which part is what. In other words, though we can add the part with a key name while creating the response, we are unable to extract the key names int he client side.
Any clues?
You can try the following approach...
At the server side...
Create a wrapper object that can encapsulate all types. For eg., it could have a Map for TEXT and another Map for Binary data.
Convert the TEXT content to bytes (octet stream).
Create a MetaData which contains references to the Key names and their type. Eg., STR_MYKEY1, BYTES_MYKEY2. This metadata can also be converted into octet stream.
Add the metadata and the wrapped entity as parts to the multipart response.
At the Client side...
Read the MetaData to get the key names.
Use the key name to interpret each part. Since the Keyname from the metadata tells if the original data is a TEXT or BINARY, you should be able to extract the actual content with appropriate logic.
The same approach can be used for upstream, from client to service.
On top of this, you can compress the TEXT data which will help in reducing the content size...
I'm using the bing api to request some results.. when I run my code the response string is truncated so that its missing the first 10-50 characters.. when I paste the exact same request in the browser it returns the results just fine..
Here is my code.. what am I doing wrong?
string AppId = "My APP ID HERE :O ";
string url = "http://api.search.live.net/xml.aspx?Appid={0}&sources={1}&query={2}";
string completeUri = String.Format(url, AppId, "web", validateforweb(Artist) + "%20" + validateforweb(Song) + "%20" + "Lyrics");
HttpWebRequest webRequest = null;
webRequest = (HttpWebRequest)WebRequest.Create(completeUri);
HttpWebResponse webResponse = null;
webResponse = (HttpWebResponse)webRequest.GetResponse();
XmlReader xmlReader = null;
Stream s = webResponse.GetResponseStream();
xmlReader = XmlReader.Create(s);
StreamReader reader;
reader = new StreamReader(s);
string str = reader.ReadToEnd();
I suspect it's related to the fact you're creating 2 readers on the stream (XmlReader and StreamReader). The XmlReader starts buffering data from the stream as soon as you create it, so when the StreamReader starts reading from the same stream, it misses the part of the data that has been buffered by the XmlReader.
You can't use 2 readers on the same stream, they will conflict with each other.
i have a person object and need to store it as byte[] and again retrieve that byte[] and convert to person object
and BinaryFormatter is not availabe in silverlight
Because the namespaces mentioned by t0mm13b are not part of the Silverlight .NET engine, the correct way to is to use this workaround leveraging the data contract serializer:
http://forums.silverlight.net/forums/t/23161.aspx
From the link:
string SerializeWithDCS(object obj)
{
if (obj == null) throw new ArgumentNullException("obj");
DataContractSerializer dcs = new DataContractSerializer(obj.GetType());
MemoryStream ms = new MemoryStream();
dcs.WriteObject(ms, obj);
return Encoding.UTF8.GetString(ms.GetBuffer(), 0, (int)ms.Position);
}
If you really need binary and want it to be super fast and very small, then you should use protobuf from Google.
http://code.google.com/p/protobuf-net/
Look at these performance numbers. Protobuf is far and away the fastest and smallest.
I've used it for WCF <--> Silverlight with success and would not hesitate to use it again for a new project.
I have used XML Serializer to convert the object to a string and them convert the string to byte[] successfully in Silverlight.
object address = new Address();
System.Xml.Serialization.XmlSerializer serializer = new System.Xml.Serialization.XmlSerializer(typeof(Address));
StringBuilder stringBuilder = new StringBuilder();
using (StringWriter writer = new StringWriter(stringBuilder))
{
serializer.Serialize(writer, address);
}
System.Text.UTF8Encoding encoding = new System.Text.UTF8Encoding();
byte[] data = encoding.GetBytes(stringBuilder.ToString());
Look at custom binary serialization and compression here
and here
Use the serialized class to convert the object into a byte via using a MemoryStream
using System.Runtime.Serialization.Formatters.Binary;
using System.Runtime.Serialization;
....
byte[] bPersonInfo = null;
using (MemoryStream mStream = new MemoryStream())
{
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bf = new BinaryFormatter();
bf.Serialize(mStream, personInfo);
bPersonInfo = mStream.ToArray();
}
....
// Do what you have to do with bPersonInfo which is a byte Array...
// To Convert it back
PersonInfo pInfo = null;
using (MemoryStream mStream = new MemoryStream(bPersonInfo)){
System.Runtime.Serialization.Formatters.Binary.BinaryFormatter bf = new BinaryFormatter();
pInfo = (PersonInfo)bf.DeSerialize(mStream);
}
// Now pInfo is a PersonInfo object.
Hope this helps,
Best regards,
Tom.