Byte Array To String Conversion Inconsistency - arrays

I am testing out AES encryption on android and i need to convert the encrypted text to a string so as to store it in my firebase database. I managed to do so sucessfully. The problem comes when I try to retrieve the string from the database and convert it back to a byte array for decryption.
The retreived string is identical to the string sent to the database but when i convert it to a byte array and compare the two values, they are different as seen in the code sample below. I even made sure to use the same Charset but this did not fix the issue.
val encryptedPair: Pair<ByteArray,ByteArray> = encrypt("text")
val enc: MutableMap<String, Any> = HashMap()
enc["ciphertext"] = encryptedPair.second.toString(Charsets.UTF_8).trim()
fireStoreDatabase.collection("enc").document("test").set(enc)
.addOnSuccessListener {
fireStoreDatabase.collection("enc").document("test").get()
.addOnSuccessListener { doc->
//The below if condition prints out encrytion method: diff
if(encryptedPair.second.contentEquals(doc.getString("password")!!.toByteArray(Charsets.UTF_8)))
Log.e("encryption method: ","same")
else
Log.e("encryption method: ","diff")
}
}
This difference leads to the decryption process failing. Note that when i try to encrypt and decrypt with the values locally, it works as expected. So i know the problem is with the byte array i get from firebase.
Any suggestions would be appreciated.

As Jon Skeet has suggested, using
val enc: MutableMap<String, Any> = HashMap()
enc["ciphertext"] = Blob.fromBytes(encryptedPair.second)
fireStoreDatabase.collection("enc").document("test").set(enc)
to save the data to firebase and then retrieving it with:
doc.getBlob("ciphertext")!!.toBytes())
worked as expected.

Related

Byte array to Firestore

I have a byte array how do I store it in Firestore?
preferable to store as a File in Firestore Storage but storing them as custom object is fine too in Cloud Firestore.
I tried to make a file and use the function write Bytes() and on retrieving use read bytes but that is not able to work proper and causing some disruption as my byte array is a encrypted form of image which I need to store in Firestore. And Retrieve later and decrypt it.
Before I was trying to convert Byte Array to string but many people told that is no good practice as it will cause loss of data.
Than when I researched a bit few people recommended to use Blob and store the byte array but unfortunately I was unable to get the sample code for it.
val z = cipher.doFinal(byteArray)
val file = File.createTempFile("$name", null)
file.writeBytes(z)
Log.d("first","quick")
val storageReference = FirebaseStorage.getInstance()
.getReference("/image/0XhL4jD4XCemk38rcRkIEjJMgjh2/Aadhar/")
storageReference.putFile(Uri.fromFile(file))
.addOnSuccessListener {
Toast.makeText(context , "Success", Toast.LENGTH_SHORT).show()
}
Basically Z is the encrypted byte array that I somehow need to push to Firestore. I tried to make file and push in storage. Its Successfully completed.
But when I retrieve it back Something like below:-
val storageRef =
FirebaseStorage.getInstance().reference?.child("/image/0XhL4jD4XCemk38rcRkIEjJMgjh2/Aadhar/")
val localfile = File.createTempFile("temp", ".txt")
storageRef.getFile(localfile)
.addOnSuccessListener {
val bytes = localfile.readBytes()
var decryptedText: ByteArray = cipher.doFinal(bytes)
val configBmp = Bitmap.Config.valueOf(bitmap.config.name)
val bitmap_tmp = Bitmap.createBitmap(width, height, configBmp)
val buffer = ByteBuffer.wrap(decryptedText)
bitmap_tmp.copyPixelsFromBuffer(buffer)
img.setImageBitmap(bitmap_tmp)
}
The problem when I read bytes I got no clue what but something is going wrong there. I can be of sure that encryption and decryption both are working fine as I tested it Locally without pushing the byte array to Firestore and Just decrypt and it works perfectly good.
But when for the same byte array I try to store in file and push and retrieve later it doesn't works. The error I get is Illegal block size.
Simplified Question:-
Assume I have a byte array 'X'
How can I store it in Firestore [Cloud Firestore / Storage in form of files[preferred] with 0 loss of data and retrieve it back.
Help would be highly appreciated guys with a snippet of code to understand,
Thanks in adv.

Getting rates from an array - json

I am trying to get the rates from this website.
So I connect with website = Faraday.get('https://bitpay.com/api/rates')).status == 200 and then try to parse this.
A segment of the response I get is:
#<Faraday::Response:0x007fcf1ce25688
#env=
#<struct Faraday::Env
method=:get,
body=
"[{\"code\":\"BTC\",\"name\":\"Bitcoin\",\"rate\":1}, {\"code\":\"USD\",\"name\":\"US Dollar\",\"rate\":586.66},{\"code\":\"EUR\",\"name\":\"Eurozone Euro\",\"rate\":528.991322},{\"code\":\"GBP\",\"name\":\"Pound Sterling\",\"rate\":449.441986},{\"code\":\"JPY\",\"name\":\"Japanese Yen\",\"rate\":59907.95922},{\"code\":\"CAD\",\"name\"
When I do a website.body I get a String class of all these values found on that website. I want to parse them though (JSON?) so that I can get each rate as a float.
I tried something JSON.parse(website.body)["GBP"]["rate"].to_f but yet again it cannot work in a string.
The return I get is TypeError: no implicit conversion of String into Integer
I was having a similar (but not the same) format from a different rates website and this is how I was handling it. Do I need to change its format first or is there a different way around it?
You're trying to access to the parsed JSON with the key "GBP" but you have an array. It's like if you did
a = [1,2,3,4,5]
a['foo']
Try out
currencies = JSON.parse(website.body)
currencies.each { |currency| puts currency['rate'] }
and change it like you need

Unable to translate bytes [...] at index ... from specified code page to Unicode when adding to index

I am using Newtonsoft.Json to create the JSON to update add items to an index, but I get the following error when I POST the request:
{"error":{"code":"","message":"The request is invalid.","innererror":{"message":"parameters : Unable to translate bytes [E3] at index 752 from specified code page to Unicode.\r\n","type":"","stacktrace":""}}}
I know the error occurs with some non letter characters in some of the strings in the data that I am serializing. The string data comes from SQL, so I'm guessing something is going on to do with encoding that I cannot figure out.
When I inspect the JSON string, and put it in manually construct a request with the same data in Fiddler it all works fine.
Does anyone have any idea what might be the problem, and how I can work around it?
I found my own solution after a bit more digging.
Adding "StringEscapeHandling.EscapeNonAscii" to the serialization options solves the problem:
jsonSettings = new JsonSerializerSettings
{
Formatting = Newtonsoft.Json.Formatting.Indented,
ContractResolver = new CamelCasePropertyNamesContractResolver(),
DateTimeZoneHandling = DateTimeZoneHandling.Utc,
StringEscapeHandling = StringEscapeHandling.EscapeNonAscii
};

MD5 of file downloaded from database, from a JSONObject

My requirement is to compare the MD5 hashes of a file on the local disk and a file downloaded from a database.
The file is stored on SQL Server in a VARBINARY(MAX) column. The file can be any type. I'm currently testing with a PDF file. I get the file from the database using a HttpPost request. A JSONObject is built using the HttpResponse object. The JSONObject contains the file contents in binary format.
Now I have to compare the MD5 hash of the received binary data against the MD5 hash of the same file on disk. I have written the following code but the MD5 hashes do not match.
I think I'm going wrong in simply calculating the MD5 of the downloaded binary contents. Is there a correct way to do this? Thanks in advance.
// Read response from a HttpResponse object 'response'
BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent()));
String line="";
StringBuilder sb = new StringBuilder();
while((line=reader.readLine())!=null) {
sb.append(line);
}
// construct jsonobject
JSONObject jsonResponse = new JSONObject(sb.toString());
//Read file from disk
FileInputStream fis = new FileInputStream(new File(this.getClass().getResource("C:\\demo.pdf").getPath()));
// Calculate MD5 of file read from disk
String md5Request = org.apache.commons.codec.digest.DigestUtils.md5Hex(fis);
// Calculate MD5 of binary contents. "binfile" is name of key in the JSONObject
// and binary contents of downloaded file are in its corresponding value field
String md5Response = org.apache.commons.codec.digest.DigestUtils.md5Hex(jsonResponse.getString("binfile"));
Assert.assertEquals("Hash sums of request and response must match", md5Request, md5Response);
When I debug, I see this value against the binfile key in the JSONObject 'jsonResponse'
binfile=[37,80,68,70,45,49,46,52,13,37,-30,-29,-49,-45,13,10,52,48...]
and what follows is a lengthy stream of binary data.
OK, in SQL there's a build-in function that looks like this:
select *,
convert(varchar(50),master.sys.fn_repl_hash_binary(a.BinaryField),2) as 'MD5Hash'
from SomeTable a
You give the fn_repl_hash_binary the name of the binary field you're reading, plus "2" as an argument which tells SQL to calc the value as an MD5; I think "1" is SHA.
And in Java, you can use something like this:
private String getMD5Hash(byte[] bytes) throws java.lang.Exception{
String s="This is a test";
MessageDigest m=MessageDigest.getInstance("MD5");
m.update(bytes,0,bytes.length);
return new BigInteger(1,m.digest()).toString(16);
}
This should do the trick. Best of luck, CodeWarrior.
It is not a new post but here is a possible solution, as I faced this problem too on python and made a bunch of test to find how to do...
As you treat all data in binary, you need to open the file to compare in binary mode.
My original code that was failing every time to read the correct MD5 checksum:
with open(filepath, "r") as file_to_check:
tile_file = file_to_check.read()
Corrected code:
with open(filepath, "rb") as file_to_check:
tile_file = file_to_check.read()
Simply adding the b (binary) after the read (r) flag to let python know it need to read the file as binary and now it works.
This might be what will help you find your problem... Hope it helps!

Intermitent Base64 Task Conversion Errors

I'm experiencing a really weird situation when passing on a POJO java object within the payload of a Pull Queue task using Gson. Without changing the code or the POJO being set within the payload of a task, this will randomly succeed or fail.
This is the code I'm using:
PullQueueTaskPayLoad tqp = new PullQueueTaskPayLoad("id","name");
tqp.uploadURL = taskPayLoad.uploadURL;
tqp.urls = taskPayLoad.urls;
tqp.sliceQueryParameter = taskPayLoad.sliceQueryParameter;
TaskOptions task = TaskOptions.Builder.withMethod(TaskOptions.Method.PULL);
task.payload(new Gson().toJson(tqp));
q.add(task);
Using an external queue consumer I then retrieve the POJO as follows:
Type GSON_TYPE = new TypeToken<PullQueueTaskPayLoad>() {}.getType();
byte[] b = new Base64().decodeBase64(leasedTask.getPayloadBase64().getBytes());
String payload = new String(b);
logger.info("About to convert payload: "+payload);
PullQueueTaskPayLoad taskpayload = new Gson().fromJson(payload, GSON_TYPE);
So from the debugging I did, the problem seems to be happening when I'm decoding the payload bytes. While encoding the same POJO (with different Ids) I randomly get 2 different decoded payload Strings as follows:
Correct decoding:
{"id":"1786024566","sliceQueryParameter": {"queryId":786024566,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?"],"http://www.slicepedia.org/ontology#hasroot":["?"]}}},"uploadURL":"http://3.linguabox0412.appspot.com/_ah/upload/AMmfu6YRjxX23Ks-yh-9AZs4-3I1p6hxrFd6d4ptxSQegUkQHN7y4hNZwX6u7PufIHJbwtsHLXFZJ5P-vs90mslZEOMw0T-amN2qhEOAj_6YdwuY50FXMi8/ALBNUaYAAAAAT7Towgs4M00M5RLI8xnEOMdIxouZzuGu/","status":"IN_PROGRESS","action":"SLICE_SEARCH_AND_CREATE"}
Incorrect decoding:
{"id":"1-1968382407","sliceQueryParameter":{"queryId":-1968382407,"sliceStart":-1,"sliceNumber":1,"params":{"DefaultAnnotation":{"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints_SIGN":["\u003d"],"http://www.slicepedia.org/ontology#hasNumberOfBulletPoints":["0"],"http://www.slicepedia.org/ontology#hasNumberOfTokens":["80"],"http://www.slicepedia.org/ontology#hasNumberOfTokens_SIGN":["\u003e"]},"VG":{"http://www.slicepedia.org/ontology#hastense":["?K??????˜?X?\YXK?ܙ?????H?\ܛ????Ȃ%?????'W??EU$?#?&?GG???2?Ɩ?wV&??C"?7?B?6??????W??B???gSe????'u?U'd?D??6?S??4UV?D?e7?%U?&%F%f?D?$???$&vu6?fF$????EG?v??6?6դvt?D???G??&D?fdֵ6%?甦??GD????F???$?V?CuF?$?F?F֤֧f?D??u?wt?4?C$?W?"?'7FGW2#?$???$?u$U52"?&7F???#?%4Ĕ4U?4T$4???E?5$TDR'
So the second string obviously fails when using Gson to convert it back to a POJO. But I dont' understand why this happens in only some cases and not others. For what I've seen, it seems to always happen after a ["?"] character string. I tried replacing and ? with other strings but it didn't change anything.
I think what is happening here is that the payload is webSafe-base64 encoded. In practice, this means swapping + and / and = for - and _ and .. Most base64 libraries have native support for decoding websafe base 64 strings.
Probably you are meeting one of these chars at a certain point, and that kills the decoding.
Here is some info on WebSafeBase64
Word of warning, though: the taskqueue implementation is actually sending padding equals (=) that you will have to convert manually before parsing.

Resources