Related
I am trying to parse some JSON from Jira to variables. This is using the go-jira package (https://godoc.org/github.com/andygrunwald/go-jira)
Currently I have some code to get the developer:
dev := jiraIssue.Fields.Unknowns["customfield_11343"].(map[string]interface{})["name"]
and team := jiraIssue.Fields.Unknowns["customfield_12046"].([]interface{})[0].(map[string]interface{})["value"]
to get the team they are a part of from.
Getting the team they are on is a bit gross, is there a cleaner way to get the team besides having to type assert, set the index, then type assert again?
Here is the complete json (modified but structure is same, its way too long):
{
"expand":"renderedFields,names,schema,operations,editmeta,changelog,versionedRepresentations",
"id":"136944",
"self":"https://jira.redacted.com/rest/api/2/issue/136944",
"key":"RM-2506",
"fields":{
"customfield_11343":{
"self":"https://redacted.com/rest/api/2/user?username=flast",
"name":"flast",
"key":"flast",
"emailAddress":"flast#redacted.com",
"displayName":"first last",
"active":true,
"timeZone":"Europe/London"
},
"customfield_12046":[
{
"self":"https://jira.redacted.com/rest/api/2/customFieldOption/12045",
"value":"diy",
"id":"12045"
}
],
}
Thanks
The way I go about problems like this is:
Copy some JSON with things I am interested in and paste it into https://mholt.github.io/json-to-go/
Remove fields that aren´t of interest.
Just read the data and unmarshal.
You might end up with something like this given the two custom fields of interest, but you can cut the structure down further if you just need the name.
type AutoGenerated struct {
Fields struct {
Customfield11343 struct {
Self string `json:"self"`
Name string `json:"name"`
Key string `json:"key"`
EmailAddress string `json:"emailAddress"`
DisplayName string `json:"displayName"`
Active bool `json:"active"`
TimeZone string `json:"timeZone"`
} `json:"customfield_11343"`
Customfield12046 []struct {
Self string `json:"self"`
Value string `json:"value"`
ID string `json:"id"`
} `json:"customfield_12046"`
} `json:"fields"`
}
The effect you get is that all extra information in the feed is discarded, but you get the data you want very cleanly.
This is a tough one since the second one is in an array form. It makes it hard to use a map.
For the first one, it's simple enough to use:
type JiraCustomField struct {
Self string `json:"self"`
Name string `json:"name"`
Key string `json:"key"`
EmailAddress string `json:"emailAddress"`
DisplayName string `json:"displayName"`
Active bool `json:"active"`
TimeZone string `json:"timeZone"`
}
type JiraPayload struct {
Expand string `json:"expand"`
ID string `json:"id"`
Key string `json:"key"`
Fields map[string]JiraCustomField `json:"fields"`
}
https://play.golang.org/p/y8-g6r0kInV
Specifically this part Fields map[string]JiraCustomField for the second case it looks like you need it in an array form like Fields map[string][]JiraCustomField.
In a case like this, I think you'll need to make your own Unmarshaler. This is a good tutorial: https://blog.gopheracademy.com/advent-2016/advanced-encoding-decoding/
What you could do with your custom Unmarshal/marshaler, is use the Reflection package and check if it's an array or a struct. If it's a struct then put it into an array, and store it in Fields map[string][]JiraCustomField.
I get data from a URL, and am working on the data to check for a few conditions. The data from the URL look like this:
1528190345":100,"1528190346":100,"1528190368":100,"1528190414":100,"1528190439":99,"1528190440":99,"1528190463":100,"1528190485":100,"1528190508":100,"1528190550":100,"1528190575":100,"1528190576":100,"1528190599":100,"1528190600":100,"1528190622":100,"1528190667":100,"1528190688":100,"1528190689":100,"1528190712":100,"1528190736":100,"1528190762":100,"1528190785":100,"1528190786":100,"1528190807":100,"1528190828":100,"1528190853":100,"1528190877":100,"1528190901":100,"1528190925":100,"1528190948":100,"1528190968":100,"1528190991":100}}]
====
I have converted that too JSON
{"metric"=>"Insta_real-unique_value", "tags"=>{"host"=>"letme.quickly.com", "tier"=>"2", "device"=>"tester1", "dc"=>"xxx"}, "aggregateTags"=>["device_name", "device_ip"], "dps"=>{"1526972408"=>100, "1526972424"=>100, "1526972440"=>100, "1526972456"=>100, "1526972472"=>100, "1526972488"=>100, "1526972504"=>100, "1526972520"=>100, "1526972536"=>100, "1526972552"=>100, "1526972568"=>100, "1526972569"=>100, "1526972584"=>100, "1526972585"=>100, "1526972601"=>100, "1526972617"=>100, "1526972633"=>100, "1526972649"=>100, "1526972665"=>100, "1526972681"=>100}}
I want to extract the value that corresponds to 100. When I do this:
url = "#{URL}"
uri = URI(url)
response = Net::HTTP.get(uri)
value = response[-6..-4]
puts value
I get the last value, but when the last value changes to 99/9/0, it prints :99 or ":9.
Is there a way to get the exact value as is?
When dealing with JSON data, it's almost always better to parse the data properly rather than using regex against the string.
In this case, we can do:
JSON.parse(response)['dps'].values.last #=> 100
If the response is a json response, you must use a json parser else if is not a json response, you can use a regex expression with a Regex Object.
In case of a json response, assuming that the object is something like is declared into the variable response of the next code, you can parse it into a JObject. (using Newtonsoft.Json available from nuget repository).
See the next example :
string response = "[{\"response\":{\"1528190345\":100,\"1528190346\":100,\"1528190368\":100,\"1528190414\":100,\"1528190439\":99,\"1528190440\":99,\"1528190463\":100,\"1528190485\":100,\"1528190508\":100,\"1528190550\":100,\"1528190575\":100,\"1528190576\":100,\"1528190599\":100,\"1528190600\":100,\"1528190622\":100,\"1528190667\":100,\"1528190688\":100,\"1528190689\":100,\"1528190712\":100,\"1528190736\":100,\"1528190762\":100,\"1528190785\":100,\"1528190786\":100,\"1528190807\":100,\"1528190828\":100,\"1528190853\":100,\"1528190877\":100,\"1528190901\":100,\"1528190925\":100,\"1528190948\":100,\"1528190968\":100,\"1528190991\":100}}]";
List<Dictionary<string, Dictionary<string, int>>> values = JsonConvert.DeserializeObject<List<Dictionary<string, Dictionary<string, int>>>>(response);
Dictionary<string, Dictionary<string, int>> firstLevel = values[0]; // Access to the first object of the list closed with ']'
Dictionary<string, int> secondLevel = firstLevel["response"]; // Access to the first object response and get's it's object context of first '}' starting from the end of response
/** This is an option, if you ever knows the name of the element (1528190991) */
int thirdLevel = secondLevel["1528190991"]; // Access to the last element of the object by it's name, context of second '}' starting from the end of response.
Console.WriteLine(thirdLevel);
/** This is another option if you doesn't know the name of the element and wants ever the last element. */
List<int> listOfValues = secondLevel.Values.ToList();
Console.WriteLine(listOfValues[listOfValues.Count-1]);
Note that i've chenged a little bit your response adding [{\"response\":{\" at the start to become a json response.
If is not a json response you can use this pattern with regular expression :
:(.{2,6})}}\]$
Hope will help!
How do I get the full message and not just the metadata using gmail api?
I have a service account and I am able to retrieve a message but only in the metadata, raw and minimal formats. How do I retrieve the full message in the full format? The following code works fine
var request = service.Users.Messages.Get(userId, messageId);
request.Format = UsersResource.MessagesResource.GetRequest.FormatEnum.Metadata;
Message message = request.Execute();
However, when I omit the format (hence I use the default format which is FULL) or I change the format to UsersResource.MessagesResource.GetRequest.FormatEnum.Full
I get the error: Metadata scope doesn't allow format FULL
I have included the following scopes:
https://www.googleapis.com/auth/gmail.readonly,
https://www.googleapis.com/auth/gmail.metadata,
https://www.googleapis.com/auth/gmail.modify,
https://mail.google.com/
How do I get the full message?
I had to remove the scope for the metadata to be able to get the full message format.
The user from the SO post have the same error.
Try this out first.
Go to https://security.google.com/settings/security/permissions
Choose the app you are working with.
Click Remove > OK
Next time, just request exactly which permissions you need.
Another thing, try to use gmailMessage.payload.parts[0].body.dataand to decode it into readable text, do the following from the SO post:
import org.apache.commons.codec.binary.Base64;
import org.apache.commons.codec.binary.StringUtils;
System.out.println(StringUtils.newStringUtf8(Base64.decodeBase64(gmailMessage.payload.parts[0].body.data)));
You can also check this for further reference.
try something like this
public String getMessage(string user_id, string message_id)
{
Message temp =service.Users.Messages.Get(user_id,message_id).Execute();
var parts = temp.Payload.Parts;
string s = "";
foreach (var part in parts) {
byte[] data = FromBase64ForUrlString(part.Body.Data);
s += Encoding.UTF8.GetString(data);
}
return s
}
public static byte[] FromBase64ForUrlString(string base64ForUrlInput)
{
int padChars = (base64ForUrlInput.Length % 4) == 0 ? 0 : (4 - (base64ForUrlInput.Length % 4));
StringBuilder result = new StringBuilder(base64ForUrlInput, base64ForUrlInput.Length + padChars);
result.Append(String.Empty.PadRight(padChars, '='));
result.Replace('-', '+');
result.Replace('_', '/');
return Convert.FromBase64String(result.ToString());
}
Background
I have a list of sObjects I need to insert, but I must first check if the insert will be successful. So, I'm setting a database save point before performing the insert and checking the save results (for the insert statement). Because, I don't want to process if any errors occurred, if there were any errors in the insert results, the database is rolled back to the save point.
Problem & Question
I need to collect the errors for each save (insert) result and associate each error to the specific sObject record that caused the error. According to the documentation Save results contain a list of errors, the Salesforce ID of the record inserted (if successful), and a success indicator (boolean).
How do I associate the Save Result to the original sObject record inserted?
Code/Example
Here's an example I put together that demonstrates the concept. The example is flawed, in that the InsertResults don't always match the sObjectsToInsert. It's not exactly the code I'm using in my custom class, but it uses the same logic.
Map<Id,sObject> sObjectsToInsert; // this variable is set previously in the code
List<Database.SaveResult> InsertResults;
Map<String,sObject> ErrorMessages = new Map<String,sObject>();
System.SavePoint sp = Database.setSavepoint();
// 2nd parameter must be false to get all errors, if there are errors
// (allow partial successes)
InsertResults = Database.insert(sObjectsToInsert.values(), false);
for(Integer i=0; i < InsertResults.size(); i++)
{
// This method does not guarantee the save result (ir) matches the sObject
// I need to make sure the insert result matches
Database.SaveResult ir = InsertResults[i];
sObject s = sObjectsToInsert.values()[i];
String em = null; // error message
Integer e = 0; // errors
if(!ir.isSuccess())
{
system.debug('Not Successful');
e++;
for(Database.Error dbe : ir.getErrors()) { em += dbe.getMessage()+' '; }
ErrorMessages.put(em, s);
}
}
if(e > 0)
{
database.rollback(sp);
// log all errors in the ErrorMessages Map
}
Your comment says the SaveResult list is not guaranteed to be in order, but I believe that it is. I've used this technique for years and have never had an issue.
I have images of various formats (.png, .jpg, .bmp, etc.) stored as compressed text in a text column in a SQL Server 2005 table. I need to read the row, unzip the image and store it in an image column in another table.
I am using the SharpZip library, and all of the examples deal with file sources and destinations. I can't find anything that covers unzipping from a variable to another variable. A code snippet illustrating this or a link to a relevant resource would be much appreciated.
EDIT: A bit more information - the data is stored in a TEXT column. It appears as follows (text column abbreviated for display):
ImageID ImageData
1 FORMAT-ZIPV3 UEsDBBQAAAAIAOV6wzxdTnDvshs...
2 FORMAT-ZIPV3 UEsDBBQAAAAIAAF2yjxGncjOLgA...
3 FORMAT-ZIPV3 UEsDBBQAAAAIAKd6yjyjnQNr6gg...
4 FORMAT-ZIPV3 UEsDBBQAAAAIALdNyzyrPC8EMJw...
5 FORMAT-ZIPV3 UEsDBBQAAAAIAA1rOD1nZY1t0f0...
6 FORMAT-ZIPV3 UEsDBBQAAAAIANZplj2seyJ+VmM...
7 FORMAT-ZIPV3 UEsDBBQAAAAIAC5vhD27LPbPcv8...
8 FORMAT-ZIPV3 UEsDBBQAAAAIAK1qKz5DJNH3xMg...
9 FORMAT-ZIPV3 UEsDBBQAAAAIAHVkEztC3th/9hs...
10 FORMAT-ZIPV3 UEsDBBQAAAAIAEtXKz7DXHUdvow...
What I know for certain is that the images were compressed at some point in the process using SharpZip before being inserted into the table. It appears that the format information was added to the beginning of the data prior to inserting.
Looking at this data, would anyone have any insight on how this image data has been manipulated? Again, I need to get the uncompressed image data into a column of a data type conducive to reading for display on a web page.
EDIT: Ok, I'm stumped. Executing the following code produces the error, "Failed to convert parameter value from a Int32 to a Byte[]". It appears to be placing the length of the byte array into the byte array's value...
commandUncompressed.Connection = connectionUncompressed;
commandUncompressed.Parameters.Add("#Image_k", SqlDbType.VarChar, 10);
commandUncompressed.Parameters.Add("#ImageContents", SqlDbType.Image);
commandUncompressed.CommandText = sqlSaveImage;
connectionUncompressed.Open();
reader = command.ExecuteReader();
if (reader.HasRows)
{
while (reader.Read())
{
Console.WriteLine(reader["Image_k"].ToString()); // Merely for testing
String format = reader["ImageContents_Compressed"].ToString().Substring(0, 12);
var offset = 13; //"FORMAT-ZIPV3 ".Length;
var s = reader["ImageContents_Compressed"].ToString().Substring(offset);
var bytes = Convert.FromBase64String(s);
if (format == "FORMAT-ZIPV2 ")
{
bytes = ConvertStringToBytes(s); // Not a Base-64 encoded string? External conversion function utilized.
}
using (var zis = new ZipInputStream(new MemoryStream(bytes)))
{
ZipEntry zipEntry = zis.GetNextEntry(); // Doesn't seem to work unless an entry has been referenced
byte[] buffer = new byte[zis.Length];
commandUncompressed.Parameters["#Image_k"].Value = reader["Image_k"].ToString();
commandUncompressed.Parameters["#ImageContents"].Value = zis.Read(buffer, 0, buffer.Length);
commandUncompressed.ExecuteNonQuery();
}
}
}
It appears to be reading the data from the source text column just fine. I just cannot figure out how to get that into the image type parameter. The value for buffer variable shows the length of the byte array, rather than the actual bytes. Maybe that's what the value property typically shows for byte arrays? I'm so close and yet so far away. :/
EDIT: Ok, I'm a knucklehead. I made the following correction, and it works!
zis.Read(buffer, 0, buffer.Length)
commandUncompressed.Parameters["#ImageContents"].Value = buffer;
At this point I am only able to process FORMAT-ZIPV3 data, as I haven't figured out how to decode the FORMAT-ZIP2 strings yet. Following is a sampling of the V2 data. If anyone is able to determine the encoding, let me know. Would it be different if zipped using BZIP instead of ZIP format?
ImageID ImageData
1 FORMAT-ZIPV2 504B03041400020008005157422A2E25FDBAF26701008D6901000E...
2 FORMAT-ZIPV2 504B03041400020008009159422A7FC94BA2B2540500D35705000E...
3 FORMAT-ZIPV2 504B0304140002000800685A422A0CAA51F4473A0600B97206000E...
4 FORMAT-ZIPV2 504B03041400020008001D5D422A770BD3ED201902002C4A02000E...
5 FORMAT-ZIPV2 504B0304140002000800325E422A4B6C2FB4045001001C6E01000E...
6 FORMAT-ZIPV2 504B03041400020008006F72422A5F793AC1A1F00200ECF302000E...
7 FORMAT-ZIPV2 504B0304140002000800D572422A1B348A731DE5000085EB00000E...
8 FORMAT-ZIPV2 504B03041400020008003D73422A8AEBB7F855640300DD1B04000E...
9 FORMAT-ZIPV2 504B03041400020008006368D528C5D0A6BA794900004A2502000E...
10 FORMAT-ZIPV2 504B03041400020008008E5B6C2A2D9E9C33D7AF05005CEC05000E...
In response to a similar question, someone on sqlmonster.com provided a nifty VarBinaryStream class. It works with a column type of varbinary(max).
If your data is stored in a varbinary(max), and is in zip format, you could use that class to instantiate a VarBinaryStream, then instantiate a ZipInputStream around that, and ba-da-boom, you're there. Just read from the ZipInputStream.
In C# it might look like this
using (var imageSrc = new VarBinarySource(connection,
"Table.Name",
"Column",
"KeyColName",
1))
{
using (var s = new VarBinaryStream(imageSrc))
{
using(var zis = new ZipInputStream(s))
{
....
}
}
}
If the images are small, then you probably wouldn't want all this streaming stuff. If the column is a binary(n) or a varbinary(n) where n is less than 8000, just use the SqlBinary type and read in all the data into memory, then instantiate a MemoryStream around that. Simpler. In VB.NET it looks something like this:
Dim bytes as Bytes()
bytes = dr.GetSqlBinary(columnNumber)
Using ms As New MemoryStream(bytes)
Using zis As New ZipInputStream(ms)
...
End Using
End Using
Finally, I'm going to question the wisdom of applying zip compression to .jpg images, and similar. The jpg format is already compressed; compressing it again before putting the data into SQL Server won't cause the data to become appreciably smaller. It only increases processing time. If possible, I'd suggest you reconsider your design for storage of compressed images.
ok, with the update you provided, containing the data format, you can draw some conclusions.
The data is an actual string. Suspecting that it is a Base64-encoded string, I did a small test and used Convert.ToBase64String() on a byte stream that contains a zip file. It looks like this: UEsDBBQAAAAIAJJyYyk3M56F+QIAA...
Aha! you have a base64-encoded (string) version of the byte data for a bonafide zip file. To decode it, strip the prefix and then use FromBase64String() to get the byte array, insert into a MemoryStream, then read it with ZipInputStream.
something like this:
var offset = "FORMAT-ZIPV3 ".Length();
var s = sqlReader["CompressedImage"].ToString().Substring(offset);
var bytes = Convert.FromBase64String(s);
using (var zis = new ZipInputStream(new MemoryStream(bytes)))
{
...
zis.Read(...);
...
}
If the data is "really long", you're going to want to stream it out of that table, rather than just read it into a big string and convert it. I don't know how large text columns can be, but supposing that it could be 500mb, you don't want a 500mb string, and you don't want to do a conversion of a 500mb string with Convert.FromBase64String(). In that case You need to use a Base64Stream, or the FromBase64Transform class in the System.Security.Cryptography namespace.
Editorial comment. It is sort of backwards to zip-compress image data. The images are probably compressed already. But to compound that backwardsness by then doing a base64 encode, thereby expanding the data... ??? That is triple backwards. That makes noooooo sense at all. I understand that's how your vendor supplied it.
Ok, with your furhter update, using this as the format:
ImageID ImageData
1 FORMAT-ZIPV2 504B03041400020008005157422A2E25FDBAF26701008D6901000E...
2 FORMAT-ZIPV2 504B03041400020008009159422A7FC94BA2B2540500D35705000E...
That data is still zipfile data, but it is encoded as simple hex digits. You need to convert that to a byte array. Here's some code to do it.
public static class ConvertEx
{
static readonly String prefix= "FORMAT-ZIPV2 ";
public static string ToHexString(byte[] b)
{
System.Text.StringBuilder sb1 = new System.Text.StringBuilder();
int i = 0;
for (i = 0; i < b.Length; i++)
{
sb1.Append(System.String.Format("{0:X2}", b[i]));
}
return sb1.ToString().ToLower();
}
public static byte[] ToByteArray(string s)
{
if (s.StartsWith(prefix))
{
System.Console.WriteLine("removing prefix");
s = s.Substring(prefix.Length);
}
s= s.Trim(); // whitespace
System.Console.WriteLine("length: {0}", s.Length);
var r= new byte[s.Length/2];
for (int i = 0; i < s.Length; i+=2)
{
r[i/2] = (byte) Convert.ToUInt32(s.Substring(i,2), 16);
}
return r;
}
}
You can use that this way:
string s = GetStringContentFromDatabase()
var decoded = ConvertEx.ToByteArray(s);
using (var ms = new MemoryStream(decoded))
{
// use DotNetZip to read the zip file
// SharpZipLib is something similar...
using (var zip = ZipFile.Read(ms))
{
// print out the list of entries in the zipfile
foreach (var e in zip)
{
System.Console.WriteLine("{0}", e.FileName);
}
}
}
The examples on the SharpZip Wiki use Stream objects - while the sample does use a File, you could easily use a MemoryStream object here and the sample would work the same.