I am trying to work more reactively with Angular 15 and RxJS observables for a UI component. I only subscribe to the data in my component template (html). I have a service that receives data from an external system. The issue I have is the data may be received for multiple days and needs to be 'split' for the display usage.
In the display, there are individual components of data, that show the rows returned from the service call. The service makes an HTTP call to an external host.
this.Entries$ = this.Http_.get<Array<IEntry>>('http://host.com/api/entry');
This data is then an array of records with an EntryDate, and a structure of information (UserId, Description, TimeWorked, etc.). The external API sends all the records back as one flat array of data which is not guaranteed to be sorted, it comes back in a database order, which was the order records were entered. A sort might be needed for any processing, but I am not sure.
[
{ "EnterDate": 20221025, "UserId": "JohnDoe", "TimeWorked": 2.5, ... },
{ "EnterDate": 20221025, "UserId": "JohnDoe", "TimeWorked": 4.5, ... },
{ "EnterDate": 20221025, "UserId": "BSmith", "TimeWorked": 5, ... },
{ "EnterDate": 20221026, "UserId": "JohnDoe", "TimeWorked": 4, ... },
{ "EnterDate": 20221026, "UserId": "BSmith", "TimeWorked": 5, ... },
{ "EnterDate": 20221026, "UserId": "JohnDoe", "TimeWorked": 2, ... },
]
Currently, my HTML template loops through the Entries$ observable, when it was for just one day.
<ng-container *ngFor="let OneEntry of (Entries$ | async)">
<one-entry-component [data]=OneEntry />
</ng-container>
I want to be able to split my array of records into different datasets by EntryDate (and apparently user, but just EntryDate would work for now), similar to the groupBy(), but I do not know how to get to the internal record references, as it would be a map within the groupBy() I believe.
With the data split, I would then be looking to have multiple one-day-components on the page, that then have the one-entry-component within them.
|---------------------------------------------------------------|
| |
| |-One Day 1-------------###-| |-One Day 2-------------###-| |
| | | | | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | | | | |
| |---------------------------| |---------------------------| |
| |
| |-One Day 3-------------###-| |-One Day 4-------------###-| |
| | | | | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | [ One Line ] | | [ One Line ] | |
| | | | | |
| |---------------------------| |---------------------------| |
| |
|---------------------------------------------------------------|
The 4 boxes would be there if there were 4 separate days in the response. If there were 2 different dates, then just show 2 dates, but this could be 5 or 6 even.
I would need an Observable that had the dates for splitting (and even users) and then be able to pass this as data to the one<one-day-component [data]=OneDateOneUser$ />. My component needs this so that I can count the time entries for the title, which I believe is a simple .pipe(map()) operation.
Within the one-day-component, I would then simply loop through the OneDateOneUser$ observable to extract individual records to send to the one-entry-component as I do currently.
I believe the RxJS groupBy is what I need. However, I am new to RxJS, and working with the inner array of data is not clear to me in the example.
If the data is individual records like the example, and not an array of data, then it does work using the example RxJS reference.
import { of, groupBy, mergeMap, reduce, map } from 'rxjs';
of(
{ id: 1, name: 'JavaScript' },
{ id: 2, name: 'Parcel' },
{ id: 2, name: 'webpack' },
{ id: 1, name: 'TypeScript' },
{ id: 3, name: 'TSLint' }
).pipe(
groupBy(p => p.id, { element: p => p.name }),
mergeMap(group$ => group$.pipe(reduce((acc, cur) => [...acc, cur], [`${ group$.key }`]))),
map(arr => ({ id: parseInt(arr[0], 10), values: arr.slice(1) }))
)
.subscribe(p => console.log(p));
// displays:
// { id: 1, values: [ 'JavaScript', 'TypeScript' ] }
// { id: 2, values: [ 'Parcel', 'webpack' ] }
// { id: 3, values: [ 'TSLint' ] }
However, simply changing the data in the of() to be an array (more like how my data comes back), breaks, and I am not sure how to fix it:
import { of, groupBy, mergeMap, reduce, map } from 'rxjs';
of(
[
{ id: 1, name: 'JavaScript' },
{ id: 2, name: 'Parcel' },
{ id: 2, name: 'webpack' },
{ id: 1, name: 'TypeScript' },
{ id: 3, name: 'TSLint' }
]
).pipe(
groupBy(p => p.id, { element: p => p.name }),
mergeMap(group$ => group$.pipe(reduce((acc, cur) => [...acc, cur], [`${ group$.key }`]))),
map(arr => ({ id: parseInt(arr[0], 10), values: arr.slice(1) }))
)
.subscribe(p => console.log(p));
What if you just turned that Array<IEntry> into a Record<number, IEntry> with something like lodash's group by and a map RxJS operator?
Then you can get the desired outcome with some flex-wrap and flex-row functionality on the template and just loop over the entries of the record:
Check this little working CodePen
import {groupBy} from 'lodash'
const fakeData = [
{ "EnterDate": 20221025, "UserId": "JohnDoe", "TimeWorked": 2.5, ... },
{ "EnterDate": 20221025, "UserId": "JohnDoe", "TimeWorked": 4.5, ... },
{ "EnterDate": 20221025, "UserId": "BSmith", "TimeWorked": 5, ... },
{ "EnterDate": 20221026, "UserId": "JohnDoe", "TimeWorked": 4, ... },
{ "EnterDate": 20221026, "UserId": "BSmith", "TimeWorked": 5, ... },
{ "EnterDate": 20221026, "UserId": "JohnDoe", "TimeWorked": 2, ... },
]
// Replace "of" with your API call
entriesByDate$: Observable<Record<number, IEntry>> = of(fakeData).pipe(
map(allEntries => groupBy(allEntries, 'EnterDate'))
)
<div *ngIf="entriesByDate$ | async as entries" class="flex flex-row flex-wrap">
<ng-container *ngFor="let [enterDate, entries] of Object.entries(entries)">
<entry-group-component [title]="enterDate" [data]="entries" />
</ng-container>
</div>
No need to import lodash if you care to write the grouping function yourself. Array#reduce should suffice:
function groupByEnterDate(entries: Array<IEntry>) {
return entries.reduce(
(acc, current) => {
const key = current.EnterDate
const groupedByKey = acc[key] ?? []
return { ...acc, [key]: [...groupedByKey, current] }
},
{}
)
}
I'm working with those JSONs:
{
"extension": [
{
"url": "url1",
"system": "system1"
},
{
"url": "url2",
"system": "system2"
}
]
}
{
"extension": [
{
"url": "url3",
"system": "system3"
}
]
}
As you can see, both JSON objects have different .extension lenght.
I'm using this command in order to map input JSONs:
jq --raw-output '[.extension[] | .url, .system] | #csv'
You can find jqplay here.
I'm getting that:
"url1","system1","url2","system2"
"url3","system3"
What I would like to get is:
"url1","system1","url2","system2"
"url3","system3",,
Any ideas about how I could map those "fields" "correctly"?
Flip the table twice using transpose | transpose to fill up the slots missing from the unrigged square shape with null:
jq -rs 'map(.extension) | transpose | transpose[] | map(.url, .system) | #csv'
"url1","system1","url2","system2"
"url3","system3",,
Demo
A fairly efficient solution:
def pad:
(map(length)|max) as $mx
| map( . + [range(length;$mx)|null] );
[inputs | [.extension[] | (.url, .system)]]
| pad[]
| #csv
This of course should be used with the -n command-line option.
I have an array of json objects, each with an array of tags. Specific tags can appear multiple times in the child array but I only want the first matching tag (key+value) copied up onto the parent object. I've come up with a filter-set but it gives me multiple outputs if the given tag appears more than once in the child array ... I only want the first one.
Sample Json Input:
[
{
"name":"integration1",
"accountid":111,
"tags":[
{ "key": "env",
"values":["prod"]
},
{ "key": "team",
"values":["cougar"]
}
]
},
{
"name":"integration2",
"accountid":222,
"tags":[
{ "key": "env",
"values":["prod"]
},
{ "key": "team",
"values":["bear"]
}
]
},
{
"name":"integration3",
"accountid":333,
"tags":[
{ "key": "env",
"values":["test"]
},
{ "key": "team",
"values":["lemur"]
},
{ "key": "Env",
"values":["qa"]
}
]
}
]
Filter-set that I came up with:
jq -r '.[] | .tags[].key |= ascii_downcase | .env = (.tags[] | select(.key == "env").values[0])|[.accountid,.name,.env] | #csv' test.json
Example output with undesirable extra line:
111,"integration1","prod"
222,"integration2","prod"
333,"integration3","test"
333,"integration3","qa" <<<
Try using first(<expression>) to get only the first matching value. In case there are no matching values at all, you can use first(<expression>, <default_value>).
jq -r '.[] | .tags[].key |= ascii_downcase | .env = first((.tags[] | select(.key == "env").values[0]),null)|[.accountid,.name,.env] | #csv' test.json
Alternatively, if you are going to want to extract other tags similarly, you might prefer to extract them all into one object like this. I'm using reverse to meet your requirement of keeping the first match for any given key, otherwise the last match would win.
jq -r '.[] | .tags |= ( map({(.key|ascii_downcase): .values[0]}) | reverse | add ) | [.accountid, .name, .tags.env] | #csv'
Let's say I have more namespaces with the similar k8s resource (some might have different images used). I am trying to get .metadata.namespace using jq from the following json object (let's call it test.json):
{
"items": [
{
"metadata": {
"name": "app",
"namespace": "test1"
},
"spec": {
"components": [
{
"database": {
"from": "service",
"value": "redis"
},
"image": "test.com/lockmanager:1.1.1",
"name": "lockmanager01",
"replicas": 2,
"type": "lockmanager"
},
{
"database": {
"from": "service",
"value": "postgresql"
},
"image": "test.com/jobmanager:1.1.1",
"name": "jobmanager01",
"replicas": 2,
"type": "jobmanager"
}
]
}
}
]
}
if following condition is met:
.spec.components[].type == "jobmanager" and .spec.components[].image != "test.com/jobmanager:1.1.1"
but can't find the correct statement.
I tried:
cat test.json | jq '.items[] | select((.spec.components[].name? | contains("jobmanager01")) and (.spec.components[].image != "test.com/jobmanager:1.1.1")) | .metadata.namespace''
but it returns all namespaces and, moreover, those I am interested in (because I know they contain different image), are returned twice.
Please advise what am I doing wrong?
You state that the selection criterion is:
.spec.components[].type == "jobmanager" and
.spec.components[].image != "test.com/jobmanager:1.1.1"
but that does not make much sense, given the semantics of .[].
I suspect you meant that you want to select items from .spec.components such that
.type == "jobmanager" and .image != "test.com/jobmanager:1.1.1"
If that's the case, you could use any, so that your query would look like this:
.items[]
| select( any(.spec.components[];
(.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1") )
| .metadata.namespace
all distinct
If you want all the distinct .namespace values satisfying the condition, you could go with:
[.items[]
| .metadata.namespace as $it
| .spec.components[]
| select( (.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1" )
| $it]
| unique[]
Efficient version of "all-distinct" solution
To avoid unnecessary checks, if .namespace is always a string, we could write:
reduce .items[] as $item ({};
$item.metadata.namespace as $it
| if .[$it] then . # already seen
elif any( $item.spec.components[];
((.name? | contains("jobmanager01")) and
.image != "test.com/jobmanager:1.1.1") )
then .[$it] = true
else . end )
| keys_unsorted[]
I have a JSON result from an ElasticSearch query that provides multiple objects in the JSON result.
{
"buckets": [{
"perf_SP_percentiles": {
"values": {
"80.0": 0,
"95.0": 0
}
},
"perf_QS_percentiles": {
"values": {
"80.0": 12309620299487,
"95.0": 12309620299487
}
},
"latest": {
"hits": {
"total": 3256,
"max_score": null,
"hits": [{
"_source": {
"is_external": true,
"time_ms": 1492110000000
},
"sort": [
1492110000
]
}]
}
}
}]
}
I wrote the following jq with help from others
jq -r '.buckets[].latest.hits.hits[]._source | [."is_external",."time_ms"] | #csv'
I need to add the perf_QS_Percentiles to the CSV but getting an error.
jq -r '.buckets[].latest.hits.hits[]._source | [."is_external",."time_ms"], .buckets[].perf_QS_percentiles.values | [."80.0",."95.0"] | #csv'
I am getting an error jq: error (at <stdin>:734665): Cannot index array with string. may be I am missing something here. I am reading the JQ manual https://stedolan.github.io/jq/manual/#Basicfilters to see how to parse different JSON objects in the array, but asking here as someone may be able to point out more easily.
You can use (....) + (....) to create the array before piping to #csv :
jq -r '.buckets[] |
(.latest.hits.hits[]._source | [."is_external",."time_ms"]) +
(.perf_QS_percentiles.values | [."80.0",."95.0"]) | #csv'