I'm using Firebase to host enumerated values that describe various events, for example (man, I wish StackOverflow supported markdown tables):
| Enumeration | Description |
| ----------- | ----------- |
| 0 | A description of a low-priority event. |
| 1 | A description of a critical event. |
I'm storing that data in Firebase as follows:
"enumerations": {
"events": {
"0": {
"description": "A description of a low-priority event."
},
"1": {
"description": "A description of a critical event."
}
}
}
The plan is to have a user download the list of enumerations using the Firebase Rest API with both the ?print=pretty and ?download=filename request parameters, like so:
https://FIREBASE_NAME.firebaseio.com/enumerations/events.json?print=pretty&download=events.json
But the data that is received looks like the following:
[ {
"description": "A description of a low-priority event."
},
{
"description": "A description of a critical event."
} ]
Now, I know why this happens -
I've read the "Best Practices: Arrays in Firebase" article. I understand how Firebase interprets objects that look like arrays and then returns them as array, but, in my case I would really like a way to force Firebase to return a JSON object with numbered keys as they are stored in Firebase.
I'd like a solution that keeps the formatting of the table as is rather than naming the keys something like "event1", "event2", etc. Again, this is really only for the REST API, as it would be easy to do this in code, but this is for an external user to download directly from Firebase, so potentially having a URL request parameter ?forceAsArray=true would be a nice solution.
Or maybe I'm overlooking something in the documentation - Is this possible today?
Related
I'm using the copy data activity in Azure Data Factory to copy data from an API to our data lake for alerting & reporting purposes. The API response is comprised of multiple complex nested JSON arrays with key-value pairs. The API is updated on a quarter-hourly basis and data is only held for 2 days before falling off the stack. The API adopts an oldest-to-newest record structure and so the newest addition to the array would be the final item in the array as opposed to the first.
My requirement is to copy only the most recent record from the API as opposed to the collection - so the 192th reading or item 191 of the array (with the array starting at 0.)
Due to the nature of the solution, there are times when the API isn't being updated as the sensors that collect and send over the data to the server may not be reachable.
The current solution is triggered every 15 minutes and tries a copy data activity of item 191, then 190, then 189 and so on. After 6 attempts it fails and so the record is missed.
current pipeline structure
I have used the mapping tab to specify the items in the array as follows (copy attempt 1 example):
$['meta']['params']['sensors'][*]['name']
$['meta']['sensorReadings'][*]['readings'][191]['dateTime']
$['meta']['sensorReadings'][*]['readings'][191]['value']
Instead of explicitly referencing the array number, I was wondering if it is possible to reference the last item of the array in the above code?
I understand we can use 0 for the first record however I don't understand how to reference the final item. I've tried the following using the 'last' function but am unsure of how to place it:
$['meta']['sensorReadings'][*]['readings'][last]['dateTime']
$['meta']['sensorReadings'][*]['readings']['last']['dateTime']
last['meta']['sensorReadings'][*]['readings']['dateTime']
$['meta']['sensorReadings'][*]['readings']last['dateTime']
Any help or advice on a better way to proceed would be greatly appreciated.
Can you call your API with a Web activity? If so, this pulls the API result into the data pipeline and then apply ADF functions like last to it.
A simple example calling the UK Gov Bank Holidays API:
This returns a resultset that looks like this:
{
"england-and-wales": {
"division": "england-and-wales",
"events": [
{
"title": "New Year’s Day",
"date": "2017-01-02",
"notes": "Substitute day",
"bunting": true
},
{
"title": "Good Friday",
"date": "2017-04-14",
"notes": "",
"bunting": false
},
{
"title": "Easter Monday",
"date": "2017-04-17",
"notes": "",
"bunting": true
},
... etc
You can now apply the last function to is, e.g. using a Set Variable activity:
#string(last(activity('Web1').output['england-and-wales'].events))
Which yields the last bank holiday of 2023:
{
"name": "varWorking",
"value": "{\"title\":\"Boxing Day\",\"date\":\"2023-12-26\",\"notes\":\"\",\"bunting\":true}"
}
Or
#string(last(activity('Web1').output['england-and-wales'].events).date)
i'm writing a script that needs to parse incoming json into line-by-line data, taking information from the json at multiple different levels. i'm using jq to parse the data.
the incoming json is an array of 'tasks'. each task [i.e. each element of the array] is an object that looks like this :
{
"inputData": {
"transfers": [
{
"source": {
"directory": "/path/to/source",
"filename": "somefile.mp3"
},
"target": {
"directory": "/path/to/target",
"filename": "somefile.mp3"
}
},
{
"source": {
"content": "<?xml version=\"1.0\" encoding=\"UTF-8\"?><delivery>content description</delivery>",
"encoding": "UTF-8"
},
"target": {
"directory": "/path/to/target",
"filename": "somefile.xml"
}
}
]
},
"outputData": {
"transferDuration": "00:00:37:10",
"transferLength": 187813298,
},
"updateDate": "2020-02-21T14:37:18.329Z",
"updateUser": "bob"
}
i want to read all of the tasks and, for each one, output a single line composed of the following fields :
task[n].inputData.transfers[].target.filename, task[n].outputData.transferLength, task[n].updateDate
i've got my filter chain to where it will choose the appropriate fields correctly, even to where it will pick the 'correct' single entry from amongst the multiple entries in the task[].inputData.transfers[] array, but when i try to get the output of more than a single element, the chain iterates over the array three times, and i get
task[0].inputData.transfers[].target.filename
task[1].inputData.transfers[].target.filename
task[2].inputData.transfers[].target.filename
...
task[n].inputData.transfers[].target.filename
then the results of the outputData.transferLength field for all elements,
then the results of the updateDate field for all elements.
here is my filter chain :
'(.tasks[].inputData.transfers[] | select(.target.filename | match("[Xx][Mm][Ll]$")).target.filename), .tasks[].outputData.transferLength, .tasks[].updateDate'
i'm thinking there must be some efficient way to group all of these multi-level elements together for each element of the array ; something like a 'with ...' clause, like with tasks[] : blablabla, but can't figure out how to do it. can anyone help ?
The JSON example contained a superfluous , that jq won't accept.
Your example filter chain appears to operate on .tasks[] even though the example appears to be only a single task. So it is not possible to rewrite what you have got into a functioning state. So rather than provide an exact answer to an inexact question, here is the first of the three parts of your filter chain revised:
.inputData.transfers | map(select(.target.filename | match("xml$"; "i")))
See this jqplay snippet.
Rather than write [ .xs[] | select(p) ], just write .xs | map(select(p)).
i finally found the answer. the trick was to pipe the .tasks[] into an expression where the parens were placed around the field elements as a group, which apparently will apply whatever is inside the parens to each element of the array individually, in sequence. then using #dmitry example as a guide, i also placed the elements inside right and left brackets to recreate array elements that i could then select, which could then be output onto 1 line each with | #csv. so the final chain that worked for me is :
.task[] | ([.inputData.transfers[].target.filename, .outputData.transferLength, .updateDate]) | [(.[0],.[2],.[3])] | #csv'
unfortunately i couldn't get match() to work in this invocation, nor sub() ; each of these caused jq to offer a useless error message just before it dumped core.
many thanks to the people who replied.
I have a large JSON file which contains an array. I am using Firebase for my app's backend and I want to use FirebaseArray to store the data.
It is simple to create a FirebaseArray from my Angular app and add data to it, but the nature of my app is that I have fetched data which I need to first import into Firebase somehow.
On the Firebase website the only option for importing is from a JSON. When I import my JSON file, the result is an object with numerical keys, which I realize is like an array, but has a major issue.
{
"posts": {
"0": {
"id": "iyo0iw",
"title": "pro patria mori"
},
"1": {
"id": "k120iw",
"title": "an english title"
},
"2": {
"id": "p6124w",
"title": "enim pablo espa"
}
}
}
Users are able to change the position of items, and the position of an item is also how items are uniquely identified. With multiple users this means the following problem can occur.
Sarah: Change post[0] title to "Hello everyone"
Trevor: Swap post[1] position with post[2]
Sarah: Change post[1] title to "This is post at index 1 right?"
If the following actions happen in a short space of time, Firebase doesn't know for sure what Sarah saw as post[1] when they changed the title, and can't know for sure which post object to update.
What I want is a way to import my JSON file and have the arrays become FirebaseArrays, not objects with numerical keys, which are like arrays and share the issue described above.
What you imported into your database is, in fact, an array. Firebase Realtime Database only really represents data as a nested hierarchy of key/value pairs. An array is just a set of key/value pairs where the the keys are all numbers, typically starting at 0. That's exactly the structure you're showing in your question.
To generate the sort of data that would be created by writing to the database using an AngularFire FirebaseArray, you would need to pre-process your JSON.
Firebase push IDs are generated on the client and you can generate one by calling push without arguments.
You could convert an array to an object with Firebase push ID keys like this:
let arr = ["alice", "bob", "mallory"];
let obj = arr.reduce((acc, val) => {
let key = firebase.database().ref().push().key;
acc[key] = val;
return acc;
}, {});
I'm new to alexa. I learnt and started to build a weather app.
right now I'm able to get weather data, but on the below condition,
I've craeated a custom slot(LIST_OF_CITIES) to hold Cities as below.
{
"intents": [
{
"intent": "WeatherIntent",
"slots": [
{
"name": "city",
"type": "LIST_OF_CITIES"
}
]
},
{
"intent": "AMAZON.HelpIntent"
},
]
}
and in my custom slot I gave as below.
Type Values
LIST_OF_CITIES Hyderabad | pune | london
and below are my Utterances
WeatherIntent give me {city} climate
WeatherIntent {city}
WeatherIntent what's the climate in {city}
WeatherIntent what's the weather in {city}
WeatherIntent {city}
when I run my program using any of the three cities mentioned in the above table, I'm able to get the correct. If I use anything apart from the above, it is sending back value as -4.
If I want to get tempreature of some other city, I need to add that city in the slot list.
Please let me know how can I get the vaues dynamically, I mean with out depending on the LIST_OF_CITIES, If I enter a city name, it should send back the result.
Also I tried adding type as LITERAL and also as AMAZON.LITERAL. When I saved it, I get the exception as
Error: There was a problem with your request: Unknown slot name '{city}'. Occurred in sample 'WeatherIntent get me weather of {city}' on line 1.
Please let me know where am I going wrong and how can I fix this.
Thanks
Amazon provides some default list Slot Types for cities or even regions. E.g.
AMAZON.US_CITY
AMAZON.AT_CITY
AMAZON.DE_REGION
...
You can use these as type when defining your custom slot.
First, don't use LITERAL - it is deprecated and isn't even supported at all outside US region.
And no, you can't manage the list of words dynamically.
Alexa will try to match what the user says with your LIST_OF_CITIES, and will try to return one of those words, but might return something else if it can't match one of those (as you have seen).
There are some custom slot types for cities that you can use and build off of, see here:
https://developer.amazon.com/public/solutions/alexa/alexa-skills-kit/docs/alexa-skills-kit-interaction-model-reference#h2_custom_syntax
But that probably won't work for you since each of them is just one country, so you will need to build your own list of cities (in your LIST_OF_CITIES).
I noticed something strange when testing my interaction model with the Alexa skills kit.
I defined a custom slot type, like so:
CAR_MAKERS Mercedes | BMW | Volkswagen
And my intent scheme was something like:
{
"intents": [
{
"intent": "CountCarsIntent",
"slots": [
{
"name": "CarMaker",
"type": "CAR_MAKERS"
},
...
with sample utterances such as:
CountCarsIntent Add {Amount} cars to {CarMaker}
Now, when testing in the developer console, I noticed that I can write stuff like:
"Add three cars to Ford"
And it will actually parse this correctly! Even though "Ford" was never mentioned in the interaction model! The lambda request is:
"request": {
"type": "IntentRequest",
...
"intent": {
"name": "CountCarsIntent",
"slots": {
"CarMaker": {
"name": "ExpenseCategory",
"value": "whatever"
},
...
This really surprises me, because the documentation on custom slot types is pretty clear about the fact that the slot can only take the values which are listed in the interaction model.
Now, it seems that values are also parsed dynamically! Is this a new feature, or am I missing something?
Actually that is normal (and good, IMO). Alexa uses the word list that you provide as a guide, not a definitive list.
If it didn't have this flexibility then there would be no way to know if users were using words that you weren't expecting. This way you can learn and improve your list and handling.
Alexa treat the provided slot values as 'Samples'. Hence slot values which are not mentioned in interaction model will also get mapped.
When you create a custom slot type, a key concept to understand is
that this is training data for Alexa’s NLP (natural language
processing). The values you provide are NOT a strict enum or array
that limit what the user can say. This has two implications
1) words and phrases not in your slot values will be passed to you,
2) your code needs to perform any validation you require if what’s
said is unknown.
Since you know the acceptable values for that slot, always perform a slot-value validation on your code. In this way when you get something other than a valid car manufacturer or something which you don't support, you can always politely respond back like
"Sorry I didn't understand, can you repeat"
or
"Sorry we dont have in our list. can you please
select something from [give some samples from your list]"
More info here