I am a web dev noob so I apologize for my confusion in advance.
Background: I am attempting to create a full-stack kanban board application (think trello). I have a database that I have written using Django that exposes an API. I am attempting to use the information exposed via the API to create the kanban board. For example, my database has a table of columns:
{
"id": 3,
"title": "Should",
"dashboard": 1,
"created_at": "2021-04-08T16:43:42.755280Z",
"updated_at": "2021-04-08T16:43:42.755316Z"
}
There also exists a table of cards:
{
"id": 14,
"title": "This goes in must",
"description": "You must do this...",
"label": "",
"comment": "",
"created_at": "2021-04-19T17:44:31.332763Z",
"column": 1,
"position": 0
}
Problem: I am confused about how to work with this API on the front end. If I were using static data, each of my columns would have a list of cards and I would simply use the list of cards attached to the respective column to manage the state of the columns (there are many examples of this online). However, in my case, my columns have no reference to the cards that are contained within. How do I work with this?
Basically, in App I fetch both the columns and cards from my database using the fetch() method. This gives me an array of card objects called cards and an array of column objects called lists. Then in Apps return I simply create the columns using the list array.
lists.map((list, index) =>
<List title={ list.title } cards={cards} listID={list.id} key={list.id} index={index}/>
Then, in my list component I iterate over the cards and create the corresponding component.
cards.map( (card, index) => {
//if(card.column === listID)
return (<CardItem title={card.title} description={card.description}
id = {card.id} index={index} key={card.id}/>);
}
Here is the start of the problem, the cards array is ALL of the cards that exist in the database. So, when I go to create these columns they each get every single card regardless of whether that card belongs to them. To solve this, I have attempted to add a condition in my list component to only render cards where column === list.id. While this does render the correct cards in the correct columns, I still have problems when I attempt to rearrange the cards because I don't know how to manage the state of the columns. Say someone drags a card to a different column, how do I reflect that when I only have an array of cards? I mean, I have an array of columns, but it is effectively useless. It looks like this and I can't use it to control the state of the column.
0: Object { id: 1, title: "Must", created_at: "2021-04-08T00:41:45.338443Z", … }
1: Object { id: 3, title: "Should", dashboard: 1, … }
I'm very confused about how to work with the columns properly. Do I need to redesign the way they are stored on the database? How do I control their state when the array of column objects pulled from the database is effectively useless? Should I partition the array of cards based on their column id and use that to somehow control the state of the columns?
Any help/suggestions/resources would be greatly appreciated. Thank you, and have a good day.
Related
I'm using the copy data activity in Azure Data Factory to copy data from an API to our data lake for alerting & reporting purposes. The API response is comprised of multiple complex nested JSON arrays with key-value pairs. The API is updated on a quarter-hourly basis and data is only held for 2 days before falling off the stack. The API adopts an oldest-to-newest record structure and so the newest addition to the array would be the final item in the array as opposed to the first.
My requirement is to copy only the most recent record from the API as opposed to the collection - so the 192th reading or item 191 of the array (with the array starting at 0.)
Due to the nature of the solution, there are times when the API isn't being updated as the sensors that collect and send over the data to the server may not be reachable.
The current solution is triggered every 15 minutes and tries a copy data activity of item 191, then 190, then 189 and so on. After 6 attempts it fails and so the record is missed.
current pipeline structure
I have used the mapping tab to specify the items in the array as follows (copy attempt 1 example):
$['meta']['params']['sensors'][*]['name']
$['meta']['sensorReadings'][*]['readings'][191]['dateTime']
$['meta']['sensorReadings'][*]['readings'][191]['value']
Instead of explicitly referencing the array number, I was wondering if it is possible to reference the last item of the array in the above code?
I understand we can use 0 for the first record however I don't understand how to reference the final item. I've tried the following using the 'last' function but am unsure of how to place it:
$['meta']['sensorReadings'][*]['readings'][last]['dateTime']
$['meta']['sensorReadings'][*]['readings']['last']['dateTime']
last['meta']['sensorReadings'][*]['readings']['dateTime']
$['meta']['sensorReadings'][*]['readings']last['dateTime']
Any help or advice on a better way to proceed would be greatly appreciated.
Can you call your API with a Web activity? If so, this pulls the API result into the data pipeline and then apply ADF functions like last to it.
A simple example calling the UK Gov Bank Holidays API:
This returns a resultset that looks like this:
{
"england-and-wales": {
"division": "england-and-wales",
"events": [
{
"title": "New Year’s Day",
"date": "2017-01-02",
"notes": "Substitute day",
"bunting": true
},
{
"title": "Good Friday",
"date": "2017-04-14",
"notes": "",
"bunting": false
},
{
"title": "Easter Monday",
"date": "2017-04-17",
"notes": "",
"bunting": true
},
... etc
You can now apply the last function to is, e.g. using a Set Variable activity:
#string(last(activity('Web1').output['england-and-wales'].events))
Which yields the last bank holiday of 2023:
{
"name": "varWorking",
"value": "{\"title\":\"Boxing Day\",\"date\":\"2023-12-26\",\"notes\":\"\",\"bunting\":true}"
}
Or
#string(last(activity('Web1').output['england-and-wales'].events).date)
Iam new to React query and my attention got the following information:
Structural sharing only works with JSON-compatible values, any other
value types will always be considered as changed. If you are seeing
performance issues because of large responses for example, you can
disable this feature with the config.structuralSharing flag. If you
are dealing with non-JSON compatible values in your query responses
and still want to detect if data has changed or not, you can define a
data compare function with config.isDataEqual
React query
I changed my server to return an image like data response
app.get("/response", (req, res) => {
res.sendFile(path.join(__dirname, "image.jpg"));
});
But still in my app data remain unchanged. So what do they mean by saying JSON-compatible values?
This is an optimization that comes out of the box but it won't work for what you are describing. This is mostly for JSON API responses. React-query instead of creating a new data every time, it will compare the previous data with the new data and modify the values that have changed.
Or in words of one of the mantainers of the library
This feature makes sure that we keep referential identity of our data on every level. As an example, suppose you have the following data structure:
[
{ "id": 1, "name": "Learn React", "status": "active" },
{ "id": 2, "name": "Learn React Query", "status": "todo" }
]
Now suppose we transition our first todo into the done state, and we make a background refetch. We'll get a completely new json from our backend:
[
- { "id": 1, "name": "Learn React", "status": "active" },
+ { "id": 1, "name": "Learn React", "status": "done" },
{ "id": 2, "name": "Learn React Query", "status": "todo" }
]
Now React Query will attempt to compare the old state and the new and keep as much of the previous state as possible. In our example, the todos array will be new, because we updated a todo. The object with id 1 will also be new, but the object for id 2 will be the same reference as the one in the previous state - React Query will just copy it over to the new result because nothing has changed in it.
This comes in very handy when using selectors for partial subscriptions:
// ✅ will only re-render if _something_ within todo with id:2 changes
// thanks to structural sharing
const { data } = useTodo(2)
As I've hinted before, for selectors, structural sharing will be done twice: Once on the result returned from the queryFn to determine if anything changed at all, and then once more on the result of the selector function. In some instances, especially when having very large datasets, structural sharing can be a bottleneck. It also only works on json-serializable data. If you don't need this optimization, you can turn it off by setting structuralSharing: false on any query.
Source:
https://tkdodo.eu/blog/react-query-render-optimizations#structural-sharing
I have a large JSON file which contains an array. I am using Firebase for my app's backend and I want to use FirebaseArray to store the data.
It is simple to create a FirebaseArray from my Angular app and add data to it, but the nature of my app is that I have fetched data which I need to first import into Firebase somehow.
On the Firebase website the only option for importing is from a JSON. When I import my JSON file, the result is an object with numerical keys, which I realize is like an array, but has a major issue.
{
"posts": {
"0": {
"id": "iyo0iw",
"title": "pro patria mori"
},
"1": {
"id": "k120iw",
"title": "an english title"
},
"2": {
"id": "p6124w",
"title": "enim pablo espa"
}
}
}
Users are able to change the position of items, and the position of an item is also how items are uniquely identified. With multiple users this means the following problem can occur.
Sarah: Change post[0] title to "Hello everyone"
Trevor: Swap post[1] position with post[2]
Sarah: Change post[1] title to "This is post at index 1 right?"
If the following actions happen in a short space of time, Firebase doesn't know for sure what Sarah saw as post[1] when they changed the title, and can't know for sure which post object to update.
What I want is a way to import my JSON file and have the arrays become FirebaseArrays, not objects with numerical keys, which are like arrays and share the issue described above.
What you imported into your database is, in fact, an array. Firebase Realtime Database only really represents data as a nested hierarchy of key/value pairs. An array is just a set of key/value pairs where the the keys are all numbers, typically starting at 0. That's exactly the structure you're showing in your question.
To generate the sort of data that would be created by writing to the database using an AngularFire FirebaseArray, you would need to pre-process your JSON.
Firebase push IDs are generated on the client and you can generate one by calling push without arguments.
You could convert an array to an object with Firebase push ID keys like this:
let arr = ["alice", "bob", "mallory"];
let obj = arr.reduce((acc, val) => {
let key = firebase.database().ref().push().key;
acc[key] = val;
return acc;
}, {});
Let's say I have the following document schema in a collection called 'users':
{
name: 'John',
items: [ {}, {}, {}, ... ]
}
The 'items' array contains objects in the following format:
{
item_id: "1234",
name: "some item"
}
Each user can have multiple items embedded in the 'items' array.
Now, I want to be able to fetch an item by an item_id for a given user.
For example, I want to get the item with id "1234" that belong to the user with name "John".
Can I do this with mongoDB? I'd like to utilize its powerful array indexing, but I'm not sure if you can run queries on embedded arrays and return objects from the array instead of the document that contains it.
I know I can fetch users that have a certain item using {users.items.item_id: "1234"}. But I want to fetch the actual item from the array, not the user.
Alternatively, is there maybe a better way to organize this data so that I can easily get what I want? I'm still fairly new to mongodb.
Thanks for any help or advice you can provide.
The question is old, but the response has changed since the time. With MongoDB >= 2.2, you can do :
db.users.find( { name: "John"}, { items: { $elemMatch: { item_id: "1234" } } })
You will have :
{
name: "John",
items:
[
{
item_id: "1234",
name: "some item"
}
]
}
See Documentation of $elemMatch
There are a couple of things to note about this:
1) I find that the hardest thing for folks learning MongoDB is UN-learning the relational thinking that they're used to. Your data model looks to be the right one.
2) Normally, what you do with MongoDB is return the entire document into the client program, and then search for the portion of the document that you want on the client side using your client programming language.
In your example, you'd fetch the entire 'user' document and then iterate through the 'items[]' array on the client side.
3) If you want to return just the 'items[]' array, you can do so by using the 'Field Selection' syntax. See http://www.mongodb.org/display/DOCS/Querying#Querying-FieldSelection for details. Unfortunately, it will return the entire 'items[]' array, and not just one element of the array.
4) There is an existing Jira ticket to add this functionality: it is https://jira.mongodb.org/browse/SERVER-828 SERVER-828. It looks like it's been added to the latest 2.1 (development) branch: that means it will be available for production use when release 2.2 ships.
If this is an embedded array, then you can't retrieve its elements directly. The retrieved document will have form of a user (root document), although not all fields may be filled (depending on your query).
If you want to retrieve just that element, then you have to store it as a separate document in a separate collection. It will have one additional field, user_id (can be part of _id). Then it's trivial to do what you want.
A sample document might look like this:
{
_id: {user_id: ObjectId, item_id: "1234"},
name: "some item"
}
Note that this structure ensures uniqueness of item_id per user (I'm not sure you want this or not).
what seemed a simple task, came to be a challenge for me.
I have the following mongodb structure:
{
(...)
"services": {
"TCP80": {
"data": [{
"status": 1,
"delay": 3.87,
"ts": 1308056460
},{
"status": 1,
"delay": 2.83,
"ts": 1308058080
},{
"status": 1,
"delay": 5.77,
"ts": 1308060720
}]
}
}}
Now, the following query returns whole document:
{ 'services.TCP80.data.ts':{$gt:1308067020} }
I wonder - is it possible for me to receive only those "data" array entries matching $gt criteria (kind of shrinked doc)?
I was considering MapReduce, but could not locate even a single example on how to pass external arguments (timestamp) to Map() function. (This feature was added in 1.1.4 https://jira.mongodb.org/browse/SERVER-401)
Also, there's always an alternative to write storedJs function, but since we speak of large quantities of data, db-locks can't be tolerated here.
Most likely I'll have to redesign the structure to something 1-level deep, like:
{
status:1,delay:3.87,ts:138056460,service:TCP80
},{
status:1,delay:2.83,ts:1308058080,service:TCP80
},{
status:1,delay:5.77,ts:1308060720,service:TCP80
}
but DB will grow dramatically, since "service" is only one of many options which will append each document.
please advice!
thanks in advance
In version 2.1 with the aggregation framework you are now able to do this:
1: db.test.aggregate(
2: {$match : {}},
3: {$unwind: "$services.TCP80.data"},
4: {$match: {"services.TCP80.data.ts": {$gte: 1308060720}}}
5: );
You can use a custom criteria in line 2 to filter the parent documents. If you don't want to filter them, just leave line 2 out.
This is not currently supported. By default you will always receive the whole document/array unless you use field restrictions or the $slice operator. Currently these tools do not allow filtering the array elements based on the search criteria.
You should watch this request for a way to do this: https://jira.mongodb.org/browse/SERVER-828
I'm attempting to do something similar. I tried your suggestion of using the GROUP function, but I couldn't keep the embedded documents separate or was doing something incorrectly.
I needed to pull/get a subset of embedded documents by ID. Here's how I did it using Map/Reduce:
db.parent.mapReduce(
function(parent_id, child_ids){
if(this._id == parent_id)
emit(this._id, {children: this.children, ids: child_ids})
},
function(key, values){
var toReturn = [];
values[0].children.forEach(function(child){
if(values[0].ids.indexOf(product._id.toString()) != -1)
toReturn.push(child);
});
return {children: toReturn};
},
{
mapparams: [
"4d93b112c68c993eae000001", //example parent id
["4d97963ec68c99528d000007", "4debbfd5c68c991bba000014"] //example embedded children ids
]
}
).find()
I've abstracted my collection name to 'parent' and it's embedded documents to 'children'. I pass in two parameters: The parent document ID and an array of the embedded document IDs that I want to retrieve from the parent. Those parameters are passed in as the third parameter to the mapReduce function.
In the map function I find the parent document in the collection (which I'm pretty sure uses the _id index) and emit its id and children to the reduce function.
In the reduce function, I take the passed in document and loop through each of the children, collecting the ones with the desired ID. Looping through all the children is not ideal, but I don't know of another way to find by ID on an embedded document.
I also assume in the reduce function that there is only one document emitted since I'm searching by ID. If you expect more than one parent_id to match, than you will have to loop through the values array in the reduce function.
I hope this helps someone out there, as I googled everywhere with no results. Hopefully we'll see a built in feature soon from MongoDB, but until then I have to use this.
Fadi, as for "keeping embedded documents separate" - group should handle this with no issues
function getServiceData(collection, criteria) {
var res=db[collection].group({
cond: criteria,
initial: {vals:[],globalVar:0},
reduce: function(doc, out) {
if (out.globalVar%2==0)
out.vals.push({doc.whatever.kind.and.depth);
out.globalVar++;
},
finalize: function(out) {
if (vals.length==0)
out.vals='sorry, no data';
return out.vals;
}
});
return res[0];
};