Firebase pagination using a timestamp/date as the orderBy - reactjs

I am struggling to get pagination working when I use a date (firebase timestamp) to retrieve data.
This is basically what I do:
let jobsRef = db.collection("jobs")
.orderBy('createdAt', 'desc')
.limit(QUERY_LIMIT);
jobsRef = jobsRef.startAfter(this.props.jobs[this.props.jobs.length - 1].createdAt);
However it seems that i get returned items sometimes that I have just already received. I am guessing because of similar dates?
So how could I basically return a list of jobs ordered by createdAt and have an offset/limit (pagination)?
createdAt looks like the timestamp type: 23 October 2020 at 17:26:31 UTC+2
When I log createdAt however I see this: {seconds: 1603537477, nanoseconds: 411000000}
Maybe I should be storing createdAt as a unix timestamp? Or what is the ideal way to deal with this?
Here is how it looks in the database (popup when i click edit on createdAt):

If multiple documents can have the same value for the field you're sorting on, passing in a value for that field is not guaranteed to point to a unique document. So you indeed may be passing in an ambiguous instruction, leading to an unwanted result.
When possible, I highly recommend passing the entire document to the Firestore API. This leaves it up to Firestore to take the necessary data from that document to uniquely/unambiguously find the anchor for your query.
So instead of:
jobsRef.startAfter(this.props.jobs[this.props.jobs.length - 1].createdAt);
Do:
jobsRef.startAfter(this.props.jobs[this.props.jobs.length - 1]);

I was facings a similar problem, after may hours I finally found a solution, all you need to do is converting that number to firestore's Timestamp
import { Timestamp }. from #angular/fire/firestore;
let createdAt: number = 56772766688383;
let timestamp = Timestamp.fromMillis(createdAt);
//then pass that to startAfter
startAfter(timestamp)

Related

Django query bases on greater date

I want to know how efficient this filter can be done with django queries. Essentially I have the followig two clases
class Act(models.Model):
Date = models.DateTimeField()
Doc = models.ForeignKey(Doc)
...
class Doc(models.Model):
...
so one Doc can have severals Acts, and for each Doc I want to get the act with the greater Date. I'm only interested in Acts objects.
For example, if a have
act1 = (Date=2021-01-01, Doc=doc1)
act2 = (Date=2021-01-02, Doc=doc1)
act3 = (Date=2021-01-03, Doc=doc2)
act4 = (Date=2021-01-04, Doc=doc2)
act5 = (Date=2021-01-05, Doc=doc2)
I want to get [act2, act5] (the Act with Doc=doc1 with the greater Date and the Act with Doc=doc2 with the greater Date).
My only solution is to make a for over Docs.
Thank you so much
You can do this with one or two queries: the first query will retrieve the latest Act per Doc, and then the second one will then retrieve the acts:
from django.db.models import OuterRef, Subquery
last_acts = Doc.objects.annotate(
latest_act=Subquery(
Act.objects.filter(
Doc_id=OuterRef('pk')
).values('pk').order_by('-Date')[:1]
)
).values('latest_act')
and then we can retrieve the corresponding Acts:
Act.objects.filter(pk__in=last_acts)
depending on the database, it might be more efficient to first retrieve the primary keys, and then make an extra query:
Act.objects.filter(pk__in=list(last_acts))

Mongodb has a bug on "$lte" (query or aggregation) while searching for dates ranges

Scenario:
I have a db hosted on MongoDb Atlas.
This db has a collection which, among other data, has a created field of type Date.
pseudoCode Schema:
... {
created: { type: Date }
}...
I want to perform a query that allows me to find all the objects existing in the collection which have a created value between specifics days including the boundary dates.
Let's assume that the date range is 2020-08-01 and 2020-08-31, the query would be
{created: {'$gte': new Date('2020-08-01'), '$lte': new Date('2020-08-31')}}
right?
Wrong.
By doing the query this way, I only get results that are greater than or equal to "2020-08-01" and lower than "2020-08-31". Meaning that, even if I'm performing an $lte query, I always get the $lt results.
Tests I did
I've tested this only for a type Date field atm on different collections and having consistently the same issue. Didn't have time for further investigations on different data types yet.
I've tested it on aggregation $match pipelines and find queries on:
my codebase
a clean script that just does this operations
directly on MongoDb Compass
In all 3 cases, the results are consisent with the problem exposed and confirm the problem.
Quick fix
Simply use $lt instead of $lte and always consider 1 day more than you intended.
Using the previous example, the query will become
{created: {'$gte': new Date('2020-08-01'), '$lt': new Date('2020-09-01')}}
and in this case, I'm getting the expected date range "2020-08-01" - "2020-08-31" results.
Note that I could have also used $lte and I would get the exact same results however, the $lt form is logically more correct for whom is reading the code.
Why I'm posting this
I did find few people have posted about this issue across the years, more relevant links are this GitHub issue (initially the dev believed the problem was with mongoose, then a solution proposed to check the schema but that's not the issue since in my case the schema is properly defined and I've tested it on Compass directly) and this google group discussion (the problem is poorly stated and received no answer).
But I did not find a solution.
Even though I've quick fixed the issue, I wanted to point it out better and understand if:
I'm doing something wrong and this is the expected behavior
there is something I'm doing wrong in my query
there is a problem with $lte which need to be addressed properly
Who has ideas?
When you run new Date('2020-08-01') then the result is actually ISODate("2020-08-01T00:00:00Z")
So
{created: {'$gte': new Date('2020-08-01'), '$lte': new Date('2020-08-31')}}
becomes
{created: {'$gte': ISODate("2020-08-01T00:00:00Z"), '$lte': ISODate("2020-08-31T00:00:00Z")}}
i.e. day 2020-08-31 is not included. You may also consider time zones if data was inserted as local time and thus not stored as 2020-08-02T00:00:00Z but 2020-08-02T02:00:00Z for example.
One solution is to add one day and use $lt:
{created: {'$gte': new Date('2020-08-01'), '$lt': new Date('2020-09-01')}}
or you can use Moment.js like this:
{created: {'$gte': new Date('2020-08-01'), '$lte': moment.utc('2020-08-31').endOf('day').toDate()}}
or perhaps moment.utc('2020-08-01').endOf('month').toDate()
Quick fix
Simply use $lt instead of $lte and always consider 1 day more than you intended.
Using the previous example, the query will become
{created: {'$gte': new Date('2020-08-01'), '$lt': new Date('2020-09-01')}}
and in this case, I'm getting the expected date range "2020-08-01" - "2020-08-31" results.
Note that I could have also used $lte and I would get the exact same results however, the $lt form is logically more correct for whom is reading the code.

Querying Firestore without Primary Key

I'd like my users to be able to update the slug on the URL, like so:
url.co/username/projectname
I could use the primary key but unfortunately Firestore does not allow any modifcation on assigned uid once set so I created a unique slug field.
Example of structure:
projects: {
P10syfRWpT32fsceMKEm6X332Yt2: {
slug: "majestic-slug",
...
},
K41syfeMKEmpT72fcseMlEm6X337: {
slug: "beautiful-slug",
...
},
}
A way to modify the slug would be to delete and copy the data on a new document, doing this becomes complicated as I have subcollections attached to the document.
I'm aware I can query by document key like so:
var doc = db.collection("projects");
var query = doc.where("slug", "==", "beautiful-slug").limit(1).get();
Here comes the questions.
Wouldn't this be highly impractical as if I have more than +1000 docs in my database, each time I will have to call a project (url.co/username/projectname) wouldn't it cost +1000 reads as it has to query through all the documents? If yes, what would be the correct way?
As stated in this answer on StackOverflow: https://stackoverflow.com/a/49725001/7846567, only the document returned by a query is counted as a read operation.
Now for your special case:
doc.where("slug", "==", "beautiful-slug").limit(1).get();
This will indeed result in a lot of read operations on the Firestore server until it finds the correct document. But by using limit(1) you will only receive a single document, this way only a single read operation is counted against your limits.
Using the where() function is the correct and recommended approach to your problem.

How to query to couchdb by key and string search

I'm new to CouchDB and I have documents like this:
{
"credit_type" : "ADJ",
"particular" : "Adjusted hours on 2018-01-01"
}
Then I have a view with this Map function
function(doc) {
if(doc.credit_type == "ADJ") { emit(doc.particular, doc); }
}
My view url is like this:
http://mywebsite.com:5984/_utils/database.html?client_docs/_design/adj/_view/adj
What I want to do is be able to query documents that will match my search key. The lookup field is the particular field.
The search key is a date like 2018-01-01 When I put they search url like
http://black-widow.remotestaff.com:5984/client_docs/_design/adj/_view/adj?key=20180-01-01
I should be able to fetch records which contains 2018-01-01 string in the particular field
I don't have any reduce function yet
In order to search by some key, you must emit that key.
For you to be able to do this, I would suggest potentially moving the date out of the particular field, into its own field, this way you wont have to do any string parsing on the particular field to grab your date.
If you do not or can not do this, the you will have to parse the particular field for the date. This can be achieved by this question and answer.*
Once you have gotten the date, you will need to emit it, so let me give you an example.
function(doc){
String date = doc.particular.parse.date.here();
emit([date, doc.credit_type], doc.particular);
}
Here, we parse the date from the doc.particular field, and emit it along with the credit type. Now, you can keep your if statement as it is, but doing it this way means you can search for ANY credit_typevs just the ADJ.
Now your keys should be 2018-01-01 and ADJ.
*You will need to make sure that CouchDB version of JS supports these functions.

all time data without setting facet.date.start to a random past date

I use the following query to get time-series data of Posts.
q=*:*&facet=true&facet.date=created_at_d&start=0
&f.created_at_d.facet.date.start=2009-06-20T14:05:28Z
&f.created_at_d.facet.date.end=2011-07-05T14:05:28Z
&f.created_at_d.facet.date.gap=%2B3600SECONDS
&fq=type:Post&rows=0&f.created_at_d.facet.mincount=1
&facet.sort=count
Is there a way to get all time data without setting facet.date.start to a random past date(say 100 years)?.
All time data => from the created_at of the very first Post.
It's a separate query for solr or your db to find that first date so probably easiest to get from the database.
Post.first(:order => 'created_at ASC').created_at.iso8601

Resources