Looking up index in a MongoDB array - database

Question

Our data provider supplies the data in a weird format. The arrays date and value are corresponding and guaranteed to have the same length. For whatever reason, they even decide to mix up int and string values in date.
[
{
"_id": "A000005933",
"date": [905270400000, 918748800000, 937843200000, 965923200000, 983289600000, 984931200000, 1152806400000, "1171987200000", "1225382400000", "1229616000000", "1286208000000", "1455552000000"],
"value": ["0.25", "0.15", "0", "0.25", "0.15", "0", "0.25", "0.5", "0.3", "0.1", "0.1", "-0.1"],
"version": 1614837436798
},
{
"_id": "A000005934",
"date": [915120000000, 923587200000, 941731200000, 949593600000, 953222400000, 956851200000, 962121600000, 967737600000, 970761600000, 989510400000, 999187200000, 1000742400000, 1005235200000, 1039104000000, 1046966400000, 1054828800000, 1133798400000, 1141747200000, 1150300800000, 1155052800000, 1160496000000, 1165939200000, 1173801600000, 1181664000000, 1215532800000, 1224000000000, 1226419200000, 1228838400000, 1232467200000, 1236700800000, 1239120000000, 1242144000000, 1302624000000, 1310486400000, 1320768000000, 1323792000000, 1341936000000, 1367942400000, 1384272000000, 1402416000000, 1410278400000, 1458057600000],
"value": ["3", "2.5", "3", "3.25", "3.5", "3.78", "4.25", "4.5", "4.78", "4.5", "4.25", "3.75", "3.25", "2.78", "2.5", "2", "2.25", "2.5", "2.75", "3", "3.25", "3.5", "3.75", "4", "4.25", "3.75", "3.25", "2.5", "2", "1.5", "1.25", "1", "1.25", "1.5", "1.25", "1", "0.75", "0.5", "0.25", "0.15", "0.05", "0"],
"version": 1614837436548
},
......
]
Our typical use case is to look up value based on _id and date, so I had to do something like this.
def get_value_from_mongo(id_: str, date: datetime.date) -> float:
result = db.indicators.find_one({"_id": _id}, {"value": 1, "date": 1})
date_list = list(map(str, result["date"]))
price_list = list(map(str, result["value"]))
dt = date.strftime("%s000")
price = float(price_list[date_list.index(dt)])
return price
This is hopelessly inefficient because the whole array is scanned each time I want to retrieve a single value. Maybe I could do a binary search, but date is not guaranteed to be sorted and I don't want to rely on that behavior.
Are there any MongoDB operators I can use to speed up the query?

Answer 1

A first possibility is to focus on the lookup: create an index on dates array
Which comes at the sake of a slower write.
In below execution plan you can see the index is used (you should benchmark if it brings that of an improvement)
> db.indicators.explain().find({dates: '1.1'})
{
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "dummy.indicators",
"indexFilterSet" : false,
"parsedQuery" : {
"dates" : {
"$eq" : "1.1"
}
},
"queryHash" : "4204704C",
"planCacheKey" : "1DBFE945",
"winningPlan" : {
"stage" : "FETCH",
"inputStage" : {
"stage" : "IXSCAN",// <------
"keyPattern" : {
"dates" : 1
},
"indexName" : "dates_1",
"isMultiKey" : true,
"multiKeyPaths" : {
"dates" : [
"dates"
]
},
"isUnique" : false,
"isSparse" : false,
"isPartial" : false,
"indexVersion" : 2,
"direction" : "forward",
"indexBounds" : {
"dates" : [
"[\"1.1\", \"1.1\"]"
A second possibility is to focus on retrieving the minimal data possible
With hint that bottleneck is not the date lookup but the data transfer
Thus this does not improve the lookup (given you "iterate" your array on db side instead of application code side).
You can make it with the use of
the positional operator
the projection as second argument in find with mongo >= 4.4
db.indicators.remove({})
db.indicators.insert([{_id: '0', dates: [1, '1.1', 2], prices: [1,2,3]}])
fetch = date => {
print(date)
res = db.indicators.find(
{
dates: {
$elemMatch: {
$in: [Number(date), String(date)]
}
}
},
{
'prices.$': 1 // <<--------
}
).toArray()
printjson(res)
}
fetch(2) // [ { "_id" : "0", "prices" : [ 3 ] } ]
fetch('1.1') // [ { "_id" : "0", "prices" : [ 2 ] } ]
Obviously you can compose 1 and 2, but I would have a try with just 2 to avoid creating an index