below is the bash script and I want to pass the value of 'a' from for loop to mongoDB command but I am getting $a is not defined error. How to pass the value to mongo command.
{ "account_id" : $a } is the part of the below command.
#!/usr/bin/bash
for a in `cat /tmp/account.txt`;
do
mongo --authenticationDatabase admin -u 'xxxxxx' -p 'xxxxxxxxx' --quiet --eval 'db.executions.find({ $and : [ { "account_id" : $a }, { "status" : "running" }, { "created_at" : { "$gt" : ISODate("2022-09-01T00:00:00.000Z") } }, { "created_at" : { "$lt" : ISODate("2022-10-01T00:00:00.000Z") } } ] } ,{"account_id":1,"created_at":1} ).count()' selfservice_production;
done
account.txt
35331
35332
35333
35334
35335
35336
35337
35338
35339
35340
need to add the variable in single quotes. Its working for me.
{ "account_id" : "'$a'" } is what I changed.
#!/usr/bin/bash
for a in `cat /tmp/account.txt`
do
mongo --authenticationDatabase admin -u "xxxx" -p "xxxxxxxxxx" --quiet selfservice_production --eval 'db.executions.find({ $and : [ { "account_id" : "'$a'" }, { "status" : "running" }, { "created_at" : { "$gt" : ISODate("2022-08-01T00:00:00.000Z") } }, { "created_at" : { "$lt" : ISODate("2022-09-01T00:00:00.000Z") } } ] } ,{"account_id":1,"created_at":1} ).count()'
done
Reference: Use variables with --eval in mongodb
Related
Getting this error, when I am using this query:
db2.collection('candata')
.aggregate([{$sort:{time:-1}},{$group:{_id:{batteryId:"$batteryId"},soctime:{$first:"$time"},GPSStatus:{$first:"$GPSStatus"},CANStatus:{$first:"$CANStatus"},soc: { $first : "$socpercentage" },charging_status: { $first : "$charging_status" },status: { $first : "$status" }}},{$sort:{'_id.batteryId':1}},{$lookup:{from : "swap_table",localField : "_id.batteryId",foreignField:"batteryId",as:"swap"}},{$lookup:{from : "battery_status_table",localField : "_id.batteryId",foreignField:"batteryId",as:"battery_statustb"}}])
What change do I need to do? I do not want to change the query, so how to increase the limit of this, or I need to go some data archived method?
Add allowDiskUse and set it to true at the end of your query so it becomes like the following:
db2.collection('candata')
.aggregate([{$sort:{time:-1}},{$group:{_id:{batteryId:"$batteryId"},soctime:{$first:"$time"},GPSStatus:{$first:"$GPSStatus"},CANStatus:{$first:"$CANStatus"},soc: { $first : "$socpercentage" },charging_status: { $first : "$charging_status" },status: { $first : "$status" }}},{$sort:{'_id.batteryId':1}},{$lookup:{from : "swap_table",localField : "_id.batteryId",foreignField:"batteryId",as:"swap"}},{$lookup:{from : "battery_status_table",localField : "_id.batteryId",foreignField:"batteryId",as:"battery_statustb"}}]).allowDiskUse(true)
Add allowDiskUse with second params of aggregate function, like this:
agg = [{$sort:{time:-1}},{$group:{_id:{batteryId:"$batteryId"},soctime:{$first:"$time"},GPSStatus:{$first:"$GPSStatus"},CANStatus:{$first:"$CANStatus"},soc: { $first : "$socpercentage" },charging_status: { $first : "$charging_status" },status: { $first : "$status" }}},{$sort:{'_id.batteryId':1}},{$lookup:{from : "swap_table",localField : "_id.batteryId",foreignField:"batteryId",as:"swap"}},{$lookup:{from : "battery_status_table",localField : "_id.batteryId",foreignField:"batteryId",as:"battery_statustb"}}]
db2.collection('candata').aggregate(agg, allowDiskUse=True)
you can add allowDiskUse with second of aggregate function eg:
db.getCollection("loans_unique").aggregate([{
"$group": {
"_id": {
phone: "$phone",
address: "$address",
title: "$title"
},
"count": {
"$sum": 1
}
}
},
{
$sort: {
"count": -1
}
},
{
$match: {
"count": {$gt: 1}
}
}
], {
allowDiskUse: true
})
I'm importing a very large csv file to mongodb which follows the following format:
"zzzàms#hotmail.com","12071988"
"zzzг ms#hotmail.com","12071988"
"zzпїѕпїѕmmbbii2#bk.ru","MA15042002"
"zzпїѕпїѕmmbbii2#list.ru","MA15042002"
"zzпїѕпїѕmmbbii2#rambler.ru","MA15042002"
"zzпїѕпїѕmmbbii2#yandex.ru","MA15042002"
However, I am not certain how many fields / columns will follow, after the email field.
I have imported using this command:
mongoimport -d emails -c second --file all.csv --type csv --fields email, number
However, any fields / columns after number field is issued a default value of 'field2', 'field3' and so on.
{ "_id" : ObjectId("5a5cd95e598f1e910d353e3b"), "email" : "00-amber-00#embarqmail.com", " number" : "number1", "field2" : "number2" }
How can I put anything after the number field in the same column, so it's classified as 'number'?
Sometimes, one entry has maybe 40 columns.
I don't wish to modify the csv file unless it is really necessary.
Sorry, english is not first language, thanks.
you can use Unix commands like awk to parse the line to json as per the logic and stdin to mongoimport
sample file
saravana#ubuntu:~$ cat sample-doc.txt
"zzzàms#hotmail.com","12071988"
"zzzг ms#hotmail.com","12071988"
"zzпїѕпїѕmmbbii2#bk.ru","MA15042002"
"zzпїѕпїѕmmbbii2#list.ru","MA15042002"
"zzпїѕпїѕmmbbii2#rambler.ru","MA15042002","34534"
"zzпїѕпїѕmmbbii2#yandex.ru","MA15042002","1232434","3435435","53534"
awk to convert json, email followed by numbers
saravana#ubuntu:~$ cat sample-doc.txt | awk 'BEGIN{FS=","}{print "{ email :" $1 ", numbers : [ " substr($0,length($1)+2) " ] } " }'
{ email :"zzzàms#hotmail.com", numbers : [ "12071988" ] }
{ email :"zzzг ms#hotmail.com", numbers : [ "12071988" ] }
{ email :"zzпїѕпїѕmmbbii2#bk.ru", numbers : [ "MA15042002" ] }
{ email :"zzпїѕпїѕmmbbii2#list.ru", numbers : [ "MA15042002" ] }
{ email :"zzпїѕпїѕmmbbii2#rambler.ru", numbers : [ "MA15042002","34534" ] }
{ email :"zzпїѕпїѕmmbbii2#yandex.ru", numbers : [ "MA15042002","1232434","3435435","53534" ] }
saravana#ubuntu:~$
mongoimport using stdin
saravana#ubuntu:~$ cat sample-doc.txt | awk 'BEGIN{FS=","}{print "{ email :" $1 ", numbers : [ " substr($0,length($1)+2) " ] } " }' | mongoimport --type json --db test --collection emailnos -v
2018-01-17T09:58:11.559+0530 reading from stdin
2018-01-17T09:58:11.559+0530 using fields:
2018-01-17T09:58:11.561+0530 connected to: localhost
2018-01-17T09:58:11.561+0530 ns: test.emailnos
2018-01-17T09:58:11.561+0530 connected to node type: standalone
2018-01-17T09:58:11.561+0530 using write concern: w='1', j=false, fsync=false, wtimeout=0
2018-01-17T09:58:11.561+0530 using write concern: w='1', j=false, fsync=false, wtimeout=0
2018-01-17T09:58:11.726+0530 imported 6 documents
collection
> db.emailnos.find()
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da90"), "email" : "zzzàms#hotmail.com", "numbers" : [ "12071988" ] }
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da91"), "email" : "zzпїѕпїѕmmbbii2#list.ru", "numbers" : [ "MA15042002" ] }
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da92"), "email" : "zzпїѕпїѕmmbbii2#rambler.ru", "numbers" : [ "MA15042002", "34534" ] }
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da93"), "email" : "zzпїѕпїѕmmbbii2#yandex.ru", "numbers" : [ "MA15042002", "1232434", "3435435", "53534" ] }
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da94"), "email" : "zzzг ms#hotmail.com", "numbers" : [ "12071988" ] }
{ "_id" : ObjectId("5a5ed0dbead4f5f7ae68da95"), "email" : "zzпїѕпїѕmmbbii2#bk.ru", "numbers" : [ "MA15042002" ] }
>
I'm trying to update an element inside an array using the $ update operator. The document contains 2 array fields and I have to query both to select the correct document.
The collection is called locations and the document looks like this:
{
"_id" : "XqEQYpitGFG3nnf3C",
"wallpapers" : [
{
"_metadata" : {
"master" : "vwb22W4MhkqtvAp89",
"isMaster" : false
},
"role" : "master",
"_id" : ""
},
{
"_metadata" : {
"master" : "vwb22W4MhkqtvAp89",
"isMaster" : false
},
"role" : "clone",
"_id" : ""
},
{
"_metadata" : {
"master" : "vwb22W4MhkqtvAp89",
"isMaster" : false
},
"role" : "pod",
"_id" : ""
}
],
"ancestors" : [
"vwb22W4MhkqtvAp89",
"tqzqfum9uMs47xcHW",
"b4d83aqTkq6TGvXts",
"XqEQYpitGFG3nnf3C"
]
}
The update operator looks like this:
db.getCollection('locations').update(
{
"ancestors": "b4d83aqTkq6TGvXts",
"wallpapers": {
"$elemMatch": {
"role": "clone",
"_metadata.master": "vwb22W4MhkqtvAp89"
}
}
},
{
"$set": {
"wallpapers.$": {
"_id": "D33WNZh7Bg4itPdhk",
"_metadata": {
"master": "b4d83aqTkq6TGvXts",
"isMaster": false
},
"role": "clone"
}
}
}
)
So I would like to have the element in the wallpapers array replaced. However, the result I get is:
{
"_id" : "XqEQYpitGFG3nnf3C",
"wallpapers" : [
{
"_metadata" : {
"master" : "vwb22W4MhkqtvAp89",
"isMaster" : false
},
"role" : "master",
"_id" : ""
},
{
"_metadata" : {
"master" : "vwb22W4MhkqtvAp89",
"isMaster" : false
},
"role" : "clone",
"_id" : ""
},
{
"_id" : "D33WNZh7Bg4itPdhk",
"_metadata" : {
"master" : "b4d83aqTkq6TGvXts",
"isMaster" : false
},
"role" : "clone"
}
],
"ancestors" : [
"vwb22W4MhkqtvAp89",
"tqzqfum9uMs47xcHW",
"b4d83aqTkq6TGvXts",
"XqEQYpitGFG3nnf3C"
]
}
So it replaces the wrong element.
It seems that the position .$ refers to is the one from the selector of the ancestors field.
Is this a bug or a limitation? Is there a solution (e.g. anything like .$1 and .$2 ?
I'm using MongoDB 3.2.6.
In your update operation query, use the dot notation as:
db.getCollection('locations').update(
{
"ancestors": "b4d83aqTkq6TGvXts",
"wallpapers.role": "clone", // <--- dot notation
"wallpapers._metadata.master": "vwb22W4MhkqtvAp89" // <-- dot notation
},
{
"$set": {
"wallpapers.$": {
"_id": "D33WNZh7Bg4itPdhk",
"_metadata": {
"master": "b4d83aqTkq6TGvXts",
"isMaster": false
},
"role": "clone"
}
}
}
)
I'm trying to find a document containing a specific term in an array of strings.
I have a schema like this:
{
"pages": {
"mappings":{
"site":{
"properties":{
"urls":{"type":"string"}
}
}
}
}
}
And the following data indexed on it:
% curl -XPOST 'http://local.dev:9200/pages/site/_search?pretty
{
...
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [ {
"_index" : "pages",
"_type" : "site",
"_id" : "ae634fea-878f-42ca-8239-c67cca007a38",
"_score" : 1.0,
"_source":{ "urls":["https://github.com/fulano","http://fulano.com"] }
}
}
I'm trying to search for sites whose urls array contains a specific url, but I can't make it work. I tried using terms - exactly as described here but I never get any results:
% curl -XPOST 'http://local.dev:9200/pages/site/_search?pretty' -d '
{
"query": {
"filtered": {
"filter": {
"term": { "urls": "https://github.com/fulano" }
}
}
}
}'
{
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
Using terms (that gets expanded into a series of bool operations by elastic):
% curl -XPOST 'http://local.dev:9200/pages/site/_search?pretty' -d '
{
"query": {
"terms" : {
"urls" : ["https://github.com/fulano"]
}
}
}'
{
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}
I'm guessing this is something really silly, but I can't spot the problem. :(
This is the problem with the analyzer you are using. You need to use not_analyzed or keyword tokenizer as outlined here.
I am trying to join 2 elasticsearch indices by using terms filter lookup. I referred to http://www.elasticsearch.org/blog/terms-filter-lookup/ and http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-terms-filter.html. These Examples lookup on an array of fields like "followers" : ["1", "3"] and join works fine for similar data.
My requirement is to join with a field inside an array of objects. When I extend the above example to include an array of objects, my query fails.
Following is the sample data:
PUT /users/user/2 {
"followers" : [
{
"userId":"1",
"username":"abc",
"location":"xyz"
},
{
"userId":"3",
"username":"def",
"location":"xyz"
}
}
]
}
PUT /tweets/tweet/1 {
"user" : "2"
}
PUT /tweets/tweet/2 {
"user" : "1"
}
I am now trying to find tweets that are created by followers of user 2
POST /tweets/_search {
"query" : {
"filtered" : {
"filter" : {
"terms" : {
"user" : {
"index" : "users",
"type" : "user",
"id" : "2",
"path" : "followers.userId"
},
"_cache_key" : "user_2_friends"
}
}
}
}
}
My search results are 0 for above query. I tried 2 other approaches as well 1)declare the followers object as a nested object during mapping and use "nested" in the query, 2)tried to add a match query for followers.userId after giving path as "followers". None yielded results.
Does terms filter lookup support array of objects? Any pointers to solving my problem would be of great help
What you're trying to do worked for me, unless I'm missing something. What version of Elasticsearch are you using? I'm using 1.3.4.
So I created both indices and added the docs you have listed:
curl -XPUT "http://localhost:9200/users"
curl -XPUT "http://localhost:9200/users/user/2 " -d '
{
"followers" : [
{
"userId":"1",
"username":"abc",
"location":"xyz"
},
{
"userId":"3",
"username":"def",
"location":"xyz"
}
]
}'
curl -XPUT "http://localhost:9200/tweets"
curl -XPUT "http://localhost:9200/tweets/tweet/1 " -d'
{
"user" : "2"
}'
curl -XPUT "http://localhost:9200/tweets/tweet/2 " -d'
{
"user" : "1"
}'
then ran your search query:
curl -XPOST "http://localhost:9200/tweets/_search " -d'
{
"query": {
"filtered": {
"filter": {
"terms": {
"user": {
"index": "users",
"type": "user",
"id": "2",
"path": "followers.userId"
},
"_cache_key": "user_2_friends"
}
}
}
}
}'
and got back this result:
{
"took": 2,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 1,
"hits": [
{
"_index": "tweets",
"_type": "tweet",
"_id": "2",
"_score": 1,
"_source": {
"user": "1"
}
}
]
}
}
Here is the code I used:
http://sense.qbox.io/gist/4a2a2d77d0b6f4502ff6c5022b268acfa65ee6d2
Clear the indices if you have any
curl -XDELETE "http://example.com:9200/currencylookup/"
curl -XDELETE "http://example.com:9200/currency/"
Create the lookup table
curl -XPUT http://example.com:9200/currencylookup/type/2 -d '
{ "conv" : [
{ "currency":"usd","username":"abc", "location":"USA" },
{ "currency":"inr", "username":"def", "location":"India" },
{ "currency":"IDR", "username":"def", "location":"Indonesia" }]
}'
Lets put some dummy docs
curl -XPUT "http://example.com:9200/currency/type/USA" -d '{ "amount":"100", "currency":"usd", "location":"USA" }'
curl -XPUT "http://example.com:9200/currency/type/JPY" -d '{ "amount":"50", "currency":"JPY", "location":"JAPAN" }'
curl -XPUT "http://example.com:9200/currency/type/INR" -d '{ "amount":"50", "currency":"inr", "location":"INDIA" }'
curl -XPUT "http://example.com:9200/currency/type/IDR" -d '{ "amount":"30", "currency" : "IDR", "location": "Indonesia" }'
Time to check the output
curl http://example.com:9200/currency/_search?pretty -d '{
"query" : {
"filtered" : {
"filter" : {
"terms" : {
"currency" : {
"index" : "currencylookup",
"type" : "type",
"id" : "2",
"path" : "conv.currency"
},
"_cache_key" : "currencyexchange"
}
}
}
}
}'
Results
# curl http://example.com:9200/currency/_search?pretty -d '{
"query" : {
"filtered" : {
"filter" : {
"terms" : {
"currency" : {
"index" : "currencylookup",
"type" : "type",
"id" : "2",
"path" : "conv.currency"
},
"_cache_key" : "currencyexchange"
}
}
}
}
}'
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [ {
"_index" : "currency",
"_type" : "type",
"_id" : "INR",
"_score" : 1.0,
"_source":{ "amount":"50", "currency":"inr", "location":"INDIA" }
}, {
"_index" : "currency",
"_type" : "type",
"_id" : "USA",
"_score" : 1.0,
"_source":{ "amount":"100", "currency":"usd", "location":"USA" }
} ]
}
}
Conclusion
Capital letters are culprit here.
You can see 'IDR' is in caps so the match is failed for it and 'JPY' is not in look up even if it was there it would not have got matched because it is in caps.
cross matching values must be in small letters or numbers like
eg:
abc
1abc