I have a MongoDB cluster running on Atlas. We connect each client to MongoDB by passing an appName with that we know who connects to the cluster and we can identify them.
At some points, we get a huge connection spike of unknown connections and we can't identify them.
If an alert happens we identify the unknown connections with this script:
const currentActivities = await db.admin().command({ currentOp: 1, $all: 1 });
const activeConnections: Record<string, number> =
currentActivities.inprog.reduce(
(acc: Record<string, number>, curr: Record<string, number>) => {
const appName = curr.appName ? curr.appName : 'Unknown';
acc[appName] = (acc[appName] || 0) + 1;
acc['TOTAL_CONNECTION_COUNT']++;
return acc;
},
{ TOTAL_CONNECTION_COUNT: 0 }
);
I tried analyzing the logs and I see a large number of connections from private IPs. These private IPs have (most of the time) an app name set. For example, they are MongoDB CPS Module.
It is not super straightforward to analyze the logs so I am not sure which IPs the UNKNOWN connections are. I only count the connections by IP.
Did anybody have a similar problem and some suggestions on how to tackle that?
Thank you 👋🏽
Sandro
Related
I am querying my database using graphQL. I am stuck on writing the graphQL resolvers to my database, using knexjs.
My problem is I want that a query or mutation to use only 1 database connection (correct me if this is wrong, but I really think this is true).
For instance, the call to the server
query {
post {
author
}
}
should use 2 database calls for the post and author fields, done in a single connection to the database.
I think transactions are the way to go, and I implemented resolvers using transactions (here is a toy example):
const trxProvider = knex.transactionProvider();
const resolvers = {
Query: {
post: async () => {
const trx = await trxProvider();
let res = await trx('posts')
return res
},
author: async () => {
const trx = await trxProvider();
let res = await trx('authors')
return res
}
}
}
How do I properly resolve this transaction? For instance, how would I call trx.commit() when a query/mutation has completed so the connection does not idle?
Are transactions the correct approach / What knex functionality should I use so that a single database connection is used for a query +mutation?
Answering any of these questions is great. Thanks!
Connection pooling is the preferred approach. Transactions are best used to wrap multiple database writes so that all the writes can be committed or rolled back together. This avoids inconsistent writes. I've found no advantage to using transactions for reads.
I've published a tutorial that covers a lot of what you need to do with knex and GraphQL. Hopefully you'll find it helpful.
ive created M0 Cluster Sandbox via Mongo Atlas. It is working pretty nice. But I want to use transactions with it. And I've read that to use transactions I need to have a replica set.
In the Atlas it seems like my DB have a replicaSet already (i didn't do anything). So how i can connect to that replica set?
My current connection link is mongodb+srv://admin:password#de.xxx.mongodb.net/db?retryWrites=true&w=majority
Thanks in advance!
It should be enough with passing the connection string when you create your MongoClient object:
const { MongoClient, ServerApiVersion } = require('mongodb');
const uri = "your_string_connection";
const client = new MongoClient(uri, { options-you-need });
client.connect(err => {
const collection = client.db("test").collection("devices");
// perform actions on the collection object
client.close();
});
This code was copy-pasted from atlas cluster instructions to connect to the cluster: Connect your applications -> check Include full driver code example.
In my (greatly simplified) model I have users, accounts and account_types. Each user can have multiple accounts of each account_type. When an account of type TT is created I'm updating the "users" field of that object so it keeps the users which have accounts of that types, and the number of such accounts they have.
users: {
some fields
},
accounts: {
userID: UU,
type: TT
},
account_type:
users: { UU: 31 }
}
I use the onCreate and onDelete cloud triggers for accounts to update the account_type object. Since multiple accounts can be created simultaneously I have to use transactions:
exports.onCreateAccount = functions.firestore
.document('accounts/{accountID}')
.onCreate((account, context) => {
const acc_user = account.data().userID;
const acc_type = account.data().type;
return admin.firestore().runTransaction(transaction => {
// This code may get re-run multiple times if there are conflicts.
const accountTypeRef = admin.firestore().doc("account_types/"+acc_type);
return transaction.get(accountTypeRef).then(accTypeDoc => {
var users = accTypeDoc.data().users;
if (users === undefined) {
users = {};
}
if (users[acc_user] === undefined) {
users[acc_user] = 1;
} else {
users[acc_user]++;
}
transaction.update(accountTypeRef, {users: users});
return;
})
})
.catch(error => {
console.log("AccountType create transaction failed. Error: "+error);
});
});
In my tests I'm first populating the database with some data so I'm also adding a user and 30 accounts of the same type. With the local emulator this works just fine and at the end of the addition I see that the account_type object contains the user with the counter at 30. But when deployed to Firebase and running the same functions the counter gets to less than 30. My suspicion is that since Firebase is much slower and transactions take longer, more of them are conflicted and fail and eventually don't execute at all. The transaction failure documentation (https://firebase.google.com/docs/firestore/manage-data/transactions) says:
"The transaction read a document that was modified outside of the transaction. In this case, the transaction automatically runs again. The transaction is retried a finite number of times."
So my questions:
What does "finite" mean?
Any way to control this number?
How can I make sure my transactions are executed at some point and don't get dropped like that so my data is consistent?
Any other idea as to why I'm not getting the correct results when deployed to the cloud?
What does "finite" mean?
It's the opposite of "unlimited". It will retry no more than a set number of times.
Any way to control this number?
Other than modifying the source code of the SDK, no. The SDK itself advertise a specific number, as it might change.
How can I make sure my transactions are executed at some point and don't get dropped like that so my data is consistent?
Detect the error and retry in your app. If you aren't seeing the transaction fail with an error, then nothing went wrong.
Any other idea as to why I'm not getting the correct results when deployed to the cloud?
Since we can't see what exactly you're doing to trigger the function, and have no specific expected results to compare to, it's not really possible to say.
So basically, I want building a test app where I need to enter server credentials in order to connect to an ElasticSearch server and I want this connection to be only available for the duration of the session.
I thought this could be managed via Backbone.Model but I'm not too sure how to handle it. Do I need a Collection as well. Do I store this in the Session or locaStorage?
Page 1
Form with protocol, host, post, username, password
Backbone.View.extend
...
events: {
'submit #connection-form' : 'connect'
},
...
connect: function(){
console.log('Connecting...');
protocol = $("#protocol").val();
host = $("#host").val();
port = $("#port").val();
user = $("#username").val();
password = $("#password").val();
App.connection = new Connection({ protocol: protocol, host: host, port: port, user: user, password: password})
self = this
App.connection.connect(function(response){
if (response.status == 200) {
App.connection.set('name', response.name)
App.router.navigate('dashboard', {trigger: true});
} else {
$(self.el).prepend(_.template(noticeTpl, { type: 'danger', message: "Could not connect to the server." }))
}
});
return false;
}
Page 2
List of indexes on my ElasticSearch server
I need to store what have been submitted on page one across all the session in order to be able to query the server at any time.
No reason you'll need a collection since you're only dealing with a single object. I also don't see any reason to use localStorage unless you're storing a lot of data, which you aren't, or want the app to be usable offline, which doesn't make any sense here.
Using sessionStorage is pretty straightforward:
sessionStorage.setItem('user', JSON.stringify(user));
var obj = JSON.parse(sessionStorage.getItem('user'));
I've written a small Socket.IO server, which works fine, I can connect to it, I can send/receive messages, so everything is working ok. Just the relevant part of the code is presented here:
var RedisStore = require('socket.io/lib/stores/redis');
const pub = redis.createClient('127.0.0.1', 6379);
const sub = redis.createClient('127.0.0.1', 6379);
const store = redis.createClient('127.0.0.1', 6379);
io.configure(function() {
io.set('store', new RedisStore({
redisPub : pub,
redisSub : sub,
redisClient : store
}));
});
io.sockets.on('connection', function(socket) {
socket.on('message', function(msg) {
pub.publish("lobby", msg);
});
/*
* Subscribe to the lobby and receive messages.
*/
var sub = redis.createClient('127.0.0.1', 6379);
sub.subscribe("lobby");
sub.on('message', function(channel, msg) {
socket.send(msg);
});
});
Here, I'm interested in the problem where certain client is subscribed to a different room, which is why I'm also using the sub Redis variable inside each socket connection: because each client can be subscribed to a different room and can receive messages from there. I'm not entirely sure whether the code above is ok, so please let me know if I need to do anything else than define the sub Redis connection inside the Socket.IO connection: this also means that a new Redis connection is spawned for each client connecting serving his messages from the subsribed room? I guess this is quite an overhead, so I would like to solve it anyway possible?
Thank you
Both node.js and redis are very good at handling lots of connections (thousands is no problem), so what you're doing is fine.
As a side note, you will want to look into upping your file descriptor limits if you do intend on supporting thousands of connections.