I'm creating a Next.js application and I noticed that many developers cache the MongoDB connection. For example
let cachedClient = null;
let cachedDb = null;
export async function connectToDatabase() {
if (cachedClient && cachedDb) {
return {
client: cachedClient,
db: cachedDb,
};
}
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true,
};
let client = new MongoClient(MONGODB_URI, opts);
await client.connect();
let db = client.db(MONGODB_DB);
cachedClient = client;
cachedDb = db;
return {
client: cachedClient,
db: cachedDb,
};
}
or
let cached = global.mongoose
if (!cached) {
cached = global.mongoose = { conn: null, promise: null }
}
async function dbConnect () {
if (cached.conn) {
return cached.conn
}
if (!cached.promise) {
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true,
bufferCommands: false,
bufferMaxEntries: 0,
useFindAndModify: true,
useCreateIndex: true
}
cached.promise = mongoose.connect(MONGODB_URI, opts).then(mongoose => {
return mongoose
})
}
cached.conn = await cached.promise
return cached.conn
}
I've never seen that in Express apllications so I have 2 questions:
Why is caching database connection such a common thing in Next.js while I've never seen that in Express.js. What's he reason for that? How does it work? And is it worth it?
As you can see in the examples above some developer use useual let-variables while some other developer use global variables. What's the difference and which is the better solution?
In Next.Js you can cache some variables and mongo connection is one of them. It will significantly improve the response time of your application because the first call to your page will make all calls to mongo to estabilish a connection and it can take more than 2 seconds just to do this, after you have this connection stabilished you can reuse in future calls to that same page (in cases of local cache) and it'll dispense those 2 seconds spent on creating connection (resulting in a WAY faster response to your user).
eg:
1° Request to your page:
2300ms spent to get a response //Had to establish a new connection with mongo
2° Request to your page:
230ms spent to get a response //Used cached connection
3° Request to your page:
180ms spent to get a response //Used cached connection
4° Request to your page:
210ms spent to get a response //Used cached connection
...
The difference between the global cache and "let" cache is which functions will use the same connection. Depending in what your application does you can use global cache and it will prevent from every function in your app create your own connection with mongo and spent that 2 seconds I've mentioned.
Related
Whenver I try to use useSubscription to get some data from the server, it fires two times (eg. it prints hello twice). It messes up my application because the code after useSubscription runs two times each time. The spring server sends the data only once as there is only one printing on the server side. Any idea why this is happening?
my frontend:
const stompClient = useStompClient();
useSubscription("/topic/getmarkedcells", (message) => {
console.log("hello")
});
const sendMarkedCells = (markedCells) => {
if (stompClient) {
stompClient.publish({
destination: "/app/markedcells",
body: JSON.stringify(markedCells),
});
}
};
my backend:
#MessageMapping("/markedcells")
#SendTo("/topic/getmarkedcells")
public MarkedCells getMarkedCells(#RequestBody MarkedCells markedCells){
System.out.println(markedCells);
return markedCells;
}
I just don't why this is happening.
I am developing a flutter application in which I am implementing hive database for caching data.
I have added both hive and hive_flutter packages.
I am getting data from APIs and store that to hive to update data, It works fine when I used app connected to internet but didn't works when I try to read while being offline. Here is the code of my API method I am calling to get data:
static Future<List<UserPost>> getPosts() async {
//I call my API in try block, if its successful, I update the data in hive
List<UserPost> posts = [];
Hive.openBox(Constants.APIDATA_BOX);
try {
var response = await http.get(Uri.parse('https://jsonplaceholder.typicode.com/posts'),);
if (response.statusCode == 200) {
//Clear hive box from old data
Hive.box(Constants.APIDATA_BOX).clear();
Hive.box(Constants.APIDATA_BOX).put(Constants.API_DATA,jsonDecode(response.body));
}
} catch (e) {
print('You are not connected to internet');
}
//I am getting data here from hive database and it works fine while connected to internet
var listMaps =await Hive.box(Constants.APIDATA_BOX).get(Constants.API_DATA, defaultValue: []);
posts = listMaps.map<UserPost>((map) {
//Here flow stucked whenever working offline,
//Data is also available but here conversion cause error, I have tried many way but fails.
return UserPost.fromMap(map);
}).toList();
return posts;
}
I don't why I am getting error, I have tried many conversion ways here but all works while being online. Any help will be highly apprerciated.
I think I've understood the error but you should explain better which type of error you're having.
Anyway pay attention to the operations on Hive, which are often async, for example Hive.openBox(Constants.APIDATA_BOX);.
So when you have internet connection, you have to await for the response and Hive has time to open the box, otherwise it will throw an error so, considering the futures, you should do this:
static Future<List<UserPost>> getPosts() async {
List<UserPost> posts = [];
await Hive.openBox(Constants.APIDATA_BOX);
try {
var response = await http.get(Uri.parse('https://jsonplaceholder.typicode.com/posts'),);
if (response.statusCode == 200) {
//Clear hive box from old data
await Hive.box(Constants.APIDATA_BOX).clear();
await Hive.box(Constants.APIDATA_BOX).put(Constants.API_DATA,jsonDecode(response.body));
}
} catch (e) {
print('You are not connected to internet');
}
var listMaps = await Hive.box(Constants.APIDATA_BOX).get(Constants.API_DATA, defaultValue: []);
posts = listMaps.map<UserPost>((map) {
return UserPost.fromMap(map);
}).toList();
return posts;
}
Note that await Hive.put() in a normal box is not strictly necessary, as explained in the docs
I am developing a typical MERN application and I've completed the authentication cycle. My NodeJS/Express back-end uses 'express-session' and 'connect-mongodb-connection' to create and handle sessions. The React front-end uses 'axios' for communicating with the API. The authentication cycle works on all browsers except Chrome. For all other browsers, a session is successfully created in MongoDB, cookies are set in the browser and I am successfully logged into a session.
But when testing this with Chrome, everything works perfectly except for the part where cookies are set. I've tested this rigorously over the span of a day and I can trace the cookie to the point where it's sent from the back-end. But Chrome refuses to save the cookie.
Here is my code for maintaining sessions:
server/app.js
var store = new MongoDBStore({
uri: DB,
collection: 'sessions'
});
// Catch errors
store.on('error', function (error) {
console.log(error);
});
app.use(require('express-session')({
secret: process.env.SESSION_SECRET,
saveUninitialized: false, // don't create session until something stored
resave: false, //don't save session if unmodified
store: store,
cookie: {
maxAge: parseInt(process.env.SESSION_LIFETIME), // 1 week
httpOnly: true,
secure: !(process.env.NODE_ENV === "development"),
sameSite: false
},
}));
//Mongo Session Logic End
app.enable('trust proxy');
// 1) GLOBAL MIDDLEWARES
// Implement CORS
app.use(cors({
origin: [
process.env.CLIENT_ORIGINS.split(',')
],
credentials: true,
exposedHeaders: ['set-cookie']
}));
The CLIENT_ORIGINS are set to the https://localhost:3000 and http://localhost:3000 where my React client runs.
Some things I've tried:
Trying all combinations of secure:true & secure:false with all combinations of sameSite:false & sameSite:'strict'
Setting domain to NULL or empty string
Trying to change path randomly
Here's my code for setting the cookies on login at the back-end:
exports.signIn = async (req, res, next) => {
const { email, password } = req.body;
if (signedIn(req)) {
res.status(406).json('Already Signed In');
return;
}
const user = await User.findOne({ email: email });
if (!user) {
res.status(400).json('Please enter a correct email.');
return;
}
if (!(await user.matchPassword(password))) {
res.status(400).json('Please enter a correct password.');
return;
}
req.session.userId = user.id;
res.status(200).json({ msg: 'Signed In', user: user });
};
This is the generic request model I use for calling my API from React using Axios:
import axios from "axios";
import CONFIG from "../Services/Config";
axios.defaults.withCredentials = true;
const SERVER = CONFIG.SERVER + "/api";
let request = (method, extension, data = null, responseTypeFile = false) => {
//setting up headers
let config = {
headers: {
"Content-Type": "application/json",
},
};
// let token = localStorage["token"];
// if (token) {
// config.headers["Authorization"] = `Bearer ${token}`;
// }
//POST Requests
if (method === "post") {
// if (responseTypeFile) {
// config['responseType'] = 'blob'
// }
// console.log('request received file')
// console.log(data)
return axios.post(`${SERVER}/${extension}`, data, config);
}
//PUT Requests
else if (method === "put") {
return axios.put(`${SERVER}/${extension}`, data, config);
}
//GET Requests
else if (method === "get") {
if (data != null) {
return axios.get(`${SERVER}/${extension}/${data}`, config);
} else {
return axios.get(`${SERVER}/${extension}`, config);
}
}
//DELETE Requests
else if (method === "delete") {
if (data != null) {
return axios.delete(`${SERVER}/${extension}/${data}`, config);
} else {
return axios.delete(`${SERVER}/${extension}`, config);
}
}
};
export default request;
Some more things that I have tested:
I have double checked that credentials are set to true on both sides.
I have made sure that the authentication cycle is working on other browsers.
I have also made sure that the authentication cycle works on Chrome when I run React on http instead of https
I have also added my self signed certificate into the trusted root certificates on my local machine. Chrome no longer shows me a warning but still refuses to save cookies
I have made sure that the authentication cycle works if I run an instance of Chrome with web security disabled.
I've tried to make it work by using 127.0.0.1 instead of localhost in the address bar to no avail.
No errors are logged on either side's console.
Any and all help would be appreciated
Chrome is always doing crazy stuff with cookies and localStorage...
It seems since chrome 80 chrome will reject any cookies that hasn't specifically set SameSite=None and Secure while using cross site requests. That issue, https://github.com/google/google-api-javascript-client/issues/561, is still open and being discussed there. I also think that using https while not setting Secure will also have it be rejected.
I have faced this same issue once and I have solved it by specifically set mentioned below:
document.cookie = "access_token=" + "<YOUR TOKEN>" + ";path=/;domain=."+ "<YOUR DOMAIN NAME>" +".com;secure;sameSite=none";
Make sure:
Your Path variable is set to /.
Your Domain is set to .<YOUR DOMAIN NAME>.com (NOTE: Here . dots is necessary part).
Your secure variable should be true.
Your sameSite variable should be none.
So I figured out the solution to my issue. My client-side was running on an https connection (even during development), because the nature of my project required so.
After much research, I was sure that the settings to be used for express-session were these:
app.use(require('express-session')({
secret: process.env.SESSION_SECRET,
saveUninitialized: false, // don't create session until something stored
resave: false, //don't save session if unmodified
store: store,
cookie: {
maxAge: parseInt(process.env.SESSION_LIFETIME), // 1 week
httpOnly: true,
secure: true,
sameSite: "none"
},
}));
Keep in mind that my client-side is running on an https connection even in development. However, despite using these settings, my login cycle did not work on Chrome and my cookies weren't being set.
Express session refused to send back cookies to the client, because despite having my client run on an https connection, it contacted my server on an http connection (my server was still running on an http connection in development), hence making the connection insecure.
So I added the following code to my server:
const https = require('https');
const fs = require('fs');
var key = fs.readFileSync("./certificates/localhost.key");
var cert = fs.readFileSync("./certificates/localhost.crt");
var credentials = {
key,
cert
};
const app = express();
const port = process.env.PORT || 3080;
const server = process.env.NODE_ENV === 'development' ? https.createServer(credentials, app) : app;
server.listen(port, () => {
console.log(`App running on port ${port}...`);
});
I used a self-signed certificate to run my server on an https connection during development. This along with sameSite: "none" and secure: true resolve the issue on Chrome (and all other browsers).
I am developing a website in NextJs and using MongoDB as a database. This code is what connects to the database and keeps it in cache.
import { MongoClient } from "mongodb";
let cache = {};
export default async function connect() {
if (cache?.client?.isConnected()) {
return cache;
}
const opts = {
useNewUrlParser: true,
useUnifiedTopology: true,
};
return MongoClient.connect(process.env.DATABASE_URL, opts).then((client) => {
cache = {
db: client.db("bd"),
client,
};
return {
client,
db: client.db("bd"),
};
});
}
I imagine that this code should cause only 1 connection to be created but it is creating many more connections. As in the photo below, 18 connections were created and dropped to 6 after I stopped use the site. Why are several connections being created? How do you make it just one?
This is an example of code that I am using on one of the routes to list users.
...
const { db, client } = await connect();
const { userThatMakeRequest } = req;
const { group } = req.query;
const users = await db
.collection("user")
.find(
{
roles: "team-user",
"team.id": userThatMakeRequest?.team?.id,
"group.id": group,
hasAccess: true,
},
{ projection: { password: 0 } }
)
.toArray();
res.status(200).json({
response: users || [],
});
The driver creates 1 or 2 connections to each known server for monitoring purposes. Application connections (the ones used for satisfying queries and writes) are separate.
If you create a client object and perform one query against a 3 node replica set running MongoDB 4.4, you'll end up with 7 total connections.
I have a react app, which must perform a weekly task every Monday #7:58 am. The task is setup as a separate function "notification()". And I want to use the 'CRON' package from NPM to call notification() at the appropriate time.
I have CRON wrapped inside of a function like this:
let mondayNotif = () => {
new CronJob('* 58 7 * * 2', function() {
notification()
}, null, true, 'America/Los_Angeles');
};
My question: where should I call the function mondayNotif(), to make sure that the CronJob is initiated correctly? I thought at first it must be on the backend, but the NPM package doesn't seem to support server-side. But if I call mondayNotif() on the client side, will the CronJob still happen if the site is inactive?
From what I know React JS is front end - it runs on client side. You need a server. In this case a node.js based server. Theroetically if nobody opens the website nothing will be fired up in react js. Look up how to schedule cron jobs on node.js
enter link description here
I found my own answer. But first, a few insights on CronJobs that would have helped me:
CronJobs are essentially a third-party function with an embedded clock. Once they are "initiated", you don't have to call them. The third-party calls them remotely, based on the time that you scheduled in the parameters (ie: "30 6 * * 5").
There is some discrepancy in different blogs about the CRON time. For instance some blogs insisted there are 6 time variables, but I found it worked with 5.
CronJobs should be in a separate file from the body of your main code, typically at the top of your folder structure near your "package.json" & "server.js" files.
It seems to be cleanest to setup all of your CronJob utilities directly inside the cronjob.js file. For instance: I used a separate database connection directly in cronjob.js and by-passed the api routes completely.
CronJobs should be initiated exactly once, at the beginning of the app launch. There are a few ways to do this: package.json or server.js are the most obvious choices.
Here is the file structure I ended up using:
-App
--package.json
--server.js
--cronjob.js
--/routes
--/src
--/models
--/public
...And then I imported the cronjob.js into "server.js". This way the cronjob function is initiated one time, when the server.js file is loaded during "dev" or "build".
For reference, here's the raw cronjob.js file (this is for an email notification):
const CronJob = require('cron').CronJob;
const Department = require('./models/department.js');
const template_remind = require('./config/remindEmailTemplate.js');
const SparkPost = require('sparkpost');
const client = new SparkPost('#############################');
const mongoose = require("mongoose");
const MONGODB_URI =
process.env.MONGODB_URI || "mongodb://localhost:27017/app";
mongoose.Promise = Promise;
// -------------------------- MongoDB -----------------------------
// Connect to the Mongo DB
mongoose.connect(MONGODB_URI, { useNewUrlParser: true }, (err, db) => {
if (err) {
console.log("Unable to connect to the mongoDB server. Error:", err);
} else {
console.log("Connection established to", MONGODB_URI);
}
});
const db = mongoose.connection;
// Show any mongoose errors
db.on("error", error => {
console.log("Mongoose Error: ", error);
});
// Once logged in to the db through mongoose, log a success message
db.once("open", () => {
console.log("Mongoose CRON connection successful.");
});
// ------------------------ Business Logic --------------------------
function weekday(notifications) {
Department.find({"active": true, "reminders": notifications, "week": {$lt: 13}}).distinct('participants', function(err, doc) {
if(err){
// console.log("The error: "+err)
} else {
console.log("received from database... "+JSON.stringify(doc))
for(let i=0; i<doc.length; i++){
client.transmissions.send({
recipients: [{address: doc[i]}],
content: {
from: 'name#sparkmail.email.com',
subject: 'Your email notification',
html: template_remind()
},
options: {sandbox: false}
}).then(data => {})
}
}
})
}
function weeklyNotif() {
new CronJob('45 7 * * 1', function() {weekday(1)}, null, true, 'America/New_York');
new CronJob('25 15 * * 3', function() {weekday(2)}, null, true, 'America/New_York');
new CronJob('15 11 * * 5', function() {weekday(3)}, null, true, 'America/New_York');
}
module.exports = weeklyNotif()
As you can see, I setup a unique DB connection and email server connection (separate from my API file), and ran all of the logic inside this one file, and then exported the initiation function.
Here's what appears in server.js:
const cronjob = require("./cronjob.js");
All you have to do here is require the file, and because it is exported as a function, this automatically initiates the cronjob.
Thanks for reading. If you have feedback, please share.
Noway, do call CronJob from client-side, because if there are 100 users, CronJob will be triggered 100 times. You need to have it on Server-Side for sure