The ERP used in the company that I work for has some information available across API requests. Currently my code just grab each document and replace at Mongo. It does not handle excluded documents, for example. Moreover, this code takes hours to get all information because API requests are limited.
Is there a tool, library, app, etc. that can sync these databases?
What is the best practice to achieve the synchronization of API and MongoDB database?
PS: why do I need to sync them? API requests are limited.
PS2: not all, but some API requests return "last edited" timestamp. I am using it to reduce running time.
Related
I am working on a mobile app, where users have complex profiles and can chat to each other. So right now I am using firebase's onSnapshot() listener in the frontend code to get the real-time data ASAP. I am thinking whether its secure enough? Would it be better to move it to the backend server? Than fetching real-time would be way more complex I guess. Could you tell me what are the dangers of keeping these listeners on frontend? Thank you.
Could you tell me what are the dangers of keeping these listeners on
frontend?
The standard way for securing your database (as well as Cloud Storage buckets) is by using Security Rules.
In particular the documentation explains that:
Every database request from a Cloud Firestore mobile/web client
library is evaluated against your security rules before reading or
writing any data. If the rules deny access to any of the specified
document paths, the entire request fails.
This evaluation is totally independent from the way you query the database: Whether you query it via a realtime listener or via a one time fetch (using one of the get() methods) the database request is evaluated against your security rules.
So, in conclusion, if you use Security Rules, there is no "danger" to use some realtime listeners.
I am thinking whether its secure enough? Would it be better to move it
to the backend server?
Based on the details you share in your question there is no reason, IMO, to switch to a back-end approach (like using Cloud Functions to read the database). One reason could be that your security logic is too complex to be implemented with the security rules language/model. But note that you can build complex logic in Security Rules, for example by basing them on Custom Claims or other Firestore documents.
I'm implementing a Django web service, which is about to have different platform apps,
Reactjs for computers, a swift app for ios, and Kotlin for android devices. the protocol is rest API and perhaps a chat feature included then Django channels are used as well. The data format is JSON. For deployment, I intend to use docker which includes Django, celery, and ReactJS app. And the database is on another separate server which is PostgreSQL. I was thinking to collect some user activity data and some history logs to show the user itself what she/he has done so far. After hours of searching, I came up with Kafka! unfortunately, I have no idea how can I use Kafka and integrate these stuff together and how can I deploy these things. I wish there was a system schema for this specific kind of system that shows what is what and where is what?
Kafka will only integrate your database and Django, with some effort, and ideally a separate Kafka Connect service.
From React (or other clients), you'll need to query some Django API routes which will then query your database. Kafka won't help with your frontend, and isn't really what is exposing the history/activity you're interested in displaying. In other words, you could simply write that to the database, and skip Kafka entirely.
Essentially, you're following the CQRS design pattern if you properly separate Kafka writes from end user / UI reads.
shows what's what and what's where!
Unclear what this means, but data lineage and metadata tools are a whole separate thing. For example, LinkedIn DataHub collects information such as this
I'm trying to host a database (on either AWS or Heroku) to store data from my web app. I also need to create an API to interact with the database. Basically, like this picture from Google.
Image source: https://dzone.com/articles/an-introduction-to-restful-apis
What I'm trying to figure out:
Is the API and database typically hosted separately? Or are they often hosted together? I'm not sure if the API and DB are together 1 component (with sub components being API and DB), or if they are 2 separate standalone components.
If they are 1 component and they can be hosted together, then I believe you can use something like Express.js for your API which can query the database and respond to HTTP requests from the website.
If they are 2 separately hosted components, I feel that means I have to have 2 APIs, unless my API can directly talk to the database (I'm not sure if this is proper). So I would need my API to talk to some server side technology (PHP, Java, etc.) which would then query the database and return result to the API. So basically my API is talking to an API which is talking to the database. I'm not sure if this is over complicating things, but it sure seems like it.
I'm trying to clarify how this process works.
Is the API and database typically hosted separately? Or are they often hosted together?
Since the API service needs to make lots of requests to the database, you want the minimum network lag possible between them.
So generally they should be hosted together. What that means depends on the style of hosting. Using the same cloud service makes sense if you are using something like AWS Lambda. Otherwise using the same machine for both the database and HTTP service until you scale to the point where you need to separate them.
If they are 1 component and they can be hosted together, then I believe you can use something like Express.js for your API which can query the database and respond to HTTP requests from the website.
This is a common approach.
I currently have an Electron ReactJS application on hand that uses CouchDB as its backend for syncing and real-time updates. I have been tasked with "porting" the backend of this application to an existing webserver. The goal is to use the existing permission management and business logic implementations. At the same time the intent is to refactor the data structures as the data is heavily relational in nature. That said, personally, I would keep the app on CouchDB as it full-fills the main "offline-first and real-time" requirements and just add the missing authentication and permission layers, but management wants otherwise.
The port would be using an existing web server (running Play Framework with Scala) and a relational database (MySQL). I have been scavenging the web for a good existing solution. The "simple" solution that came to my mind sounds tedious and like I'm reinventing the wheel. My idea was to create an API and on top of it also send real-time updates to the users to which a change is relevant via web sockets. For state management on the client I would use Redux + Redux Offline. While this would work, this would require a lot of manual wiring of CRUDs on the backend server and the according requests and mutations on the client.
I have taken a look at AWS AppSync, Meteor.js and Apollo. AWS AppSync sounds like exactly what I could use, but it relies on the database being accessible to it, which is not an option due to my DB instance being on premise. From Apollo, the client part sounds like an option I could go with and then use Sangria on the backend. I suppose I could also drop the idea of Redux and use Apollo's "local state", although this requires more thought as I'm not familiar with it.
All of the latter solutions involve GraphQL. While this would still require quite some work on the backend, the communication itself between the frontend and backend would be simpler to handle.
In my case where the use of an existing backend server is a must, are there any more elegant solutions for offline-first collaborative real-time apps? Am I missing something?
I'm new to web development and was curious about something. When posting to an endpoint to then receive a value from a server-side function, is it problematic if multiple users are writing to the same endpoint? Can this corrupt the value returned?
For instance, I'm using Stripe in a project and you're supposed to post to an endpoint to generate a user-specific ephemeral key. There's a 1-2 second delay in the response at times, so would there be a problem if two users posted to the same endpoint within a few milliseconds?
Capable web server software is designed with concurrency in mind, meaning a server can handle multiple user requests at the same time.
If you're curious about the specific techniques of how this is done, or web server architecture in general, this article is pretty interesting and offers some sample applications
http://www.linuxjournal.com/content/three-ways-web-server-concurrency