I'm currently having some question on how some tools are working.
For a product build a test project to connect a database from notion to a twitter account in order to schedule some publication.
I will use OAuth2 and if possible nodeJs for the backend. I've seen that Zappier is doing this but I'm also curious on how to build this tool, so it's also a "improve your knowledge project" but I've some questions that I struggle with.
To retrieve notion database info I need to have a loop like:
Loop every x minutes:
- check for database update
- if there is a new tweet inside the database
- send the tweet to the twitter account related
- update the database status and add a published tag
This is globally the main action that needs to be done.
My questions are:
How can I loop to all my connected user in an effective maner?
If I have 400 users and I need to post 400 tweet with a one minute loop I think the second loop will start before the 400 requests might be sent, specially if I await for the answers.
How to not stop the loop for all my user if one error happen during the loop?
I can't really think of a solution to have one "thread/process" for each user in order for on loop to run for one user.
How can I start a new "loop" every time a new user connect, is it a good solution?
Is node js the best technology to go with?
If you know how zappier is doing it, or if you know or have already done something similar with this kind of isolation or if you know how I can "start" a new loop everytime a new customer connect to the app? any information are welcome.
In my app i have an api layer that defines routes, services layer to perform business logic, and a repository layer that queries the database. Services will call into the repository layer to retrieve data, perform various business logic and then return a response back to the api layer.
Now, I have some business logic that needs to be completed inside a database transaction. For example, let's assume my app allows you to buy tickets to a given event:
Service layer calls repository to atomically reserve tickets in the db by decrementing number of available tickets.
Service layer then makes an api request to process the payment for the ticket.
If the payment succeeds, service layer calls repository to create an "order" for the user and the given event.
If the payment fails, however, i want to rollback the changes made to the ticket availability. (or if for some reason creating the order for the user fails, same thing).
The problem is that it seems like a leaky abstraction if the services layer is aware of database transactions. Is there a better paradigm for this?
You don't want to hold a database transaction open the whole time you are making an API request anyway... You probably want to insert a 'pending' order, then once you get the payment change the status to 'confirmed'. This lets you store details about the error and report on failed payments too
My question relates to CakePHP 3.x and the problem of blocking selected pages and blocking all user sessions while the database is updated. The database of the website is in my case updated by the external program (I have influence on its code). The time of the update is long aproximatelly 40 minutes and I do not have any good idea how should I solve the following issues:
a) How should I automatically block some selected pages while the update takes place? More specifically, only while the update of the database is conducted no-one should be able to enter some pages eg.:
www.mypage/information/updateinfo
www.mypage/information/updatestatus
The user could be automatically redirect to the "update information page" or be informed otherwise.
b) How from the external program can I "throw away" all logged users and destroy all other sessions in the website? During the update no-one can be logged and no-one can have the ongoing session because the data in the database is being changed.
Pretty simple:
Put the user session in the DB
Have a table or lock file both apps have access to
When updating first delete all entries in the session DB table from your updating app
Then set the lock file / DB config lock entry
Check for the lock, redirect as needed
I have a multi-tenant REST app. When a new user first tries to access my application ( and assuming their admin has already granted the app permission for their directory ) I create a user row in my User table and store their name/email and other fields. I perform this in the TokenValidated event of JwtBearerEvents.
Unfortunately, I'm ending up with multiple users rows attempting to be inserted because of simultaneous (parallel request) hitting my web API. I do a simple SQL query for the User by ObjectId, and then create if necessary. This isn't threadsafe. I tried wrapping it in a SQL transaction, but the select isn't blocking and I'm not sure EF Core lets me perform the kind of locking I'd need to block other selects from completing.
I'm basing my code off the TailSpin PnP and they perform the same logic here as well. My guess is their site logic is forcing a single call the the WEB API first as part of the sign-in/login process, where the new user is created if they don't exist. In my flow, the REST API is hit right off the bat with multiple HTTP GET's and I just have to validate the bearer token in the headers and let ADAL cache it.
Aside from changing my client logic, and forcing the first call to API to be a single HTTP GET, how else can I make this work in a REST world? I can't use SESSION logic to block other calls in the same session. I'm not sure how I can perform a lock across the whole server ( Which works only if there's one server ). I could use the DB layer to hold a write lock, but that seems dirty. Maybe there's a better place to put the Create new user logic? Is there some other way for me to safely perform a one time atomic operation?
Based on the description, it seems you were create the user record(sign-up) when the users call the REST API and after the token is validated.
To fix the duplicates records issue, one possible way is that separate the sign-up progress from token validation as same the code sample TailSpin PnP. For example, we can custom the token handler to verify whether the users is sign-up and provide the UI for the users sign-up.
Another way is that, you can insert the users sequentially by using the lock. For example, here is the code for your reference:
private Task tokenValidated(TokenValidatedContext context)
{
lock (obj)
{
//query db and insert users here
}
return Task.Delay(0);
}
What is the best practice to handle transaction between application and paypal.
Consider:
I'm Alice and I want send money to Bob
In my DB I see that Bob has $200 and I want to send him $150.
Once transaction is sent I want to update the Bob's account such that it would contain $50.
Now according to PayPal API I can send Pay and receive success. However what happens
if I for example send Pay it succeeds but I fail to receive a response due to network problem. So I assume that error happened and try again and technically I'll send $300 to Bob instead of $150?
How can I handle such a transaction - between a local database that keeps an account and the remote PayPal API?
I had this exact concern recently with an ASP.NET MVC project I was completing for a client.
I learned two things:
Communication between Paypal and your database cannot be trusted (well, didn't really learn this, but it was entirely reinforced)
I now understand why so many websites that have Paypal as a transaction type mention there could be a processing period between the time that the transaction was completed and shipping/delivery of the product is completed.
The way you handle the situation is similar to the way a business could handle personal checks:
A personal check looks like currency (and typically is), but many businesses would like some sort of verification from the bank that funds are available before they accept payment - so they use a machine that asks the bank if funds are actually available.
If the machine says the funds are available, the business trusts it and you complete the transaction. However, the machine can give an error message that typically means "the funds are not available or something went wrong" and the business has a decision to make:
We can trust the customer and accept the check, deliver the product, and hope for the best when later depositing the check to the bank.
Or we can tell the customer that it will take time for the check to clear, deposit the check, wait for the funds to actually arrive in our account, and (if successful) deliver the product after the business receives funding.
This sounds inefficient with the way many businesses operate today, but it is something that does come up. In fact, this is why a lot of businesses stray away from accepting personal checks, they are unreliable when compared to other methods of payment.
Now how does this correlate to handling a Paypal payment?
A Paypal payment looks like currency (and it typically is), but many businesses would like some sort of verification from the Paypal that funds are available before they accept payment - so they use Paypal PDT, IPN, or other method for checking that the transaction was handled appropriately.
If Paypal properly responds to one of the verification requests, the business can trust it and complete the transaction. However, your website may throw an error of some sort (i.e. Paypal could reply with an IPN response of NOTVALID, or you could never get a reply from Paypal). The business has a decision to make:
The business can trust the customer and accept that they have made a Paypal payment and everything should be alright (very bad decision in the case of a Paypal transaction)
Or the business can tell the customer at check-out time that there may be a 72 hour processing period for Paypal payments.
This may not sound like the best way to operate your business, but it is the way we have to deal with an imperfect internet.
I would set up the Paypal payment flow similar to this:
UserA wants to send $100 to another UserB using Paypal
UserA enters the value in the 'checkout field' and is sent over to Paypal to verify the transaction.
UserA is sent back to your website from Paypal and your website performs the IPN check with the details that Paypal has POSTed to your site(I chose IPN in this case - as if we were using Express Checkout as opposed to some other payment gateway that Paypal offers).
If the IPN is VALID, process the transaction as expected.
If the IPN is not VALID, mention to the customer that there may be a delay in processing, have your application send you a notification that a possible Paypal transaction issue has occurred (you may want to include a reference id so that you can quickly find which transaction this notification is referencing), and mark the transaction as pending as opposed to complete or something similar.
An admin of the site who handles these notifications will manually investigate the transaction (or force the website to check with Paypal again - see the Paypal API documentation for details on this) and manually mark the transaction as complete or failed.
Notify those involved of the status of the transaction.
It is annoying that we have to have extra steps involved to make sure the money was transferred, but, as mentioned earlier, we are using an imperfect system and we want to be very certain of the success / failure of financial transactions.
An added bonus to this process is that there is likely to be notifications when someone is tampering with the Paypal payment system - leaving you better equipped to deal with evil-doers in the future.
Please refer this link ,Hope PayPal Authorization & Capture method will be suitable for you ,since you don't want lose the response as well as miscalculated amount transfers, PayPal provides correlation id that can be referred for PayPal to confirm your order status,it will be better to pass the order id to PayPal API.
https://cms.paypal.com/us/cgi-bin/?cmd=_render-content&content_ID=developer/howto_admin_authcapture
http://www.scribd.com/doc/6303345/40/CorrelationID-for-Reporting-Problems-to-PayPal
Good Luck!
This is known as 2-phase commit. As long as paypal does not participate in the same transaction, you will run into problems.
I would debit Alice the $150 and reflect the transaction is "Pending Confirmation", then periodically poll PayPal to synchronize your DB, since you have no control of when the network or PayPal may be available, post, reverse or adjust the transaction. Once PayPal processes the transaction, you can change the status in your DB from "Pending" to "Completed". BTW, this how bank accounts and credit cards are processed. You could apply a double-entry accounting method to your DB. (see this Q&A)
From what i see you need to make sure the transaction is complete otherwise nothing should be done .
If you deposit the money into PayPal API and you do not receive an response from PAYPAL API then you need to rollback the transaction in you DB.