We have a React.js web application for one of our clients. The client has thousands of customers, and each customer makes payments/transactions every couple of months. The web application is developed using React and node, and is deployed using firebase-cli. In order to process transactions, we work with a 3rd party vendor (a credit card processing company) who supplies us with a Transaction Report REST API.
The issue is arising with our cloud function that is deployed to Firebase. This cloud function runs every night at 11:50PM EST, using Google Cloud cron jobs as the automation tool. The function makes a call out to the REST API using axios, and when a payload response is returned, our logic iterates through the response. The logic in this function gathers data from each transaction (e.g. custom Account Number, name, address, payment amount, etc.) and updates the history node within our Realtime Database.
Here's where things get really weird. We have both a DEV and PROD project for our client. When this function is deployed and running in DEV, it works with no issues! But when in PROD, it only works about 70% of the time (that is a rough guess). When I say it "only works sometimes", the function logs are printing out Function execution took 1556ms, finished with status: 'ok' - but the updating does not always occur.
Node version is v12.16.2, npm version is 6.14.4
Is there something specific that we need to set in regards to the timeout or memory usage? It is just bizarre that it works with no issues in one Firebase project but not the other.
Please advise!! Any help is great, as our client processes hundreds of transactions a night. I can show some code if necessary, just would like to protect our client's code if at all possible...
Related
I am currently creating a react native + expo application upon which essentially each page makes an API call, which is a lot of API calls. I have this app also connected to firebase for different information. The things is, each of these pages don't update more than once or twice a day for the most part, so I really don't want the End User to be calling the API that much either.
My question is, is there a way to write and host a script that will continuously run that knows to call this API once every hour (or so) and then rewrite to the firebase db from which I can then only need to pull from the database as compared to having each user individually making dozens of API calls.
Please let me know! I have spent days on google and am no closer than I was before. I'm also willing to change my set up from firebase if it is not possible to accomplish that way. Thanks!
You can use a Cloud Functions scheduled trigger to run code periodically that can make changes to your database.
I'm attempting to create a microservice on Google App Engine that is not intended to handle HTTP requests.
Instead, I was hoping to have a continuously running Python script that monitors a remote queue--RabbitMQ, to be precise--and sends out an api-call to another service as tasks are pushed to the queue.
I was wondering, firstly, is it possible to run a script upon deployment--one that did not originate with a user action/request?
Secondly, how would I accomplish this?
Thanks in advance for your time!
You can deploy your "script" as a manually scaled module -- see https://cloud.google.com/appengine/docs/python/modules/ -- with exactly one instance. As the docs say, "When you start a manual scaling instance, App Engine immediately sends a /_ah/start request to each instance"; so, just set that module's handler for /_ah/start to the handler you want to run (in the module's yaml file and the WSGI app in the Python code, using whatever lightweight framework you like -- webapp2, falcon, flask, bottle, or whatever else... the framework won't be doing much for you in this case save the one-off routing).
Note that the number of free machine hours for manual scaling modules is limited to 8 hours per day (for the smaller, B1 instance class; proportionally fewer for larger instance classes), so you may need to upgrade to paid-app status if you need to run for more than 8 hours.
Like #brant said, App Engine is designed to handle HTTP requests. It's not a perfect fit for background jobs, unless you try to wrap your logic into one http request.
Further, App Engine will emit an error when the response timeout, depending on your scaling settings. If you want to try it, consider basic or manual scaling.
For this type of workload, I would suggest you use a VM.
I think there are a few problems with this design.
First, App Engine is designed to be an HTTP request processor, not a RabbitMQ message processor. GAE is intended for many small requests, not one long-running process.
Second, "RabbitMQ should not be exposed to the public internet, it wasn't created for such use case."
I would recommend that you keep the RabbitMQ clients on the same internal network as the RabbitMQ broker, and have the clients send HTTP requests to App Engine.
I planning to create a NodeJS program, that work 24/7, that ping and make requests to an external server (outside of google cloud) every minute. Just to see that it the external services are are live.
If there is any error it will notify me by SMS & Email.
I don't need any front-end for this app, and no one needs to connect to it. Just simple NodeJS program.
The monitoring and configuration will be by texts files.
Now the questions:
It looks like it will cost me just $1.64. It sounds very cheap. Am I missing something?
It needs to work around the clock, I will request it to start it once, and it need to continue working, (by using setInterval). Is it will be aborted?
What it is exactly mean buy 1 instance. What an instance can do? Only respond to one request or what?
I tried to search in Google: appengine timeout, but didn't found anything that helps.
Free Quota
If you write your application in Python, PHP, Go or Java it can fit in free usage quota:
https://cloud.google.com/appengine/docs/quotas
So there will be absolutely no costs to run it on Google App Engine platform.
There are limit of 657,000 UrlFetch API Calls per day (more than 450 calls per minute in 24/7 mode) for free apps. 4GB traffic may also be sufficient for this kind of work.
Keep in mind there is no SMS sending services provided by Google App Engine and you will need to spend additional UrlFetch API calls to use external SMS services.
Email sending is also limited to 100 Emails per day (or 5000 Emails to admin address), so try not so send repeated notifications about same monitored server every minute, or you'll deplete your Email quote in 1.5 hours.
Scheduled Tasks
There is no way to run single process indefinitely without interruption on App Engine. But you don't have to!
You'll need to encapsulate all the work you're planning to execute in every iteration into single task and then schedule it to run every minute with Cron. See this documentation for Python: https://cloud.google.com/appengine/docs/python/config/cron
It is recommended to have some configuration page where you can set some internal configuration or see monitoring statistics, at least manage flag to temporarily pause tasks execution without redeploying your app.
I have a C# Web API project on server side and on front-end I have ExtJS 4.2.1 (Javascript framework client).
There is a section in my app where I request to start a long running process (about 5 minutes) and I want to show the user the status of the process being executed.
Basically, the process will run a special calculation for every employee in the database (about 800), so I want to let the user know which Employee is being processed in that moment.
So I was thinking in two ways of doing this, and maybe I don't know if having both is ok.
Use SignalR to show the information of the process in Real Time.
Write to a database table all the process log (every employee that its being processed).
If I use the first approach, if the user close the browser he will loose all the information about the process and if he log into the app again he will only see the actual status.
If I use the second approach, if he log into the app again he could see all the information, and using maybe a timer on client side the data could be refreshed every 5 seconds.
Does anyone have implemented something like this? Any advice is appreciated.
You should use a combination of the two. When you have calculated a employee save the state to the database and publish the change on a service bus.
Let SignalR pick these messages up and forward them to the client. This way the user will see old state when he connects and new state then they arrive with SignalR. I have created a Event aggregator proxy that makes this very easy.
https://github.com/AndersMalmgren/SignalR.EventAggregatorProxy/wiki
Follow the wiki to set it up, here is a demo project
https://github.com/AndersMalmgren/SignalR.EventAggregatorProxy/tree/master/SignalR.EventAggregatorProxy.Demo.MVC4
Live demo
http://malmgrens.org/Signalr/
We have been using push queue for a very long time and have no problems in consuming the tasks from a dev server.
However during implementing a new service with pull queue, it became difficult to figure out how to do the same thing on the dev server.
Basically from the docs, what we can see is that you should use a REST api (we can't use the direct queue api as it is consumed by an external app) to lease/delete a task with the end point of
https://www.googleapis.com/taskqueue/v1beta1/projects/taskqueues
But obviously this will not work in local dev server, and it appears that no place have talking about this.
Just wondering if anyone had ever run into the same issue had can shed some light?
With Pull Queue, task consumer can be internal or external.
If you need it to work on dev server, then just create a handler (a servlet) and use internal API to add, lease and delete tasks.