I switched my ExpressJS application from Heroku to Google App Engine.
Everything works ok now.
But I am curious about how to debug my application if some exceptions happen on GAE.
On heroku, I can do heroku logs -t to trace the errors. I can check the variable printed by console.error(var) as well.
However, I don't know how to do the same thing on GAE.
I have checked logging of Stack Driver, it seems that it only shows some informations of every HTTP request instead of the detailed logs like heroku.
I have found that there is error report service of Stack Driver. It may what I want.
Here is the tutorial telling us how to setup. But the steps are confusing for me.
Does anyone have the experience to setup the error reporting?
I am finding a more clear steps to setup this.
Thanks a lot and appreciated!
For Stackdriver Error Reporting:
You can use code similar to the one advised for Google Compute Engine: https://cloud.google.com/error-reporting/docs/setup/compute-engine#send_exception_data
Here is what worked for me using Express and Winston on App Engine flexible environment:
var winston = require('winston');
winston.add(winston.transports.File, { filename: '/var/log/app_engine/custom_logs/my.errors.json' });
var report = function (err, req) {
var payload = {
serviceContext: {
service: 'my service',
},
message: err.stack,
context: {
httpRequest: {
url: req.originalUrl,
method: req.method,
referrer: req.header('Referer'),
userAgent: req.header('User-Agent'),
remoteIp: req.ip,
responseStatusCode: 500,
}
}
};
winston.error (payload);
};
// Handle errors
app.use(function (err, req, res, next) {
report(err, req);
res.status(500).send(err.response || 'Something broke!');
});
For Stackdriver Logging:
Indeed, the request_log only contains HTTP requests log entries on App Engine flex. Look in the stdout log to see the output of your application.
If you just want to read data from the logs, you can try:
$ gcloud preview app logs read
2016-05-30 18:46:29 default[alpha2] saved to datastore: mountain biking
2016-05-30 18:46:29 default[alpha2] saved to datastore: adventure
2016-05-30 18:46:29 default[alpha2] saved to datastore: mountain bike
2016-05-30 18:46:29 default[alpha2] saved to datastore: cycle sport
Related
I have been using Datastore in AppEngine since few weeks ago and there was no such access issue in production. Today 1pm SGT, my service was suddenly returning 500 error with this error message although I never deploy to production.
Error: Could not load the default credentials. Browse to https://cloud.google.com/docs/authentication/getting-started for more information.
This error probably happened when accessing Datastore with my GCP's default credential:
const {Datastore} = require('#google-cloud/datastore');
const datastore = new Datastore();
const [shop] = await datastore.get(
datastore.key(['Shop', Number(phone)])
)
My stacks:
Standard AppEngine with nodejs10
Koa, Next, Datastore
In App Engine dashboard, as a random solution, I changed to the old version. Then, it suddenly started working. Then, I changed back to the original version. It worked well too. Any clue?
Suspect 1: Around the time, I was working locally. Although I never deploy, I was executing few GCP commands. These should not affect production tho
export GOOGLE_APPLICATION_CREDENTIALS="[my local credencial json file]"
gcloud config set project [project-name]
Seems like this problem has to do with App Engine/Cloud Function cold starts. In this scenario, the environment would not have loaded the credentials yet when the client library is being initialized.
Apparently it is fixed in newer versions of the client libraries: https://github.com/googleapis/gapic-generator-typescript/issues/287
But if not or if you do not want to upgrade your #google-cloud/ client libraries, this is the suggested workaround:
const {Datastore} = require('#google-cloud/datastore');
let datastore;
// If you are using it in a request handler
exports.handler = async (req, res) {
if (!datastore) {
datastore = new Datastore();
}
const [shop] = await datastore.get(
datastore.key(['Shop', Number(phone)])
);
}
See also: https://github.com/googleapis/google-auth-library-nodejs/issues/798#issuecomment-591622283
Is there a way to run firestore locally (e.g. for testing purposes)?
What would the approach to write tests against the DB (except of using mocks)
Update 2020:
There's now also a Firebase Emulator Suite.
Update Nov 2018:
Local emulation, at least for the purpose of testing Firestore rules, was demoed at Firebase Summit 2018 using #firestore/testing and documented under Test your Cloud Firestore Security Rules.
It looks like it's along the lines of:
const firebase = require(`#firebase/testing`)
const app = firebase.initializeTestApp({
projectId: 'my-project',
auth: { uid: '123', email: 'name#domain.com' }
})
const attempt = app.firestore()
.collection('colId').doc('docId').get()
firebase.assertFails(attempt)
firebase.assertSucceeds(attempt)
It seems early-on, as it's not been noted in the release-notes, but I'm sure it's coming along.
There is not currently, but stay tuned as it's something we want to provide.
In the meantime we suggest uses a separate testing project to cover this. The daily free tier per project helps with this too.
You can run the Firestore emulator by running:
gcloud beta emulators firestore start
and then set the FIRESTORE_EMULATOR_HOST environment variable as per the console output (e.g. run export FIRESTORE_EMULATOR_HOST=::1:8505).
This requires the Google Cloud SDK and a Java 8+ JRE installed and on your system PATH.
for a firestore testing write a js example test.js
you could test write with this format example
var data = {
value: {createTime: new Date(),
updateTime: new Date(),
fields:{
name:{stringValue:'new value data'},
age:{integerValue:50}
}
},
oldValue: {createTime: new Date(), //old create time
updateTime: new Date(), //old update time time
fields:{
name:{stringValue:'olvalue data'},
age:{integerValue:50}
}
}
};
testFireStoreEvent(data);
for run execute
firebase experimental:functions:shell < test.js
UPDATE!!!! VALID FOR WRITE AND UPDATE EVENTS
var data = {
before: {
//your before data
},
after: {
//your after data
}
};
testFireStoreEvent(data);
There are two libraries which attempt to facilitate mocking of the firebase sdk.
1) https://github.com/soumak77/firebase-mock
2) https://github.com/mikkopaderes/mock-cloud-firestore
I currently use the first one, since it seems to have a bit more of the SDK implemented.
They're not perfect, but they're currently sufficient for my needs, and are preferable to the other approaches since they're entirely in-process.
Note that firebase-mock (#1) does cause a webpack error if used as-is from Webpack/web code. To resolve, you can use option #2 (mock-cloud-firestore), or use the workaround mentioned here (until a fix gets merged): https://github.com/soumak77/firebase-mock/issues/157#issuecomment-545387665
Other options:
3) Firestore emulator: needs the google-cloud-sdk, and relies on a separate process
4) Separate test project: relies on connection to the internet, which also means possible quota limitations/costs
5) firebase-server: Only supports the realtime-database api, not Firestore
Firestore can be setup in local using gcloud.
Start the firestore emulator by running gcloud beta emulators firestore start --host-port=localhost:8081 and if it started successfully you will be seeing Dev App Server is now running
In case if you are using #google-cloud/firestore then create the Firestore instance in this way
// Firestore instance only for test env
const { Firestore } = require('#google-cloud/firestore')
const instance = new Firestore({ projectId; 'Your project id', host: 'localhost', 'port': 8081})
Now you have an option to work with local firestore emulator by setting local host:
var db = firebaseApp.firestore();
if (location.hostname === "localhost") {
db.settings({
host: "localhost:8080",
ssl: false
});
}
https://firebase.google.com/docs/emulator-suite/connect_and_prototype#instrument_your_app_to_talk_to_the_emulators
I'm currently working on an Angular 2 web application which communicates with an API. In the application the user is able to choose a payment option and the API will return the url to the payment service.
The problem is that the payment service uses POST to go to the confirmation page which WebPack does not accept (for some reason it only allows GET requests) and we get the following error:
Cannot POST /selection/payment-method
Does anybody know how we could configure that WebPack allows POST requests also? I've contacted the payment provider but it is not possible to do a GET request instead of POST.
Thanks
Based on #Sven's answer, modification to the setup so that it works for POST all throughout
Add dependencies on body-parser, sync-request and add require dependencies on both in your webpack.config.js
var bodyParser = require('body-parser');
var request = require('sync-request');
In devServer part of webpack.config.js
devServer: {
setup: function(app) {
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({
extended: true
}));
app.post(/^\/(URL1|URL2|URL3)\//, function(req, res) {
var serviceCallResponse = request('POST', 'your app server url here' + req.originalUrl, {
json:req.body
});
res.send(serviceCallResponse.getBody('utf8'));
});
},
proxy: {
'*/other URLs proxy/*': 'your app server url here'
}
}
Change URL1/2 to the URLs you want to proxy and you place your app servers address.
This will work for all sorts of POST request proxy (working on json payload)
A hackish way to at least not get 404 errors is to proxy requests to /selection/payment-method and send back an empty response (or whatever content you want, I think that res is an instance of Express's Response class) for those, by adding the following to webpack.config.js:
devServer: {
proxy: {
'/selection/payment-method': {
bypass : (req, res) => res.end()
}
}
}
Documentation here.
Thanks to #robertklep who send me the link to the proxy documentation we found a way to deal with it. What we needed to do was instead of handling the POST request we needed to transform it someway into a GET. After reading some more of the documentation we came across the setup: property for your webpack-dev-server configuration.
With the setup: property you get the expressjs object and you are able to catch urls before it hits the route that says Cannot POST /url/to/page.
We ended up with this:
devServer: {
setup: function(app) {
app.post('/selection/payment-method', function(req, res) {
res.redirect('/selection/payment-method');
});
},
}
This way we got a GET request instead of POST and our application does an API request to check if the payment succeeded or not.
This is only used in development!
The Webpack-dev-server is only intended as your front-end server only, eg. to serve your static assets. Therefore only GET requests are supported.
If you would want to use a proxy or backend server, then you should implement this. You can use Express for this.
See how you can setup basic routing.
I have a Node/Angular app I'm trying to deploy. It uses two node servers: One to essentially serve the app; another to get data from an API, when a specific port is requested by the app, and store that data locally.
I've got it working perfectly on my own local machine. However, when I deploy to production environments -- either Heroku or AWS Elastic Beanstalk -- I find that the second script either won't run or won't start properly. The end result is, it doesn't get the data I need.
Here are the two scripts; they're both set to run in package.json under "start": "node main.js & node node-server.js"
main.js (again, this one seems to be serving the app just fine):
var express = require('express');
var app = express();
app.use(express.static(__dirname + '/app'));
app.listen(process.env.PORT || 3000);
node-server.js (the one that doesn't seem to work; no data is gathered or populated in the app):
var http = require('http');
var port2 = 1234
var fs = require('fs');
//We need a function which handles requests and send response
function handleRequest(req, res) {
request.get({
url: 'http://sample-url.json',
qs: {
url: 'http://sampletool/pb/newsletter/?content=true'
}
}, function (err, result) {
res.end(result.body);
fs.writeFile('app/data.json', result.body, function (err) {
if (err) return console.log(err);
console.log('API data > data.json');
});
});
}
//Create a server
var server = http.createServer(handleRequest);
//Lets start our server
server.listen(port2, function () {
//Callback triggered when server is successfully listening. Hurray!
console.log("Server listening on: http://0.0.0.0:%d", port2);
});
Then, the main Angular app calls this port (http://0.0.0.0:1234) when the page is loaded, to request new data.
Elastic Beanstalk is using nginx, something I'm not super familiar with and that I don't have running on my local.
Is there something big I'm missing in configuring multiple node.js servers to be running on different ports in a production environment? Thanks in advance for any help.
For security reasons, cloud service providers typically allow the usage of only one port (which is dynamically and randomly assigned to the PORT environment variable) for an application to use from a node server. Read this section from Heroku documentation to understand more about this.
This is why the main app (main.js) that uses process.env.PORT is working and the other app (node-server.js) that uses hard-coded 1234 is not.
This question has some pointers about the feasibility of multiple ports on Heroku (though, there is no good news there, I am afraid).
As how to go about fixing this, one thing that could be tried is to split this into two separate apps that are deployed separately with separate package.json etc.
I've coded an Java App Engine app that uses Cloud Endpoints. I want to talk to these endpoints from a Node.js app. Is it possible to use the google-api-nodejs-client for this purpose?
I've already tried this:
var googleapis = require('googleapis');
googleapis
.discover('myapiname', 'v1dev', {baseDiscoveryUrl: 'http://localhost:8888/_ah/api/discovery/v1/apis/'})
.execute(function(err, client) {
console.log(err);
console.log(client);
client.myapiname.domains.list().execute(function(err, resp) {
console.log(resp);
});
});
But the library does not discover my endpoint, the callback returns null for err and client looks like this:
{ clients: [],
ops: {},
authClient: null,
undefined:
{ apiMeta: '<html><head><title>Error 404</title></head>\n<body><h2>Error 404</h2></body>\n</html>',
authClient: null,
defaultParams: null } }
I've replaced my real api name with 'myapiname' and of course the URL http://localhost:8888/_ah/api/discovery/v1/apis/ is reachable (it returns me the right discovery JSON if I open it in the browser on the same computer).
have you tried the same code with code deployed on some vm on google app engine.
I tried the same with our app engine url and it works like charm.