I have been using prerenderIO for the past 3 years, but with recent changes to their subscription plan, it has become too expensive. I'm in need of a new solution as soon as possible. I use MEAN stack (MongoDB, Express, AngularJS) and don't have the time to learn server-side rendering, which is why I initially used prerenderIO.
Despite my attempts to return a static .html file, it still runs my AngularJS code and retrieves data from MongoDB.
I have tried debugging the crawl using the Facebook Debugger, but the issue remains unresolved.
app.get('/blog/best-place-in-xxx', (req, res, next) => {
if (!isCrawler(req.headers['user-agent'])) {
// Serve the pre-rendered HTML file
res.sendFile(__dirname + '/app/public/best-place-in-xxx.html');
return;
} else {
next();
}
});
function isCrawler(userAgent) {
const crawlers = [
'googlebot','bingbot','yandexbot','baidubot','facebot'
];
for (const crawler of crawlers) {
if (userAgent.toLowerCase().indexOf(crawler) !== -1) {
return true;
}
}
return false;
}
Can someone help me find a good alternative that won't negatively impact my Google ranking?"
Related
I did the lighthouse test on my website the report shows some issues but I don't know where to start to fix these issues
I see one of the issues fix to enabling text comparison
I am using Angular.JS with gulp
my backend Node.js
I tried using https://www.npmjs.com/package/compression but nothing changed
function shouldCompress(req, res) {
if (req.headers['x-no-compression']) {
// don't compress responses with this request header
return false;
}
// fallback to standard filter function
return compression.filter(req, res);
}
app.use(compression({ filter: shouldCompress }));
I am testing out the possibilities of Amazon AWS and DynamoDB, however, I can't find a solution as to why my Lambda function is executed with a delay.
I have set up a very simple test app with React and a simple table (less than 1kb) in DynamoDB. In react I have this call:
async readTasks() {
axios.get("https://13n6ump8q4.execute-api.us-east-1.amazonaws.com/default/serverlessToDoFunctionGET")
.then(res => {
let tasks = res.data.Items
this.setState({
tasklist:tasks,
});
});
}
Which connects through AWS API Gateway to my Lambda serverlessToDoFunctionGET function:
const AWS = require('aws-sdk');
const docClient = new AWS.DynamoDB.DocumentClient({region: "us-east-1"});
exports.handler = function(event, context, callback) {
let scanningParameters = {
TableName: "todo-app",
Limit:20
};
docClient.scan(scanningParameters, function(err, data) {
if(err){
callback(err, null);
} else {
callback(null, data);
}
});
}
When I test the Lambda function from AWS Console it executes on average anywhere between 30-100ms (ignoring cold start values) which is great.
However, when I run it from my app (both locally and deployed) the average execution time is around 250ms! Lowest I got was still a whopping 220ms.
What am I doing wrong here? To make me use of DynamoDB in my app I need to get below 100ms, otherwise it's just not worth it.
Things I've tried, but didn't have any effect:
Increased Lambda function memory size from 128MB to 3008MB
Double checked if the regions are everywhere the same
Instead of using axios in React, I used fetch inside componentDidMount()
What are you actually testing for here? What is the network latency between your client and the server? If you want to calculate execution time on the Lambda and any roundtrips to DynamoDB, it needs to happen inside the function.
Is is possible to get the onload event for the pwa application in general. I meant we had implemented the a custom versioning logic in-order to keep the app version based on database field.(ie clearing the service worker cache). The issues here is the logic almost works but when ever a new version is updated in the database, then we need to clear the cache of the respective browser in-order to trigger the update. On more investigation I found that when once the pwa app is opened, it is keeping the some sort of cache image, on reopening the pwa app again won't trigger the start-up code of the app, but load app from cache.
So is it possible to get an onload sort of event for pwa ?
For testing purpose I added some alert() in the app component, but didn't fired, on reopening a pwa app
this.httpService.GetAppVersion(ver).subscribe(
res => {
if (res != null || res !== undefined) {
this.version = res.versionNumber;
ver = localStorage.getItem("appVersion");
if (ver === null || ver === undefined) {
localStorage.setItem("appVersion", "1.0");
ver = "1.0";
}
let localVersion = ver.split(".");
let incomingVersion = this.version.split(".");
let result = this.helperService.compareVersion(
localVersion,
incomingVersion
);
//alert("result : " + result);
if (result === 1) {
const snackBarRef = this.snackBar.open(
"New version available. Load New Version?",
"Yes",
{ duration: 50000000 }
);
snackBarRef.afterDismissed().subscribe(() => {
console.log("The snack-bar was dismissed");
});
snackBarRef.onAction().subscribe(() => {
localStorage.setItem("appVersion", this.version.toString());
this.helperService.Update(); // which clears the cache
setTimeout(() => {
window.location.reload(true);
}, 500);
});
}
}
},
error => {
alert("http error" + JSON.stringify(error));
}
);
at least the code in the app component's constructor will execute every time when the app is reopened after closing.
See: How to display a "new version available" for a Progressive Web App
I know this question is very old, but what I'm doing now (and I'm trying to find a better approach because I don't really like this one) is storing the version on the service worker code.
Then, when the window.onload fires, the main JavaScript code sends a message to the service worker (using postMessage()) and the service worker replies with the version number.
It's not exactly what you need, but it's an approximation.
Still, and as I said, I'm looking for a better, more maintenable approach. If I find one I'll post it here, just in case someone is searching for this (as I did).
Working on a SSR app using React/Express i'm trying to get a grip on renderToNodeStream and streams in general.
I have a large page (400kb not compressed) and using renderToNodeStream give a really good TTFB (time to first byte), all I miss is some compression to make the HTML response smaller sent back smaller but I cant make it work with renderToNodeStream.
am I missing something ?
Is it possible to stream freshly rendered responses AND compress them ?
const stream = renderToNodeStream(<MyApp/>)
// this doesn't work
stream.pipe(zlib.createGzip())
stream._flush = zlib.Z_SYNC_FLUSH
stream.pipe(
res,
{ end: "false" }
)
// stream.on("data", data => {
// console.log(data)
// })
stream.on("end", () => {
res.end(pageEnd())
})
If you are using express and compression middleware, you probably need to write
res.setHeader('Content-Type', 'text/html');
before streaming to the client.
As shown in compression middleware code, if Content-Type does not exist, the middleware won't compress the content.
The final code would be:
res.setHeader('Content-Type', 'text/html');
const stream = renderToNodeStream(<MyApp/>)
stream.pipe(
res,
{ end: "false" }
)
stream.on("end", () => {
res.end(pageEnd())
})
https://reactjs.org/docs/react-dom-server.html#rendertonodestream
renderToNodeStream() returns a readable stream. With this function you are assembling a snippet of HTML and send it to the users' browser and the process repeats.
The reason you may want to take this approach as you know is for performance, but I am not too sure if compression is a part of the process.
I am new to SEO and just want to get the idea about how it works for Single Page Application with dynamic content.
In my case, I have a single page application (powered by AngularJS, using router to show different state) that provides some location-based search functionalities, similar to Zillow, Redfin, or Yelp. On mt site, user can type in a location name, and the site will return some results based on the location.
I am trying to figure out a way to make it work well with Google. For example, if I type in "Apartment San Francisco" in Google, the results will be:
And when user click on these links, the sites will display the correct result. I am thinking about having similar SEO like these for my site.
The question is, the page content is purely depending on user's query. User can search by city name, state name, zip code, etc, to show different results, and it's not possible to put them all into sitemap. How google can crawl the content for these kind of dynamic page results?
I don't have experience with SEO and not sure how to do it for my site. Please share some experience or pointers to help me get started. Thanks a lot!
===========
Follow up question:
I saw Googlebot can now run Javascript. I want to understand a bit more of this. When a specific url of my SPA app is opened, it will do some network query (XHR request) for a few seconds and then the page content will be displayed. In this case, will GoogleBot wait for the http response?
I saw some tutorial says we need to prepare static html specifically for Search Engines. If I only want to deal with Google, does it mean I don't have to serve static html anymore because Google can run Javascript?
Thanks again.
If a search engine should come across your JavaScript application then we have the permission to redirect the search engine to another URL that serves the fully rendered version of the page.
For this job
You can either use this tool by Thomas Davis available on github
SEOSERVER
Or
you can use the code below which does the same job as above this code is also available here
Implementation using Phantom.js
We can setup a node.js server that given a URL, it will fully render the page content. Then we will redirect bots to this server to retrieve the correct content.
We will need to install node.js and phantom.js onto a box. Then start up this server below. There are two files, one which is the web server and the other is a phantomjs script that renders the page.
// web.js
// Express is our web server that can handle request
var express = require('express');
var app = express();
var getContent = function(url, callback) {
var content = '';
// Here we spawn a phantom.js process, the first element of the
// array is our phantomjs script and the second element is our url
var phantom = require('child_process').spawn('phantomjs',['phantom-server.js', url]);
phantom.stdout.setEncoding('utf8');
// Our phantom.js script is simply logging the output and
// we access it here through stdout
phantom.stdout.on('data', function(data) {
content += data.toString();
});
phantom.on('exit', function(code) {
if (code !== 0) {
console.log('We have an error');
} else {
// once our phantom.js script exits, let's call out call back
// which outputs the contents to the page
callback(content);
}
});
};
var respond = function (req, res) {
// Because we use [P] in htaccess we have access to this header
url = 'http://' + req.headers['x-forwarded-host'] + req.params[0];
getContent(url, function (content) {
res.send(content);
});
}
app.get(/(.*)/, respond);
app.listen(3000);
The script below is phantom-server.js and will be in charge of fully rendering the content. We don't return the content until the page is fully rendered. We hook into the resources listener to do this.
var page = require('webpage').create();
var system = require('system');
var lastReceived = new Date().getTime();
var requestCount = 0;
var responseCount = 0;
var requestIds = [];
var startTime = new Date().getTime();
page.onResourceReceived = function (response) {
if(requestIds.indexOf(response.id) !== -1) {
lastReceived = new Date().getTime();
responseCount++;
requestIds[requestIds.indexOf(response.id)] = null;
}
};
page.onResourceRequested = function (request) {
if(requestIds.indexOf(request.id) === -1) {
requestIds.push(request.id);
requestCount++;
}
};
// Open the page
page.open(system.args[1], function () {});
var checkComplete = function () {
// We don't allow it to take longer than 5 seconds but
// don't return until all requests are finished
if((new Date().getTime() - lastReceived > 300 && requestCount === responseCount) || new Date().getTime() - startTime > 5000) {
clearInterval(checkCompleteInterval);
console.log(page.content);
phantom.exit();
}
}
// Let us check to see if the page is finished rendering
var checkCompleteInterval = setInterval(checkComplete, 1);
Once we have this server up and running we just redirect bots to the server in our client's web server configuration.
Redirecting bots
If you are using apache we can edit out .htaccess such that Google requests are proxied to our middle man phantom.js server.
RewriteEngine on
RewriteCond %{QUERY_STRING} ^_escaped_fragment_=(.*)$
RewriteRule (.*) http://webserver:3000/%1? [P]
We could also include other RewriteCond, such as user agent to redirect other search engines we wish to be indexed on.
Though Google won't use _escaped_fragment_ unless we tell it to by either including a meta tag; <meta name="fragment" content="!">or using #! URLs in our links.
You will most likely have to use both.
This has been tested with Google Webmasters fetch tool. Make sure you include #! on your URLs when using the fetch tool.