Deployment of react-snap on a CRA app has been mostly painless, giving huge page load speed boosts and requiring zero specialized configuration.
However, I'm seeing occasional issues with deploys (both locally and from netlify) only crawling a single page and then appearing done. Like this:
Normal result (maybe 50% of the time) means crawling ~50 pages and then everything else successfully finishes.
I've tried limiting concurrency to 1 without improvement. What other tools can I use to figure this problem out or configuration options can I include to fix this?
Figured this out: Webpack was setting PUBLIC_URL to the production domain, and new deploys were looking on that domain for a JS file that looked like main.1234abcd.js, using a hash of the js file for cache busting. This didn't exist on the production domain before it was deployed so loading the page failed and no links were detected.
Setting the JS links to root-relative URL (i.e. /static/js/main.1234abcd.js) loaded the JS correctly from the snap-created server and allowed it to be crawled correctly.
In addition, it was helpful to debug via the anchor crawling section in react-snap here: https://github.com/stereobooster/react-snap/blob/master/src/puppeteer_utils.js#L108-L119
I'm pretty new with AngularJS and server config stuff, and this is a problem I haven't found a satisfactory solution so far.
I would like to be able to use the HTML5 url on a website (without hashbangs), so that I could use addresses like "mydomain/contact" to navigate (I'll stick with the "contact" example for simplicity).
To do that, as I've found so far, one should do two things:
Enable HTML5 on the client side
Enable the HTML5 format on the app/app.js file (also adding the dependency)
$locationProvider.html5Mode(true);
It makes possible to click on links and get the proper url. Still, it doesn't allow someone to access directly the HTML5 url. To get to the "contact" page, I still can't directly access "mydomain/contact" (we get a 404) and I know it makes sense. To solve this, it is still necessary to implement something server-side.
Server-side config
Configure the server to respond with the right file, i.e., I should configure the server to make it respond the same way when I request "mydomain/#/contact" and "mydomain/contact".
The last item is where I'm stuck. I've found many answers like this: "you should configure your server" (they assume the reader already knows how to do this. I don't), but I can't find a complete example on how to do that or where to put any needed files.
I'm using AngularJS 1.6.x and npm 3.10.9 (npm start). My question is: is there any complete example on how to fully use HTML5 urls with AngularJS?
The only problem that exists is that angular can't handle requests it doesn't receive. You need some catch-all so that all routes (/contact etc) are passed to your main index-file.
When you say .htaccess I assume apache. Even so I'd still put nginx in front of apache since it's lightweight and easy to configure (at least compared to the apache behemoth). Sorry, I know that is a very opinionated answer. Anyway with nginx the entire config could look like this:
# usually this file goes in /etc/nginx/conf.d/anyfilename.conf
# but it might vary with os/distro.
server {
listen 80
root /var/www/myapp;
location / {
try_files $uri $uri/ index.html;
}
# And if you want to pass some route to apache:
location /apache {
proxy_pass http://127.0.0.1:81; # Apache listening on port 81.
}
}
I'm sure the same can be achieved with apache alone, but I couldn't tell you how. But perhaps this can be of help: htaccess redirect for Angular routes
There are so many silly toolpacks and utilities I've wasted time learning in my life, but nginx is the one tool I'll never regret I picked up.
I've got a fairly simple Angular2 application in the works, and I'm trying to demo it to my boss on our staging server (typical Ubuntu LAMP). Every time I update a template and redeploy, I also have to clear my browser cache -- and so does my boss, and so do my future users! Not good!
I'm using templateUrl to load in my components' templates, and they are always being cached by the browser. I've tried disabling caching in .htaccess but it only seems to set the headers on the initial request and not for any of the XHR.
Is there a way to configure apache to disable caching for XHR? Or just completely disable browser caching no matter what?
Or am I doing something wrong in Angular? Is it a bad idea to run an Angular app on Apache?
Any advice or ideas would be greatly appreciated!
It was Apache doing the caching, but the problem was simply a misunderstanding on my part. I never cleared my cache once I set Apache to stop caching. I assumed it would happen automatically, which is obviously not the way browser caching works.
So I'll have to clear my boss's cache one last time, and from there on out the problem will be solved.
First time using Backbone's pushState and not getting it 100% working, so help would be appreciated. My needs are:
Must work with deeply nested URLs both navigating to them from the default route and
With direct linking (or page refresh) to deeply nested URLs
Must be able to call my PHP backend API (using Slim Framework) properly with Backbone sync.
I was unable to get all 3 of these things working, although with Nginx rewrites I could achieve #1 and #2.
To achieve #1 I did the standard
location / {
root html;
index index.html index.htm index.php;
try_files $uri $uri/ /index.html;
}
redirecting to index.html which is documented well.
However this does not work with #2. If I go directly to a nested URL like www.example.com/store/item123/subitem345 I would get errors being unable to load my require.js files, which were being looked for at www.example.com/store/item123/ . Naturally this is not right.
I could make #2 work with some rewrite rules that remove the unwanted part of the URL (the /store/item123 part). Is this correct? And if so, is there a universal rewrite to make this work?
I could never get #3 to work fully. Whenever I was in a nested route (e.g. store/someitem123/subitem345), Backbone would append the intermediate parts of the URL to backend API call, which would give a 404 naturally. So instead of the needed (/php/api/args/) I would get (store/someitem123/php/api/args).
There must be a way to either override Backbone's sync function or use an Nginx rewrite to remove the intermediate parts that aren't needed (the store/someitem123 part in my example).
For reference I have to have this block in the Nginx configuration to make the backend calls work at all. But currently they will only work when in at routes that don't have deeply nested URLs.
location /php/ {
try_files $uri $uri/ /php/chs_rest.php?$args;
}
Looks to me like the crux of the issue is route URL rewriting. Based on the Slim documentation for nginx, your conf file is incorrect. See the nginx section here: http://docs.slimframework.com/#Route-URL-Rewriting.
EDIT: Updated to address OP comment below.
In your specific situation, the best recommendation I can make is to split the Backbone.js application and the Slim application into separate apps.
The popular Backbone Wine Cellar tutorial's sample application is an excellent example of this. The Slim portion of the app is a few years old, but it would make an excellent reference for building a current app.
The benefits of splitting the app are numerous, perhaps the largest of which is the fact that each app will be much closer to standard Backbone and Slim applications. The resources at your disposal for learning and problem solving would expand greatly, as blog posts and documentation and SO questions would apply directly to your applications. Future maintenance and continued development will be much easier.
I'm confident that the effort to split the applications would have an extremely high return on investment.
I pushed a new version of my website, but now the CSS and static images are not deploying properly.
Here is the messed up page: http://www.gaiagps.com
Appengine shows the latest version as being correct though: http://1.latest.gaiagps.appspot.com/
Any help?
I've seen this before on App Engine, even when using cache-busting query parameters like /stylesheets/default.css?{{ App.Version }}.
Here's my (unconfirmed) theory:
You push a new version by deploying or changing a new version to default.
While this update is being propagated to all GAE instances running your app...
...someone hits your site.
The request for static resource default.css{{ App.Version }} is sent to Google's CDN, which doesn't yet have it.
Google's CDN asks GAE for the resource before propagation from step #2 is done for all instances.
If you're unlucky, GAE serves up the resource from an instance running the old version...
...which now gets cached in Google's CDN as the authoritative "new" version.
When this (if this is what happens) happens, I can confirm that no amount of cache-busting browser work will help. The Google CDN servers are holding the wrong version.
To fix: The only way I've found to fix this is to deploy another version. You don't run the risk of this happening again (if you haven't made any CSS changes since the race condition), because even if the race condition occurs, presumably your first update is done by the time you deploy your second one, so all instances will be serving the correct version no matter what.
Following is what has worked for me.
Serve your css file from the static domain. This is automatically created by GAE.
//static.{your-app-id}.appspot.com/{css-file-path}
Deploy your application. At this point your app will be broken.
change the version of the css file
//static.{your-app-id}.appspot.com/{css-file-path}?v={version-Name}
deploy again.
Every time you change the css file. you will have to repeat 2,3 and 4.
Your link looks fine to me, unless I'm missing something.
You may have cached your old CSS, and not getting the new CSS after updating it. Try clearing your browser cache and see if that works.
Going to 1.latest downloads the new CSS since it's not in your cache, so it appears correctly to you.
I had this problem as well. I was using flask with GAE so I didn't have a static handler in my app.yaml. When I added it, the deploy works. Try adding something like this
handlers:
- url: /static
static_dir: static
to your app.yaml and deploy again. It worked for me. Apparently Google is trying to optimize by not updating files that it thinks users can't see.
As found by shoresh, the docs for the standard environment for Pyhton state that both settings for static cache expiration, the individual element expiration and the top-level element default_expiration, are responsible for defining "the expiration time [that] will be sent in the Cache-Control and Expires HTTP response headers". This means that "files are likely to be cached by the user's browser, as well as by intermediate caching proxy servers such as Internet Service Providers".
The problem here is that "re-deploying a new version of the app will not reset any caches". So if one has set default_expiration to, e.g., 15 days, but makes a change to a CSS or JS file and re-deploy the app, there is no guarantee that those files will be automatically served due to active caches, particularly due to intermediate caching proxy servers, which may include Google Cloud servers - what seems to be the case since accessing your-project-name.appspot.com also serves outdated files.
The same documentation linked above states that "if you ever plan to modify a static file, it should have a short (less than one hour) expiration time. In most cases, the default 10-minute expiration time is appropriate". That is something one should think about before setting any static cache expiration. But for those who, like myself, didn't know all of this beforehand and have already been caught by this problem, I've found a solution.
Even though the documentation states that it's not possible to clear those intermediate caching proxies, one can delete at least the Google Cloud cache.
In order to do so, head to your Google Cloud Console and open your project. Under the left hamburger menu, head to Storage -> Browser. There you should find at least one Bucket: your-project-name.appspot.com. Under the Lifecycle column, click on the link with respect to your-project-name.appspot.com. Delete any existing rules, since they may conflict with the one you will create now.
Create a new rule by clicking on the 'Add rule' button. For the object conditions, choose only the 'Newer version' option and set it to 1. Don't forget to click on the 'Continue' button. For the action, select 'Delete' and click on the 'Continue' button. Save your new rule.
This newly created rule will take up to 24 hours to take effect, but at least for my project it took only a few minutes. Once it is up and running, the version of the files being served by your app under your-project-name.appspot.com will always be the latest deployed, solving the problem. Also, if you are routinely editing your static files, you should remove the default_expiration element from the app.yaml file, which will help avoid unintended caching by other servers.
Ok For newer people seeing this problem i tried the cache-bursting approach and seem to have fixed it here is an example of what i did for the css import on app.cfg file create a variable to hold your appid as set in app.yaml file and set it as one below
<link href="{{ url_for('static', filename='file.css') }}?{{config.APP_ID}}" rel="stylesheet">
Also for the app.yaml file add this config to be on the safe side
handlers:
url: /static
static_dir: static
Here whats worked for me:
First, I've changed the version on app.yaml.
Then follow these steps below
Go to your console -> Click on your Project.
On the side menu, click on Computation -> Versions:
There it will be all versions, and which version is default. Mine was set to an older version.
Mark the new version.
For me worked. Any concerns?
From the docs for the standard environment for Pyhton: static_cache_expiration.
After a file is transmitted with a given expiration time, there is
generally no way to clear it out of intermediate caches, even if the
user clears their own browser cache. Re-deploying a new version of the
app will not reset any caches. Therefore, if you ever plan to modify a
static file, it should have a short (less than one hour) expiration
time. In most cases, the default 10-minute expiration time is
appropriate.
Make sure to add wildcard at the end of the url and service setup on dispatch.yaml file.
Example:
dispatch:
- url: "example.com/*"
service: default
- url: "sub.example.com/*"
service: subexample
If you use 2020 GCP App Engine, just add default_expiration to your app.yaml file and set it to 1m.
default_expiration: "1m"
More info: https://cloud.google.com/appengine/docs/standard/python3/config/appref/#runtime_and_app_elements
For new people coming to this old questions/set of answers I wanted to give an updated answer. I think in 2018-19 the following information will probably fix most of the CSS update issues people are having:
Make sure your app.yaml has the following:
handlers:
- url: /static
static_dir: static
Run gcloud app deploy
Chill for 10 minutes.. and the shift-reload your website
This approach is suggested by google as well (https://cloud.google.com/appengine/docs/standard/python/getting-started/serving-static-files).
Try clearing cache on your browser. Had exact same issue and got it fixed by simply clearing cache.