I want to take on a project, but I’m not sure how to handle the updating process.
Normally, when asked to update a site, you back-up the database & site files, then make the updates locally or on a development server. Then when the updates are finished, you push them live.
My problem is that the site I’ll be working on registers new members every day, makes blog posts every day, and gets new comments on those posts every day. If I were to pull the site on Monday, update it in a testing environment, then push those changes live on Friday, every member who signed up and blog entry written during the week would be overwritten.
So what’s the best way to go about doing this? How do I update/add features to a site without losing the data gained on the live site during development? Surely it must be possible, since high-traffic sites like TechCrunch and Gizmodo make huge sitewide updates all the time without losing data.
It depends on what changes you're making. Is it file/template changes or database changes?
If it's just file changes, just pull the files and database to your local server, make changes to your files and then just push them (files only) to the live server when done. As long as no database changes have happened, that will work.
If there are db changes, things get a bit trickier. You would basically follow the same process, but make note of any db changes you are making on the local site. when everything is ready to be pushed to the live server, you have no other option but to take the site offline for users while you update.
You would then push all updated files to live server, and mirror any db changes you did on the local server (install/update plugins etc). When all that is done and tested, you can then put the site online again. Downtime should be minimal if you have made good notes on db changes.
This is dependant on being able to block access to users but still allow access for yourself, but that's standard with most CMSs.
Also, if you dont already you should look at integrating git into your workflow. If the changes you'll be making take a considerable amount of time you'll need a system in place where you can branch your code off into new versions while still keeping the original state of code that's on the live server.
That way, if there is an urgent fix that needs doing to the live site while your in the middle of developing new features locally, you can switch back to your master/original branch and make changes to the code that doesn't include any of the new stuff you have been working on the other branch.
Well, I've only done this for small traffic wordpress/drupal sites, but not having a "live" version hasn't been an issue for me. I have my development copy, make test the changes I want, and then roll those changes out to the live site on the fly by FTPing' them back up.
Are you going to be editing these registrations? Or are you just tweaking static files?
In the case of wordpress, I test a plugin out, and then just install it on the live site.
Typically the changes I'm making involve plugins/modules and some PHP stuff. This is obviously not the most nuanced solution, and I'm interested to see what more knowledgeable people have in mind.
Related
I have a dev site and a live site for my Django app. A lot of front end copy and other object details are stored in the DB. Conversely, a lot of client data is also stored in the DB. What's the most efficient way to get the dev site ready and then copying over the new copy and objects to the live server?
I guess the easiest is to keep the changeable dev site data up to date with the live site, then update the dev site, then do a data dump over to the live site. But what happens if/when someone updates the live site whilst this is happening? Their update will be overwritten.
Is there a Django app to allow me to 'check' entries in the admin list view, press 'copy' then somehow paste these into the new site?
Or what are the other patterns people use for this common scenario?
It's a bit like using GIT - having a master branch, then a branch for each live site iteration, then creating hot fixes on the live site, whilst still working on the dev site — the hotfix can be merged into the dev site. How do we merge the data?
I'm not sure I get it correctly, please tell me if I'm wrong: so basically, you want to make changes on the (same) data both in production and in your dev instance ?
IMHO it is generally not a good solution for many reasons, one of those being this fragile need to merge you mentioned. To me, the best way to approach this would be in fact to keep the two environments as detached as possible (data-wise).
The part to avoid at all costs, IMHO, is this final dev->production merge. I can understand a little better the need to transfer in the other direction (from prod to dev). I can see two common cases here:
You want the same data in dev. to try to reproduce a data-related bug. In that case, why not simply dumping the production database and load it on the dev. machine ? IMHO, this dev. database should be considered "wipeable" all the time.
You have some "initial" data on the production site that is generic enough to be also needed in dev (something like Male/Female entries in a "Gender" table). In that case, you can imagine using Fixtures.
While thinking about you data flows and environments, you should probably also have a look at South, data migration is a very useful tool.
Hope this helps !
Is there a Django app to allow me to 'check' entries in the admin list view, press 'copy' then somehow paste these into the new site?
Not that I know off. But I think you could write such an app with a reasonable amount of effort. I would suggest the following:
Use admin actions for the change list view. That way you could publish or transfer multiple objects from your development system to your live system. See the documentation for further information.
Add another button to the change view of a single object. Maybe "Publish to Live site".
Add yet another button to the change view that saves and publishes in one step, like "Save and publish to Live site".
To add buttons to the change form I would override the corresponding template:
http://code.djangoproject.com/svn/django/trunk/django/contrib/admin/templates/admin/change_form.html
Override the block 'submit_buttons_bottom' like
{% block submit_buttons_bottom %}
{{ block.super }}
<!-- your buttons here -->
{% endblock %}
Preferably you would write just one function that does the job of transfering the object from the dev to the live system. Use this function in all three use cases.
But there is a catch!
It could be fairly hard to copy one object from one Django instance to another. Especially so, if those objects have relationships with other objects, which in turn might have relationships with yet another object. Retaining referential integrity should be a nightmare. Maybe someone has a nice solution for that.
I modified a page layout in our QA environment using Apex and didn't refresh from the server (my last refresh from the server was 10 days ago). Someone else made a TON of changes using the web interface in between then. I just overwrote all of her changes!
Does anyone have any advice on how I can undo my changes and go back to hers?
Unfortunately there's no tracking of layout changes so there's no way you can just revert this — are there any other developers who may have had a later copy of the metadata and haven't refreshed theirs yet?
It can be a good idea to use ANT to run a metadata backup system so that things like this don't cause problems :) There's a guide to using the Force.com Deployment Tool here, essentially you just want to use half of that process on a regular schedule.
following on from this question:
GWT detect GAE version changes and reload
I would like to further clarify some things.
I have an enterprise app (GWT 2.4 & GAEJ 1.6.4 - using GWT-RPC) that my users typically run all day in their browsers, indeed some don't bother refreshing the browser from day to day. I make new releases on a pretty regular basis, so am trying to streamline the process for minimal impact to my users. - Not all releases concern all users, so I'd like to minimize the number of restarts.
I was hoping it might be possible to do the following. Categorize my releases as follows:
1) releases that will cause an IncompatibleRemoteServiceException to be thrown
and 2) those that don't : i.e. only affect the server, or client but not the RPC interface.
Then I could make lots of changes to the client and server without affecting the interface between the two. As long as I don't make a modification to the RPC interface, presumably I can change server code and or client code and the exception won't be thrown? Right? or will any redeployment of GAE cause an old client to get an IncompatibleRemoteServiceException ?
If I was able to do that I could batch up interface busting changes into fairly infrequent releases and notify my users a restart will be required.
many thanks for any help.
I needed an answer pretty quick so I thought I'd just do some good old fashioned testing to see what's possible. Hopefully this will be useful for others with production systems using GWT-RPC.
Goal is to be able to release updates / fixes without requiring all connected browsers to refresh. Turns out there is quite a lot you can do.
So, after my testing, here's what you can and can't do:
no problem
add a new call to a RemoteService
just update some code on the server e.g. simple bug fix, redeploy
just update some client (GWT) code and redeploy (of course anyone wanting new client functionality will have to refresh browser, but others are unaffected)
limited problems
add a parameter to an existing RemoteService method - this one is interesting, that particular call will throw "IncompatibleRemoteServiceException" (of course) but all others calls to the same Remote Service or other Remote Services (Impl's) are unaffected.
Add a new type (as a parameter) to any method within a RemoteService - this is the most interesting one, and is what led me to do this testing. It will render that whole RemoteService out of date for existing clients with IncompatibleRemoteServiceException. However you can still use other RemoteServices. - I need to do some more testing here to fully understand or perhaps someone else knows more?
so if you know what you're doing you can do quite a lot without having to bother your users with refreshes or release announcements.
I am currently running a SL3 project where we are in a highly iterative development mode with about 25 active test customers. I am making small changes at a clip of about 4 new builds per day. It is important to know this application is mission critical line of business for these 25 people, it is the tool they use all day to do their work so they are using it constantly and often launch their browser and the app in the morning and never close it until the end of the day.
The challenge is that when I make an update to the application I have no clean way to notify the users, in most cases this is ok as it is rare that I introduce a data contract change or something that would be a classic 'breaking' change to the app/service. Users keep plugging along and will get the change next time they refresh.
Right now we have resorted to emailing everyone and telling them to force refresh or close the browser and log back in.
Surely there is a better way...
Right now my train of thought is to have a method on the server that compares client xap versions and determines if the client being used is the most up to date, if so I will notify the user and make them update.
What have you done to solve this problem?
One way of doing it is to use a push mechanism (I used Kaazing Websoocket Gateway but any would do). When a new version of the XAP is released a message (either manually entered into the system by admin or automated triggered by XAP file change event) would be sent to all the clients. In the simplest scenario some notification would be shown to a user (telling him that a new version is released and the application needs to refresh) and then the app would refresh (by simply reloading the page) saving user's state if necessary.
If I would do this I would just keep it simple. A configuration value in web.config and a corresponding service method that simply returns that value (the value itself could be anything, but a counter is probably wise). Then you could have your Silverlight app poll that service method at regular intervals. Whenever the value changes (which you would do manually when you deploy a new version), just pop up a dialog telling the user to refresh the browser or log in/out. This way you don't have to force them to refresh every time. If you go with the idea of comparing xap file versions they will always be required to refresh, even for non-breaking changes.
If you want to take it further you could come up with some sort of mechanism to distinguish between different severity levels. For instance, if the new config value would contain the string "update_forced", you could force the users to reload the app by logging them out automatically (a little harsh, perhaps). If it contains the string "update_recommended", just show a little icon at the top right corner saying that there is a new version and that they should upgrade in their own time.
Granted, this was targeted at Silverlight 3, but with the PollingDuplex client and such in the newer versions of Silverlight, you could publish an "Update Now" bit to the clients, and build a mechanism in the client to alert the user that there is an update that is now out... that they should update it shortly, etc. You may even be able, through serialization and such, to save the state that they are in when they close the app to reload it.
We've done stuff similar with a LOB app that we built, so that as users are changing things, the rest of the userbase sees those changes immediately. Next up will be putting the flags in to change authorization and upgrades "on the fly" if you will.
Assumption: live/production web app suppresses errors being shown to end-users.
Suppose your tech support team wants to see live data but through the eyes of the development-side of the application (maybe you want to see what errors are occurring, or want to see when you've got an issue fixed using an end-user's data).
Right now we've got one database serving both the dev and live boxes (not my idea - I know it's gross).
Ideas?
Edit: Best/handy tools for implementing your suggestion?
We replicate the data back to a different database. Yes, there is a delay, but it keeps people hands out of the production servers. This also allows us to "hide" information that tech support (and other people for that matter) aren't supposed to see.
In addition to replicating data down, on production, we see who's logged into the application, and if it's a member of the company, send them to the real error page versus the happy kitten playing with a ball of yarn apologizing.
Back up and restore from live to dev on a regular basis (once, twice a day). It doesn't need to be realtime (as you might be entering data from the dev side anyway, which could cause problems).
If you have PCI or HIPAA data, make sure you don't put that in your dev environment -- that might break laws.
I generally like to have a 3-tier system for web development:
Development
Testing
Live
Most of the time testing is an exact copy of the live system, except that errors are turned on, when a new version is about to be moved live it's replaced with the new version BEFORE live is, to detect upgrade issues.
Development is completely separate from live, to allow for major changes to things like the database, or changes to the production environment.
I would firstly make errors are either emailed to someone with details of how the user got there or at minimum logged so you can watch the error log while you perform similar actions to see if you get the same messages in the log.
And yes, copying the database on the dev server/site is probably your only option. You don't want any changes made by the development team to live data and you'll probably also have changes that won't work with the production database at some point.
I wouldn't recommend doing a nightly copy as a developer might be in the middle of some new feature where they have added data and then it's erased that night. I usually copy the production database(s) to dev each time a major version is released. This also allows me to do speed testing with a lot of live data. On some systems I also change everyones password to a default so I can login easily as any user.
If your configuration permits it:
a. Add a logging function (if there isn't one already) to write messages of interest to a log file.
b. Run the unix command
tail -f < logfile.txt
which will stream the growing log file to your console.
http://www.monkey.org/cgi-bin/man2html?tail
If you have Windows, you might try this:
http://tailforwin32.sourceforge.net/