RPM seems to be pretty good at checking dependencies and handling individual file updates, but what is the best practice for handling cumulative updates to, say, a relational database across multiple versions?
For instance, say you have product Foo with versions 1.2.1, 1.2.2, 1.2.3, and 1.3.0. In each of these, there were database schema changes that required SQL upgrade scripts. Running each upgrade script in sequence is required to get up to the current version of the schema.
Say a customer has 1.2.2 installed and wants to upgrade to 1.3.0. How can one structure the RPM package so that you have the appropriate scripts available and execute the correct upgrade scripts against the database? In this instance, you'd want to execute the upgrade scripts for 1.2.3 and 1.3.0, but not the ones for 1.2.1 or 1.2.2. since those have presumably already been executed.
One alternative is to require upgrading to each intermediate version in sequence, forcing the user in this example to upgrade to 1.2.3 before 1.3.0. This seems less than optimal. Also, this would presumably need to be "forced" through external process, since I don't see anything in the RPM SPEC file that would indicate this.
Are there any known techniques for handling this? A bit of Googling didn't expose any.
EDIT: By "known", I mean "tried and proven" not theoretical.
Use the right tool for the job. RPM probably isn't the right tool. Something like Liquibase would be better suited to this task.
Related
I'm working on a legacy SqlServer database with no version control. I've tried importing it into a VS 2017 database project, but it takes more than an hour to load ("your project will be ready after 1200000 operations are completed"), and usually crashes out in less time than it took to load.
Does anyone have any suggestions for a version control system I can try that will cope with real-life databases?
Baseline your Database and call this Version 0.1.0
As you need to make changes to it, like add columns, data etc. Script this and add this to the source control of your choice. Call this file something like:
Version-0.1.1.sql
As you make more and more changes the amounts of files will be added to.
Version-0.1.2.sql
Version-0.1.3.sql
Version-0.1.4.sql
Of course you will test these before you deploy to live. As you are working with what is a legacy system I would probably shy away from investment in expensive tools for what is a legacy system in the first place.
To bring a database to a particular version you would run the scripts in the order. Obviously in each script you have failover etc that handles anything that may go wrong within the scripts.
It is a manual process but the best points are it's cheap, easily to understand, does not require much expense and it's a methodical system to manage change.
Note: Obviously deploy scripts to a UAT version before directly on Live.
I have had a lot of success using flyway with both sql server and postgres. It allows you to create numbered versions as betelgeuce described in his answer, but also offers additional protection of ensuring your earlier versions haven't been changed before deploying any new changes
If a Chef recipe (or any of it's cookbook dependencies) use the package resource without specifying a version, then the latest version of the package is installed. If you want to control and test exactly what you are installing, then you must always supply the package version. What can you do when the cookbooks that you depend on do not take the same precautions?
See for example the default recipe in the ark cookbook. If this recipe is used on a production server, it could install packages that have not been tested. This is just one example (with over 5m downloads) so I am wondering how people are getting around this problem.
What can you do when the cookbooks that you depend on do not take the same precautions?
I don't think there is a simple answer. This is basically "the Chef way" ...
(Actually, I would suggest that hard-wiring package versions could do more harm than good. One of the good things about using a (good) distribution's package repo is that they regularly release updates with patches for security issues and bugs. But if you wired fixed package versions into your recipes or roles/nodes or something, you would prevent any such patches from propagating to your system.)
However, if it is critical to you that package versions are stable then maybe ...
Clone and hack the cookbooks in question to use specific versions. (Actually, you probably need to do this anyway, to avoid being bitten by unstable cookbooks!)
Use a distro (such as RHEL or its "clones") that values long-term stability, and only pushes out package updates that are "really important".
Create your own private mirror of the distro's package repos with only the "good" versions of the critical packages in it.
Modify Chef so that the package resources pick/install specified versions by default. (I don't imagine this would be easy. But if you did come up with a good solution at this level, it would be a pretty useful addition to Chef! IMO.)
UPDATE
Actually, there is a way to do this for (at least) Debian-based systems; see the apt cookbook and in particular the references to "pinning".
Or with yum, you could "lock" particular versions using "yum versionlock ..." as described here: https://www.zulius.com/how-to/yum-install-specific-package-version/
UPDATE - 2
Another possible trick would be to "inject" a version attribute into the "unsafe" package resources. Something like this:
# first, include_recipe a recipe that specifies 'package "foo"' without
# a version attribute
# then ...
r = resources("package[foo]")
r.variables['version'] = "1.2.3"
With a little ingenuity, one could create a "package version lock" recipe that pulled the versions from a databag, and dealt with missing resource exceptions and version attributes that were actually provided. But I don't know if this is "A Good Idea" (tm).
Chef's package resource uses the package manager of the node's operating system (like apt, yum, etc). These tools always install the most recent version that is available through the repositories. That's why chef's package resource also installs this version.
What the ark cookbook is that it downloads the source code and then compiles it - obvious that you can specify the version to install (through the passed URL).
So it depends on your actual need. If you want to install the version that is available through the distro's or your own package repo, then it's totally fine (and that's what most cookbooks do). If you want to compile everything from source (where you usually have the option to specify the version, the coverage of chef cookbooks supporting this is lower.
Personally, I'd suggest that you set up an own apt/yum/whatever repo for the software for which have specific version requirements.
In short : I'm not managing this.
In a more complete answer:
All distro/release go throught a validation phase before releasing new packages, I'm confident over it and it helps me keep in sync with security fixes.
As far as I know all package managers takes care of not upgrading a package in a breaking way if it is a dependencies of a package installed manually, again you have to trust the package maintainer about this.
i.e.: the package ressource without version won't update make nor gcc if it is a dependency of one package you installed with a fixed version.
For exemple under ubuntu, if you set the nagios package to manual, it will never try to update the libc package over a breaking change, and so I could break other package isntallation as dependencies are not satisfied.
If you're absolutely concerned about it, you have some choices:
Rewrite each pacakge ressource to use fixed version of packages
Fork any cookbook to fix thooses issues (you can write a foodcritic rule to help you detect package ressources without specific versions)
Have your own repos, stable and testing, and move packages inside stable repo once tested and use testing repo on your staging/QA environement.
The 3 is the most conservative as you choose what is in the repo in the stable branch and it won't change magically. The drawback is security fixes you'll have to manage.
Hope it would help.
We had the same issue in our cookbook. So we decided to use data bags.
Data bags can be easily changed, for example:
knife data bag from file my_data_bag host1
OR
knife data bag edit my_data_bag host1
Your recipe will be able to see the specified version from the data bag using the code like this:
my_bag = data_bag_item('my_data_bag', 'host1')
Chef::Log.info("You have changed the version to: #{my_bag['version']}")
package 'java' do
version my_bag['version']
action :install
end
So finally you don't need to modify Cookbook or Recipe. All you need is to pass the version to the data bag.
In creating a package with Data Types and Items, what are the best practices to follow for subsequently upgrading a package in order to retain the data that's created on the site while adding a change to the associated Type?
Should the package name remain the same or should it include version information to retain uniqueness by version? The version number of the package does not seem to enforce any type of upgrade policies.
Currently during development the package is uninstalled then the newer package is subsequently installed however after the package reaches production I believe this type of upgrade workflow will not be sufficient without affecting the associated data.
As long as you increase the package-version but keep its unique GUID and Name, you're able to install the same package again, forcing C1 to overwrite the existing files, configuration etc.
This has a few caveats though, mainly if you need to delete some old files or configuration entries during installation, since that's what you normally do during un-installation. But you can still do it, writing your own installaton-steps code.
It is classes inheriting from some core base package-classes that you can invoke during installation. It can contain logic like checking if certain other packages are installed, do custom special logic, or clean up from earlier upgrades.
But yeah, long story short, as long as the version number increases you can "upgrade" a package, by installing the newer version, while the older one is still installed.
Since I've not done this before I am not sure if the way I am planning to do this is okay or is there a better way. Like using Windows Installer or Install Shield or Windows Installer XML (WiX) toolset. Any help would be great, as I have no clue.
We have a product and we ship new version every few months. So far we've only been rolling out complete versions i.e. Either Version 1.0, or Version 1.5, but no upgrade from 1.0 to 1.2 to 1.3 to .... you get the picture, right! So any customer that get version 1.0 cannot upgrade to version 1.2 or 1.3 or even the latest. They'll have to uninstall old version and install the latest version. This is not right, but thats what we could do until now. But we'd like to change it.
My plan is to have a install file with (Sql Scripts) for each upgrade path. Check the table in database that stores the version info and depending on it run different script to upgrade database.
My concern is that this method may not be scalable, once we have more than 5 or 6 different versions.
If you could point to any articles or books on this topic, that would help a lot too.
Also, could we use Windows Installer or Install Shield for this?
thanks,
_UB
We've been using DBGhost for a year or so now to keep our database under source control along with our codebase, and it makes this kind of thing dead easy. It's not just well thought through, but they've been using it to roll out their own code for years, so it's dead solid.
Your problem is a pretty common one, and I've had to deal with this kind of problem at my last job. There is another tool aside from the RedGate tool that may help you do what you need to do. It's a tool called DB Ghost. They explicitly address the versioning problem, and have a packager as well. I would suggest doing a trial of the DB Ghost product because they have some interesting claims concerning multiple version upgrades. This was taken from their FAQ (http://www.innovartis.co.uk/faqs/faqs.aspx):
Q: Our problem is going to be managing
data structure changes during
upgrades. Our product line is
Shrink-Wrapped, or downloadable from
the website. So when a user downloads
an upgrade, they can be upgrading from
a very recent version, with few
database structure changes, or the
upgrade may be from a very old version
with a multitude of structural
changes. One upgrade needs to manage
it all. The user would be offsite, so
we can't hold their hand. We have
users in Greece, Australia, Malaysia,
Norway, etc. How would DB Ghost, if at
all, handle updates in remote
locations?
A: The DB Ghost Packager Plus product was
design to specifically address this
issue as it can dynamically handle the
required updates to a target database
seamlessly.
I'm just mentioning this because our company is trying to do something similar and I was doing research on this tool.
Thanks,
Eric
Do you insist on doing it yourself, or could you see yourself committing and investing in a tool?
I really like the idea of Red-Gate's SQL Packager, which will "diff" your two database versions, and then create a SQL script, a C# project, or a stand-alone executable to upgrade from version 1 to version 2.
Not 100% how you'd be able to upgrade from 1.0, 1.1, 1.2, 1.3 all to 2.0 - check out their website and see if they offer something for that scenario!
Otherwise, I guess it'll get quite thorny and messy......
Marc
In the Rails world they are using a tool/method called Migrations.
Basically is boils down to creating a small sql script to upgrade and downgrade each little change to the database.
When you are testing the application you migrate your database to the version you want and on deployment the application can check what version it needs and migrate to that version.
There are free migration toolkits for most popular languages, they might be part of some MVC framework though.
A nice side effect of migrations is that you have database source code that is easily stored in you source control repository.
I'm working at a company that does several releases to production every year and during the build up to each release we gather up a collection of 1 time sql install scripts like table creation and dataports.
The way things currently work is that after the release to production, we branch, tag then we delete all 1 time scripts from subversion.
This seems to get the job done but to me it never seemed like the proper way to solve the problem.
Could you imagine deleting all your sourcecode every release and then writing patches for production?
The downsides that I see is if you want to reference and old script you have to checkout a tag or branch from subversion.
Our SVN Repo currently looks something like this
svnrepo/mywebsite/src
svnrepo/mywebsite/database/storedprocs
svnrepo/mywebsite/database/installscripts
I was thinking that a more accurate way to model what we want to do in SVN is the following.
Use an svn:externals attribute to point to the latest version. Then after every release just point it to the latest.
svnrepo/mywebsite/trunk/src/
svnrepo/mywebsite/trunk/src/database/installscripts/
-> svnrepo/mywebsite/trunk/database/Release_3
svnrepo/mywebsite/trunk/database/Release_1
svnrepo/mywebsite/trunk/database/Release_2
svnrepo/mywebsite/trunk/database/Release_3
Using this model we no longer svn delete any sql scripts and enable a database developer to check out svnrepo/mywebsite/trunk/database/ and easily view all the database development that has occurred.
Any comments on my ideas, the current structure, or the best way to manage this situation?
Thanks
Synchronising database changes and code changes in subversion is hard
If you have the option of building the Database from scratch you can put the whole DDL into the repository along with the code, then you don’t need to worry about which changes go with which release.
Looking at your situation I don’t think you need to use externals (they can cause headaches). You also don’t need to delete everything. It is not too difficult to check out a branch (or you could just use a repository browser).
You could even put the old db releases into a separate tag when you release so they are all in one place, which the database people can have checked out. If you are doing releases once a year this won’t be hard.
This question may also help