i've just finished upgrade in my local env (hybris 1811 to 2105). After compiling and deploy the new version in qas, we've noticed that some things don't work as they should. When i deploy in the portal i selected the option 'no migrate data'. I don't know if it could be because of that. Another thing to keep in mind, the update in the hac is frozen.
When doing an upgrade, you always need to perform it with data migration
Data migration does all required database changes, new types, new properties,... Definitely with an upgrade like that, a lot of new types and properties are added. Not running that will cause a lot of errors in your logs.
To fix this, you can deploy the same build, but with data migration enabled this time (update system)
Related
I have a flask app that is deployed on Google's App Engine. I have noticed a minor bug and I would like to fix it but my database is already populated.
How can I make this minor code change and push / deploy back to my app without losing all my data? (which is probably a basic question but I'm not finding much. all tutorials online are focused on creating the app and deploy, not updating)
Thus far, I have been dropping and re-creating the tables whenever I redeploy, mostly out of ignorance. Here are the steps I have followed
1). make the change in my app
commit and push changes to bitbucket source code
in Google Cloud SDK: git pull
Google Cloud SDK: gcloud app deploy
These steps result in an empty database because the directory I am pushing from my local computer has an empty database. Is this where I should be using git merge?
Is this a database "migration" or is this a "git merge"? I'm not sure what the right terms are to use to research this further. Thanks.
There are a couple of angles to your question. I'm going to try to give you some information, but let me warn you, this isn't going to be a trivial change to your workflow, you'll have to change some things.
First of all, based on the way you worded your question I get the idea that you commit your database to git along with your code. If I got this right, then this is something that you need to stop doing. The database is not code, so it should not be committed to source control.
You should have a completely independent database on each installation of your application. For example, you will have a database on your own machine to do development. You will also need another database in your gcloud deployment. You may need more databases if you have other uses for your application. A very common third database for many people is one that is used for automated tests, which could also be located in your local development machine, but is not the same database that you use for day to day development.
To make changes to your database schema you will not drop and recreate tables anymore, that is clearly something that you already realized that needs an improvement. A good approach to make these changes is to use a database migration framework. These tools allow you to generate short scripts that make these changes to the database in a more focused way, without destroying and recreating everything, and for that reason, the data is in general not lost. For Flask-SQLAlchemy, the best option for database migrations is Flask-Migrate, which is a lightweight wrapper around the Alembic migration framework. (I might be biased here as I'm the author of the Flask-Migrate extension!).
Documentation for Flask-Migrate: https://flask-migrate.readthedocs.io/en/latest/.
I'm thinking of a good deployment strategy for magento. I already have managed to deploy code with git from my local installation to my stage server. (The jump to live is not a problem then)
Now I'm thinking about how to deploy backend changes like the following:
I'm adding a new attribute set and I want it to be available on my stage and later the live server. Since these settings are in the database, I could just do a mysqldump and restore this dump on my stage/live systems.
But I can't do this, since the database has more data like orders, articles (with current stock availability) and a lot more stuff which I don't want to deploy from my testing system.
How are others handling this deployment "problem"?
After some testing, I chose the extension Mageploy, which is nicely to install via modman (I prefer modgit which relies on the same data for installation) and already captures a lot important backend settings.
If you need more, it's possible to extend it to more backend settings by yourself (and then contribute to the git project. Pullrequests are concidered quickly)
I'm working on an OpenCart project. (Note: this is my first time dealing with it.)
I want to somehow implement my usual workflow of:
working on localhost, experimenting, etc,
deploying the changes to the production server (sometimes to a staging server before that),
adding the database changes.
Now, how should I achieve this?
What I already did with GIT is I created an automated deployment flow, which consists of the following:
building a deployment version (Checking out master/HEAD's upload/ directory, and removing the upload/install directory.),
copy the upload/ dir's contents to the target server.
This work fine, but won't solve the database migration issue.
I think it's not even as simple as updating certain tables in the target server's database from my local database, since for example: the "settings" table contains data that's specific to the environment.
So I can't just overwrite the settings table with my local version.
It seems to me, that the easiest - and ugliest - solution would be to develop on the prod server in parallel to the localhost changes. So for example: If I install a module, which causes changes in the database, then I would need to replicate every step I took in the local environment installing and placing that module. Same goes for every admin setup I take. (Meta changes, etc.)
This sounds awfully painful to me, so I hope there's a better solution out there other than doing every database-related change twice...
Thanks in advance!
Twist to the standard “SQL database change workflow best practices”
Background
ASP.NET/C# Web App
MS SQL
Environments
Production
UAT
Test
Dev
We create patch scripts (XML and sql) that are source controlled in Mercurial. We have cmd line utility that installs patches to DB (utitlity.exe install –patch) from a Release folder the build packages. Patches have meta data that helps with when patch should run and we log patches installed in a table in the target DB. All these were covered in the 3 year old question:
SQL Server database change workflow best practices
Our Problem/Twist
I think this works well for tables, views, functions and stored procedures. We struggle with application configuration data. Here are some touch points on application configurations.
New client. BA performs system study and fit analysis. Out of this comes a configuration word document of what application configurations need to be setup. Note some of these may also come in phases over time. We need to get these new configurations into the system for the developer and client UAT.
Developer works on feature request or bug fix. A new configuration change comes out of that change. The configuration needs to make it into the system for testing and promotion to UAT and up.
QA finds that the developer missed an associated configuration change. That configuration needs to make it into the system for promotion to UAT and up.
Build goes to UAT. Client performs acceptance testing but find they really want to change another unassociated configuration and have it promoted with the changes. In other words they found they want to change a business process by a configuration. The configuration needs to make it into the system for promotion to PRD.
As the client operates in PRD they may tweak application settings. These configurations need to make it into the system for future development and testing.
The general issue is making sure we are accounting for all the configurations and accidently not miss any during promotions which causes grief.
Our Attempts At A Process
a. We have had member of the QA team to write patches (xml and sql) and check those in. This requires a build to make sure those get into the package. With this approach it really just took care of item 1 above and we fell apart on the other items. The nice thing is for the items that made it into the patches it was just an install with the utility.
b. A developer threw together a Config page on the application. All the configurations could be uploaded and downloaded via XML document but it requires the app to be running. For item 1, member of QA team would manually setup configurations in the application and then would download the Config.xml file. This XML file would be used to upload configurations in other environments. We would use text diff tool to look at differences between config.xml files from different environments. This addressed item 1 and the others items but had problems. Problems were not all configurations made it into the XML document (just needs to be fixed by developer), some of the configurations didn’t have a UI in the application so you still had to manually go to the database on some, comparing the XML document with text diff was difficult at time (looked mostly due to sorting but I’m sure there are other issues), XML was not very human readable and finally the XML document did not allow for deleting existing incorrect or outdated configs.
c. Recently we went with option B, but over time for a new client we just started manually tracking configs and promoting them manually by hand (UI and DB) through the promotions. Needless to say lots of human errors.
So we have been looking at solutions. Eventually it would be great to get as much automation in as possible. I’m looking at going with the scripting approach and just focusing on process, documentation and looking at using Redgate data compare in addition to what we had been doing with compare on config.xml. With Redgate we have to create views though and there is no way to create update scripts from that approach except to manually update the scripts. It does at least allow a comparison without the app running. I’m also looking at pulling out the configs from our normal patches and making it a system independent of the build (utility.exe –patch –config). When I say focus on process it will be things like if we compare and find a config change either reported by client or not, we still script it, just means we have to have a process in place to quickly revalidate config install before promoting to the next level. As for documentation looking at making the original QA document a living document instead of just an upfront document. The goal is to try and enhance clarity and reduce missing configurations during promotion. Unfortunately it doesn’t improve speed of delivery.
Does anyone have any recommendations or best practices to pass along. Thanks.
Can I ask exactly what you mean by application configuration. I'm interpreting that as both:
Config files in the web application
Static reference data inside the database
Full disclosure I work for Red Gate. You might be interested in taking a look at Deployment Manager, it's a deployment tool that deploys applications, databases and configuration. It's free for up to 5 projects and target servers.
The approach it uses is to package application code and the database state into packages. These packages can be deployed into dev, test, staging and production environments. The same package is deployed to each environment.
Any application configuration that needs to change between environments is handled in one of the ways below:
Variable substitution in web.config. The tool allows you to specify override values for variables in these files, and set these per environment/server
Substituting the web.config file per environment.
Custom powershell scripts that are run pre/post deploy. You could use these to execute custom SQL based on the environment or server.
Static data within the database, using SQL Source Control's static
data feature. I've written a blog post about how to supply
different sets of static data to different environments/customers.
This allows you to source control the application configurations and deploy them to different environments.
I'm working at a company that does several releases to production every year and during the build up to each release we gather up a collection of 1 time sql install scripts like table creation and dataports.
The way things currently work is that after the release to production, we branch, tag then we delete all 1 time scripts from subversion.
This seems to get the job done but to me it never seemed like the proper way to solve the problem.
Could you imagine deleting all your sourcecode every release and then writing patches for production?
The downsides that I see is if you want to reference and old script you have to checkout a tag or branch from subversion.
Our SVN Repo currently looks something like this
svnrepo/mywebsite/src
svnrepo/mywebsite/database/storedprocs
svnrepo/mywebsite/database/installscripts
I was thinking that a more accurate way to model what we want to do in SVN is the following.
Use an svn:externals attribute to point to the latest version. Then after every release just point it to the latest.
svnrepo/mywebsite/trunk/src/
svnrepo/mywebsite/trunk/src/database/installscripts/
-> svnrepo/mywebsite/trunk/database/Release_3
svnrepo/mywebsite/trunk/database/Release_1
svnrepo/mywebsite/trunk/database/Release_2
svnrepo/mywebsite/trunk/database/Release_3
Using this model we no longer svn delete any sql scripts and enable a database developer to check out svnrepo/mywebsite/trunk/database/ and easily view all the database development that has occurred.
Any comments on my ideas, the current structure, or the best way to manage this situation?
Thanks
Synchronising database changes and code changes in subversion is hard
If you have the option of building the Database from scratch you can put the whole DDL into the repository along with the code, then you don’t need to worry about which changes go with which release.
Looking at your situation I don’t think you need to use externals (they can cause headaches). You also don’t need to delete everything. It is not too difficult to check out a branch (or you could just use a repository browser).
You could even put the old db releases into a separate tag when you release so they are all in one place, which the database people can have checked out. If you are doing releases once a year this won’t be hard.
This question may also help