I'm building a website with a few volunteer developers located around the world. We're using a HostGator shared hosting plan, so we get cPanel, FTP, etc. but no on-server versioning.
For good development practices, it'd be nice to have a test version of the site available with a versioning system behind it. However, we're all on networks behind NATs, so even if we use a centralized versioning system (GitHub, etc.), we still have to pull changes down to a local server (ie. a 192.0.0.x private SQL/HTTP server) for testing between versions.
Are there any (free-preferred) systems available for versioned test sites? I've been avoiding VPS because of cost and maintenance complexity.
Are there any (free-preferred) systems available for versioned test sites?
Because
Test site is "site with production code, used in test environment"
Versioned site is the same object as "versioned code"
any SCM, which can be used for managing some code, can be used for this task. Web-development doesn't have any remarkable differences from any other software-development with VCS behind the scene: only build&deploy process have changes, because you
have to compile nothing in case of Web
must to deploy to some destination in order to run|test|use development's results
Create local test-environment (for every developer in team (?) separately), use any SCM (DVCS can be better and easier somehow, maybe) of choice and appropriate *-hooks of your SCM (or handwork) in order to deliver changed code from working space|repository to test site
Related
Twist to the standard “SQL database change workflow best practices”
Background
ASP.NET/C# Web App
MS SQL
Environments
Production
UAT
Test
Dev
We create patch scripts (XML and sql) that are source controlled in Mercurial. We have cmd line utility that installs patches to DB (utitlity.exe install –patch) from a Release folder the build packages. Patches have meta data that helps with when patch should run and we log patches installed in a table in the target DB. All these were covered in the 3 year old question:
SQL Server database change workflow best practices
Our Problem/Twist
I think this works well for tables, views, functions and stored procedures. We struggle with application configuration data. Here are some touch points on application configurations.
New client. BA performs system study and fit analysis. Out of this comes a configuration word document of what application configurations need to be setup. Note some of these may also come in phases over time. We need to get these new configurations into the system for the developer and client UAT.
Developer works on feature request or bug fix. A new configuration change comes out of that change. The configuration needs to make it into the system for testing and promotion to UAT and up.
QA finds that the developer missed an associated configuration change. That configuration needs to make it into the system for promotion to UAT and up.
Build goes to UAT. Client performs acceptance testing but find they really want to change another unassociated configuration and have it promoted with the changes. In other words they found they want to change a business process by a configuration. The configuration needs to make it into the system for promotion to PRD.
As the client operates in PRD they may tweak application settings. These configurations need to make it into the system for future development and testing.
The general issue is making sure we are accounting for all the configurations and accidently not miss any during promotions which causes grief.
Our Attempts At A Process
a. We have had member of the QA team to write patches (xml and sql) and check those in. This requires a build to make sure those get into the package. With this approach it really just took care of item 1 above and we fell apart on the other items. The nice thing is for the items that made it into the patches it was just an install with the utility.
b. A developer threw together a Config page on the application. All the configurations could be uploaded and downloaded via XML document but it requires the app to be running. For item 1, member of QA team would manually setup configurations in the application and then would download the Config.xml file. This XML file would be used to upload configurations in other environments. We would use text diff tool to look at differences between config.xml files from different environments. This addressed item 1 and the others items but had problems. Problems were not all configurations made it into the XML document (just needs to be fixed by developer), some of the configurations didn’t have a UI in the application so you still had to manually go to the database on some, comparing the XML document with text diff was difficult at time (looked mostly due to sorting but I’m sure there are other issues), XML was not very human readable and finally the XML document did not allow for deleting existing incorrect or outdated configs.
c. Recently we went with option B, but over time for a new client we just started manually tracking configs and promoting them manually by hand (UI and DB) through the promotions. Needless to say lots of human errors.
So we have been looking at solutions. Eventually it would be great to get as much automation in as possible. I’m looking at going with the scripting approach and just focusing on process, documentation and looking at using Redgate data compare in addition to what we had been doing with compare on config.xml. With Redgate we have to create views though and there is no way to create update scripts from that approach except to manually update the scripts. It does at least allow a comparison without the app running. I’m also looking at pulling out the configs from our normal patches and making it a system independent of the build (utility.exe –patch –config). When I say focus on process it will be things like if we compare and find a config change either reported by client or not, we still script it, just means we have to have a process in place to quickly revalidate config install before promoting to the next level. As for documentation looking at making the original QA document a living document instead of just an upfront document. The goal is to try and enhance clarity and reduce missing configurations during promotion. Unfortunately it doesn’t improve speed of delivery.
Does anyone have any recommendations or best practices to pass along. Thanks.
Can I ask exactly what you mean by application configuration. I'm interpreting that as both:
Config files in the web application
Static reference data inside the database
Full disclosure I work for Red Gate. You might be interested in taking a look at Deployment Manager, it's a deployment tool that deploys applications, databases and configuration. It's free for up to 5 projects and target servers.
The approach it uses is to package application code and the database state into packages. These packages can be deployed into dev, test, staging and production environments. The same package is deployed to each environment.
Any application configuration that needs to change between environments is handled in one of the ways below:
Variable substitution in web.config. The tool allows you to specify override values for variables in these files, and set these per environment/server
Substituting the web.config file per environment.
Custom powershell scripts that are run pre/post deploy. You could use these to execute custom SQL based on the environment or server.
Static data within the database, using SQL Source Control's static
data feature. I've written a blog post about how to supply
different sets of static data to different environments/customers.
This allows you to source control the application configurations and deploy them to different environments.
This is a subject of common discussion, but through all my research I have not actually found a sound answer to this.
I develop my websites offline, and then launch them live through my hosting account.
I utilize codeigniter, and on that basis there are some fundamental differences between my offline and online copies, namely base urls and database configurations. As such I cannot simply develop and test my websites offline and then upload them as it requires small configuration changes which are easy to overlook and good lead to a none working live website.
The other factor is that when I am developing offline, I might add a database table or a column whilst creating some functionality. When I upload my local developments to my host, they often do not work as I have forgotten to upload the new database structure. Obviously this cannot happen - there cannot be any opportunity for a damaged or broken live website.
Further to this, I'd like to be able to have logs of my development - version control of sorts such that if i develop a feature, and then something else stops working I can easily look backwards to at least see the code changes which could have caused the change.
My fourth requirement is as follows: if i go away on holiday for a week without my development laptop, and then get a bug report, I have no way of fixing it. If i fix it on the live copy, not only is it dangerous, but i'll inevitably not update it on my local copy - as such when i update my live copy next time, that change will be lost. Is there a way that on any computer i can access my development setup, edit and test, launch to the live site, whilst also committing it such that my laptop local copy is up to date.
So yes.. in general im looking for a solution to make my development processes more efficient/suitable. Any ideas?
Thanks
Don't deploy by simply copying. Deploy by using a script (I use Apache Ant) that will automate the copy of specific files for each environment, the replacement of some values, etc.
This just needs rigor. Make a todo list while developing, and check that every modification on the server is done. You might also test the deploy procedure on a pre-production server which has an similar configuration as the production server, make sure everything is OK, and then apply the same, tested procedure on the production server
Just use a version control system. SVN or Git are two free candidates.
Make your version control server available from anywhere. If it's an open-source project, free hosting solutions exist. Of course, if you don't have a development computer wvailable, you'll have to checkout the whole project, and probably install some tools to be able to develop, test and deploy. Just try to make it as easy as possible, or always have your laptop available. If you plan to work, have your toolbox with you. If you don't plan to work, then don't work. When you have finished some development, commit to the server. When you go back to your laptop, update your working copy from the server.
Small additions and clarifications to JB
Use any VCS, which can work (in a good way) with branches - your local and prod systems are good candidates for separate branches, where you share common code but have branch-specific config. It'll require some changes in your everyday workflow (code in "test", merge finished with "prod", deploy /by tools, not hand/ only after merge...), but it's fair price
Changing of workflow, again. As JB noted - don't deploy by hand, don't deploy wrong branch, don't deploy "prod" before finished merge. But now build-tools are rather smart, you can check such pre-condition inside builder
Just use VCS, maybe DVCS will be somehow better. I say strong "No-no" for Git as first VCS, but you have wide choice even without it - SVN (poor branch|merge comparing to DVCS), Bazaar (not a tool of my dream, but, who knows), Mercurial, Fossil SCM, Monotone
Don't work on live, never do anyting outside your SCM. One source of changes is a rule of happy developer. Or don't work at all at free-time, or have codebase always reacheable for you (free code-hosting /GoogleCode, SourceForge, BitBucket, Github, Assembla, LaunchPad/ or own server), get it as needed, change, save, deploy
I'm in the process of reengineering a desktop application to a Silverlight4-WCF client-server architecture. Apparently, some (not all) of our clients have very limited resources or dysfunctional relationships with their I/T support staff. Consequently, I've been asked to come up with a solution that would enable these clients to install the new SL app on a local desktop, possibly running IIS locally. Will probably wrap all this up in a installer to make deployment super easy.
While I don't endorse the idea of running IIS locally on each user's desktop, my thinking is that it will probably work. The biggest problem I envision is security -- the server code and configuration would be available locally. Still, I'm looking for input from the wider developer community because I'm uncomfortable by the proposal. What kind of hell and difficulties do you envision, or can this work with minimal fuss?
Using IIS locally may be more trouble than it's worth. Most configuration/maintenance tasks require the user to have Administrator access on the computer. Securing it is a bitch. Making sure its configuration doesn't drift and remains working as you release new versions is even more so. Moreover, it's not included in all Windows editions, e.g. Windows 7 Home edition. I recommend trying to roll with IIS Express as an embedded HTTP server instead.
As for code security - if you can't trust your clients' IT to look after deployment and security, or they can't/don't want to, you could host the server-side yourself and bill your customers for it, i.e. SaaS.
Just my 2 cents.
EDIT:
Oh, yeah. If you're concerned about trade secret theft, e.g. reverse engineering of your server code, just run it through an obfuscator, e.g. SmartAssembly. That'll give you an edge. As for IP theft, e.g. piracy, use could use some online license checking scheme - it won't compromise client privacy, and it'll deter copying.
I just heard that a company I do work for may be bringing in the Pyxis Mobile application development system. When I google it most of what I find is from the company's web site and that is not very informative from a geek perspective. Can any one shed some light on what sort of programming environment it is and what programing language is involved (please let there be a text based language). Any additional information would be great.
Note: the company/product changed their name to Verivo in January.
Full Disclosure - I work as an engineer at Pyxis Mobile. However, I have been in the mobile space for 7+ years and have evaluated several approaches to mobile so hopefully this is helpful.
Pyxis Mobile provides a set of tools and components to build cross platform mobile applications. Let me outline them first.
1. Application Studio - All application development, backend integration, user provisioning and application maintenance/debugging is done w/in this tool. Application Studio (for now) is a Windows based desktop app.
2. Application Clients - Pyxis Mobile provides native client runtimes for iPhone, iPad, BlackBerry, and Android devices. These runtimes get branded for the customer through a build service and are primed to point to a specific Application Server URL.
3. Application Server - Pyxis Mobile App Server runs on the .NET stack (on IIS). All client communication is proxied via this server. This server is able to connect to varied of backend systems (via the Plugin Framework listed below) and respond to the client in a mobile optimized manner. This server needs a SQL Server (2005 or newer) for configuration access, session management, logging and more.
4. Plugin Framework - The Plugin Framework is a backend component that provides system specific pre-built access to several of the enterprise and cloud based systems (Oracle, Siebel, SAP, Salesforce.com, social feeds, REST/SOAP web services, etc.) and also offers an API layer in .NET and Python (using IronPython) to allow even further customization. A plugin is essentially comprised of one or more DLLs or a Python file. These assets are then dynamically loaded to normalize communication between Pyxis Mobile and the customers' backend systems.
5. Push Services - This provides a cross-platform push layer that can poll a backend system for change and alert a mobile device via BlackBerry Push, Apple Push Notification Services (APNS) or Android's Cloud to Deice Messaging (C2DM).
6. OverWatch Analytics - This is an optional (but included) component to track users/devices and provide integrated analytics on what the users are using and what kind of devices and locales makes up your users.
The application itself is "coded" via configuration that is build in App Studio. Pyxis Mobile abstracts away from the code so that you can work at a higher level without having to worry about the wide array of device variances (GPS, touch screens, camera, accelerometer, push, screen resolution, etc.). You can drag fields onto a from, connect screens via menus or buttons, set up caching rules and more in this graphical utility. This configuration (essentially think of an XML like document) is interpreted by the native client layer to produce a rich application. There is also a scripting layer in Lua that allows to really customize behavior via code.
The real value of Pyxis Mobile comes up when you have change to make. The clients check for new configuration at app startup or if the server forces the client to get new configuration. This gives you great agility. Lets say once your application is deployed you want start using the swipe gesture to go next/prev through a set of records. This change on other platforms would mean writing some platform specific code to trap and interpret the swipe to perform a navigation (you couldn't trap a swipe on a non-touch screen). However, in Pyxis Mobile this is a simple configuration change that can be quickly deployed to the App Server and the clients automatically download and use the new configuration. No compilation, no redeployment or re-download for the end users.
I could keep going, but hope this provides some level of guidance.
Beware of Pyxis Mobile. While many of the things they say do work, there are some serious platform issues (as a geek) which I've experienced.
1) No version control system process. The Application studio can basically only be developed on by one person at a time or you risk having your changes overwritten by a fellow developer. The "principle of last save" is very much in play.
2) No unit test coverage. This isn't the biggest issue for a lot of people, but it's a concern for anyone who wants to work in the Enterprise world.
3) The middleware server gets you some value, but it's also a PITA to work with. There is no concept of "client side storage" unless you consider the middleware server the client side. If your phone goes out of coverage, your app won't work. Again, this might not be an issue for you.
4) The application has no true scripting language to work with. The middleware server allows you to intercept requests and responses and modify what you're doing there, but it's not the most elegant solution considering that a native application can have something as simple as "if this then X else Y." This can be accomplished with Pyxis, but the whole process is convoluted and more complicated than one would think it needs to be.
5) Lack of documentation. There's some training guides and the GUI is easy enough to get around for simple apps; however, when you need to do something with guts, you're left relying on Pyxis professional services. There's really no developer community to pose questions to.
I have more complaints, but they are more opinion oriented than Q/A oriented.
I just got note about the most recent comments. I don't want to turn this into a thread of back and forth, but did want to throw a couple of quick notes.
Regarding the points on version control and documentation/developer community - no big contest there. We are definitely working on these shortcommings. We have some basic pieces in place, but we have big plans to focus on this.
Regarding unit testing - we provide a very open interface to our middleware and backend components and they can be very easily unit tested with a bit of instrumentation. We run a ton of unit and integration tests internally. However, mobile unit testing is extremely difficult to get right. We'll investigate this further.
Regarding #4 around middleware and offline capabilities - things are a lot different now. With version 7.1, 7.2 and 7.3 our products have increasing become more capabale offline and now features a secure local database if necessary. I can provide more details as necessary, but you can certainly login and operate the app even if you are out of coverage for weeks at a time!
Regarding #5, we've had a scripting engine for over 2 years. Its Lua based and its actually quite powerful and fast. It was BlackBerry only till the most recent release. Given Apple's change of stance on allowing scripting we now allow scripting on BlackBerry, iPad, iPhone and Android as well now!
#RockMeetHardplace - feel free to reach out to me directly and I'll be happy to give you more detailed live demos of our latest platform. I am at - arunSPAMNOTatpyxismobiledotcom (drop the "SPAMNOT" and replace the at and dot). I happen to be the Director of Software and interested in knowing more about the issues you had.
I would like to get a general consensus for the minimal/boilerplate professional PHP team development environment. I can not find this information anywhere on the web. In the open-source world there are so many choices and so many ways to do things but I've yet to find any common best-practice for the infrastructure/plumbing side of things.
Consider a small shop with a team of 5-10 developers/designers, doing LAMP CRUD apps.
They need to manage development, staging and production builds. They want quality software and they can't be stepping on each others toes trying to get things done. Deployment needs to be easy and fast. Sometimes there will be hotfixes. Rolling back production server to a previous version needs to be just as fast.
Things to consider are:
Source code management (SVN, git,
Hg)
Database schema/data continuous
integration, tied to Source-code
revision. This is one I'm
particularly interested in.
Individual development environments
(e.g. each developer has a VMware
instance of the development
environment to tinker with (DB
server, web server, code, data,
etc))
Managing central development,
staging and production builds
Production deployment (e.g. tar
balls, .rpm/.deb)
Automated testing (e.g. SVN commit
hooks, nightly cron tests for slower
tests)
Team communication (bug tracking, internal documentation,
irc/im, etc)
I've left this open to edit by the community so feel free to edit/add. Ideally someone can visit this page and a few hours later have the foundations in place for their team to start developing.
I'll start. feel free to edit and improve this
This is for a ficticious product called: dundermifflin.com
Setup a development virtual machine running the same software you plan on using in production: e.g. Ubuntu with PostgreSQL, Apache and PHP5.
Each developer runs their own copy of this VM with the hostname set to their username, (e.g. phpguy.dundermifflin.com)
Setup a central staging server (same as the development VM). This is staging.dundermifflin.com.
Setup a central Subversion server with a new repository for dundermifflin.com. This is devel.dundermifflin.com.
4a. Add post-commit hook to run tests for "trunk" commits
4b. Add post-commit hook to package/deploy to staging server for commits tagged "staging"
4c. Add post-commit hook to package/deploy to production server for commits tagged "release"
This method does not address database continuous integration which means rolling back SVN to a previous revision will break the build unless your database is very static. Suggestions?
Use Bugzilla on the central Subversion server (devel.dundermifflin.com) for bug tracking.
Write a shell script to run PHPUnit/SimpleTest tests (to be called by item 4a).
For continuous integration, linked with your version control system, and automated unit testing I find this article very interesting:
Continuous builds with CruiseControl, Ant and PHPUnit