I need a bit of help/advice with versioning with SemVer.
I'm working on a client's website who has sent several related amends both large and small to his site in a word document (like they always do).
I have a branch based off my master branch for these new amends, and have created commits for each completed amend I have done so far.
The idea was that I would complete all of the amends and then release them in the next release (v2.0.0) because I think all of these changes are related and all of them combined are significant enough to warrant a bump in version number.
The issue I have is that the client wants a few of these amends to be made live immediately, before the release of 2.0.0, so what would the best way of handling this be - would I upload these few completed amends into the existing version and increment the minor number, or would I bump it up to 2.0.0 even though all of the amends aren't complete?
I am a bit of a noob when it comes to versioning, but am trying to learn as best I can by reading and trying to make sense of Semantic Versioning site.
You should always consider these two things:
What the real changes are? If there are no visible changes, and/or if there are no major changes underneath, it may be better to avoid to increment the major version number.
What the customer should perceive of your changes? Saying version 1.1 or version 2.0 may make some difference in how the changes are perceived.
So if the modifications are limited and/or there are no visible things that have changed, it may make sense to increment the minor only and then wait for all of them to be complete to bump it to 2.0.0.
Related
For developers that use Trunk-based development, how have you approached dealing with an ever-growing collection of feature flags in your codebase?
My concern is if you are heavily leveraging feature flags for every release and every new feature, wouldn't the amount of feature flag code start to make the code less readable and possibly harder to maintain? For the sake of this question, assume feature flags are being handled by an external FFaaS.
Based on my own reasoning I can see a few options:
Never delete feature flags. Keep them around in case you may need them (ie for sunsetting a feature you are phasing out at some later date).
Periodically remove old feature flags that have remained on for X amount of time. This solves the code readability issue, but this breaks the trunk-based deploy paradigm as you lose out on the fallback measure of turning on/off a flag since the flag itself is being removed. You may also lose out on the above case where you'd have to manually track down functionality to phase out, or maybe re-introduce feature flags back in to facilitate some similar transition.
How have people handled the logistics around using this development system?
Regarding best practices and efficiency within Flink, what are the recommendations of when to split analytics into multiple tasks
For example, given a single topic in Kafka as the source of the data. If there were many simple operations that were to be carried out over the stream, such as: if some value is greater than x, or if x & y etc. What would be the point at which you would stop spending more rules into the same task and start to run them in parallel?
Is there any official recommendation for this?
It's hard to give a general recommendation. Performance-wise, it makes sense to put as much as possible into one job.
However, it's much more important to think about maintenance. I'd put everything in one job that is closely related, such that new features or bug fixes will likely only affect one job while at the same time, you also don't want to stop all analytics when upgrading one particular query.
Another dimension to think about is state size. It's related to restarts and update frequency (point above). If the state size becomes too big, restarting this one monster job takes a long time, which would be inefficient if you only touched a small fraction of the code.
Finally, it also depends on the relevance. If some part of your job is super important as it reflects the one KPI that drives your business, then you probably don't want to mix that with some fragile, unimportant part.
So we use the Gitflow process for a safe and sound CID
(https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)
But something's wrong.
Let's say we have 10 critical bugs on prod - they must be fixed asap.
So 10 bugs are assigned to 10 devs. Dev 1 creates a hotfix 0.0.1, commits the fix to it, fix gets uploaded to staging env and handed to QA for verifying. In that time dev 2 wants to create hotfix 0.0.2 for his bug, but he can't as Gitflow forbids creating new hotfix if one exists already.
So he either have to wait for closing hotfix/0.0.1 or commit to the same hotfix. Of course the normal choice is to commit to that hotfix/0.0.1, because his bug is critical and cant wait fix for bug 1.
So does every other 10 devs with their 10 critical bugs.
QA verifies bug 1 and confirm for deployment. But hotfix can't be closed and deployed, because the other 9 bugs are not tested or just not working
So to be closed, every bug in that hotfix must be working and confirmed.
In our case this takes time, a lot of time - 1,2 weeks, luckily - as more and more things needs to be fixed asap on production. So devs commit to that branch more and more, getting the first bugs to be put on hold even more.
So eventually, after all those bugs are fixed and confirmed it is time to finally close the hotfix and deploy those critical bugs on prod...with quite of a big time delay. But there is also another big problem - THE MERGE with dev branch, where other devs do their work (like minor bugfixings for next release). A horrible, several-hours lasting merge.
So we obviously are doing something wrong - what could be a solution for this? For our hotfixes to be on time and to not have terrible merges to dev.
One solution we were thinking, is to put a limit in the hotfix for the number of bugs which it should contain - for example 5 bugs max, other bugs must wait until those are fixed. This, though, means that only lead-dev should commit to hotfix, as he must ensure the limit (it is hard to control otherwise). But this way bugs are still put on hold and fast-deployment is not achieved.
What would be the normal procedure? Please share your ideas.
If you are spending several hours on a merge, then you are probably doing it wrong. One approach is to get a version control expert in for a day to train your team. That is a relatively small cost compared to salaries. I should think you could find a very good person for £500 ($600 US) per day, and you might only need them for a day or two.
I wonder also whether your testing (with a QA team) is too manual. Can bugfixes be accompanied by unit tests to prove they are an improvement? If so, the developer should be able to merge master into their bugfix branch, spin up a new instance for some simple QA checking, get QA to check the system, get a team lead to PR the fix and unit tests, then merge straight into master and redeploy on its own.
Also, make sure your deployment to live is fully automated. Doing several live (small) releases per day is a good target to have.
Updates
The advice above still stands, but since you have now said this is an old project:
Get some automated tests in place. Try to set up functional/browser tests for the bulk of the project, so you can have confidence in changes.
Don't reject unit tests out of hand. Perhaps they could be just for new code or new changes, at least to start with? Yes, the project is old, but don't let yourself off the hook easily. Unit tests will pay dividends in the long run. If you don't have dependency injection in the project, get that added, even if it is not used for all instantiation immediately.
Repeated from above: make your releases smaller.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 5 years ago.
Improve this question
We have a small scrum-team which develops a webseite with a lot of users. We would like to improve our development and relase/deployment process, but there are a lot of concepts out there, which in our eyes don't fit together perfectly. Since we're not the only company developing a website, I thought it might be a good idea to ask here ;)
First of all, the software is not in good shape, so things like continuous delivery or continuous depoloyment are not possible because of bad test coverage and that's nothing we can change soon.
We have two-week sprints, so we develop new features or resolve bugs in that period, after the final sprint meeting we merge the feature branches into master (we use feature branches in git and pull-request for review and merging), do some testing, and deploy master to a public beta enveriment. There we usually find some bugs, resolve them and after that we deploy master inckuding beta fixes to the production enviroment. After that we start with our next sprint.
But that process is far from being perfect. First of all it's difficult to show features with all those branches in our sprint review meeting, and because we only have one master branch which gets deployed, we can't easyly deploy hotfixes to different enviroments. Sometimes we need more time for testing in our beta enviroment, so we can't merge new features to master, or same for production, once deployed, we can't deploy hotfixes if we are already testing new features on beta. If we expect bugs in beta or production due to larger changes, development has too keep feature branches for a longer period, so merging feature branches later on becomes more and more painful.
First we thought about long running branches, like master for development, production and beta. So we could merge whatever feature we want into one of the thre branches. But we really like to work with pull requests (reviews, feedback and deleting feature branch after merging), it's really nice to work with, but we can only apply a pull-request branch to one other branch. So here, we could merge to master without deleting the branch, and have to swithc to another tool to merge a feature to beta or production, or create new pull requests for beta and production. It works that way, but its not such a nice workflow as only merging to one master branch.
We also thought about git-flow (Vincent Driessen's branching model), which looks nice, but it seems to be better suited to software with traditional release cycles and versioning, not 100% for web aplications, which have no real versions but deploy everything which is ready after a sprint. It solves the hotfix problems, has an extra develop branch, but it requirs release-versions. So we could create a release branch, get problems sorted out, release it to production and delete the release branch. We could open a pull request for merging into master with the release branch, but problems start if we want to release it to beta (we use capistrano for deployment), because the branch changes every sprint. And what if we want to test features in our beta enviroment? We could use the release branch for beta, but this way a release to production has to wait until all features in the release/beta branch are ready. If testing of large features takes long, we can't upload small updates to production.
Next problem is versioning. We use jira and jira likes to use versions for releases. We could use versions like "1,2,3"..., but should one sprint be one version? Doesn't feel right, because sprint planning is not the same as release planning, or should it be the same when developing a web application? Because sometimes we develop featurs in a sprint, which take longer and are released later after being completed in the next 1-2 sprints. With git-flow, this changes could not be merged to develop until they are ready, because every release is branched from develop branch. So this way, some feature branches are not merged for a long time and merging them becomes more and more difficult. It's the opposite of continuous integration.
And last but not least, the deployment and QA process does not fit to scrum so very much, we don't have an own web operations team, when a sprint is ready, the product owner has to review the stories, we have to test them, deploy them to beta/production, and have to fix the problems in it immediately, which also interrupts or next sprint. We are not sure when is the right time to merge the feature branches? Before review meeting? This way we could have merged features which are not accepted by the product owner into the branch which should be released soon, but if we don't merge them, we have to demonstrate each feature in it's own branch and merge every accepted feaure branch, so integration and testing could only start after the sprint review meeting. Integration and testing could take days and needs development resources, so when should the next sprint start? After release to production? But this way we could not start a sprint every two weeks, and not every developer is needed for integrating and QA, so what should they work on? Currently we start the planning meeting for the next sprint immidiatly after the review meeting of the last sprint, but if we do it this way we are not sure how many time we need for the release which draws resources from the sprint...
So how do you release web applications?
What workflow do you use?
What's the best way if we want to integrate pull requests in the workflow?
How do you integrate QA and deployment into the scrum workflow?
I try to make a backlog and prioritize it:
Prio 1: Refit you Definition of Done.
I think you are betraying yourself how many features you can develop in a sprint.
Scrums defines the result of an sprint as "useful increment of software".
What you do is to produce a "testable increment of software".
You have to include the test into the sprint. All stories that are not tested and have a "production ready" stamp on it, are simply not DONE.
Prio 1: Do your retrospectives
Get the team together and talk about this release mess. You need a lightweight way to test stories. You and your team knows best what's available and what is hindering you.
If you already doing those retrospectives, this one is done. ;)
Prio 1: Automated Tests
How can you get any story done without UnitTests?
Start making them. I would propose that you search for the most delicate/important component in the system and bring test coverage (ECLemma/JaCoCo helps) up to at least 60% for this component.
Commit a whole sprint to this. I don't know your DOD, but this is work that has been procrastinated and has to be redone.
If a manager tells you its impossible, remind him of the good old waterfall days, where 2 weeks of development have not raised his eyebrow, why should it now...
Prio 2: Integration testing
You have Unit-Tests in place and you do some additional manual testing in the sprints.
What you want is to include the integration test (merging with master) to the sprint.
This has to happen, because you have to(!) solve the bugs in the sprint they have been introduced. So a early integration has to happen.
To get there it is important that during a sprint you work by priorities: The highest prio task/Story has to be solved by the whole team first. Goal is, to have it coded and unit tested, then you hand it over to your tester for smoke testing. Bugs found have the prio of the story. When it works good enough this part is deployed to the integration and tested there.
(Of course most of the time, not all team members can work effectively on 1 story. So one other story is processed in parallel. But whenever something stops your Prio1 Story, the whole team has to help there.)
Prio 2: Reduce new features increase Quality
To include more quality you have to lower expectations regarding amount of new features. Work harder is not what scrum tells you. It tells you to deliver more quality because this will speed up your development in mid term.
Regarding QA and deployment, I suggest the following:
Commit your code early to the main branch, and commit often. I am totally unfamiliar with git so I can't comment on how you're using it but I've always worked off the main trunk (using Subversion) with all the developers committing new code to the trunk. When we create a release we branch the trunk to create a release branch.
If you haven't done so already then set up a continuous integration system (we use TeamCity, but it'll depend to some degree on your stack). Get what unit tests you do have running at each commit.
Create a definition of 'done' and use it. This will stop you from being in the situation of testing features implemented during the previous sprint. Make sure your definition of done includes the passing of acceptance tests that have been co-written by your Product Owner, in my opinion the PO should not be rejecting features during the Sprint Review.
Lighttpd, nginx and others use a range of techniques to provide maximum application performance such as AIO, sendfile, MMIO, caching and epoll and lock free data structures.
My collegue and I have written a little application server which uses many of these techniques and can also server static files. So we tested it with apache bench and compared ours with lighttpd and nginx and have at least matched the performance for static content for files from 100 bytes to 1K.
However, when we compare the transaction rate over the same static files to that of G-WAN, G-WAN is miles ahead.
I know this question may be a little subjective but what techniques apart from the obvious ones I've mentioned might Pierre Gauthier be using in GWAN that would enable him to achieve such astounding performance?
Following G-WAN server for years, I have read the (many) talks covering this question on the old G-WAN forum.
From what I can remember, what was repeatedly addressed were the program:
architecture (specific comparisons were made with nginx, lighty and cherokee)
implementation (how overall branching, request parsing and response building were made)
lean common path (the path followed by all types of requests: dynamic, static, handlers)
Pierre often mentionned other servers to explain what in their specific architecture and implementation was slowing them down.
As time goes, since G-WAN seems to stack more and more features (C# scripts support, a reverse-proxy and a load balancer are expected with the next version), it seems that those 3 points above are more and more important.
This is probably why each new release of G-WAN seems to be willing to be faster than the previous: the more work you do, the more extra fat must be eliminated because its cost gets higher. And like for a race car or a plane this is an incremental process, one calling for more of the other.
If you are looking for the 'secret' of G-WAN's speed then I guess that here is the key point. But if you want more details then you should rather talk directly to the G-WAN author.
Check out G-WAN's timeline. An update on August 8, 2011 might give you idea on what he is using.
G-WAN Timeline
Pierre mentioned that G-WAN uses it's wait-free Key-Value store a lot on G-WAN's core functions. Which gives it more speed since there's no locks being used.
He also uses a Lorenz Waterwheel inspired technique to handle threads. I am not sure how it works but he said that it allows G-WAN to run faster in every possible case.