I've slightly "abused" the front-end "version" concept in App Engine (java), to implement modules before they were introduced. I have a configuration consisting of: module1-dot-myapp.appspot.com, module2-dot-myapp.appspot.com, module3-dot-myapp.appspot.com, etc., based on the version concept (more commonly used with numbers: 1-dot-myapp, etc.).
Specifically, the code in all versions is identical, but each is practically used for different purposes. This separation allows different clients to use different api versions, separate deployment schedule, staging versions, logs separation, etc.
My question is, under theses conditions, what is the best way to convert my application to "real" modules? such that "module1" is an actual module (still mapped to the same url - module1-dot-appspot.com)?
Note: my answer comes from a somehow similar exercise but in the python GAE runtime, there is likely aditional Java-specific stuff to look at as well.
First things to look at (possible show stoppers) are the app-level configs - those will need to be merged in from your different old app versions (if they exist) and will be shared by all your modules (or directed to the default module only), so they might not work as before, best to revisit the latest documentation on these configs:
dispatch file
queue
cron
DB indexes
Note: in multi-module python apps these configs might not be updated automatically at app upload, each of them may need to be uploaded explicitly, using the respective app configuration utility options.
The separate deployment schedule is almost free (each module can be deployed independently). But there may be some impact due to the app-level configs (multiple CLI invocations instead of a single one, for example)
The logs separation comes for free.
The staging story might need to be revisited, depending on what exactly you mean by that.
Other than that - you'd bring the different old versions of your app in separate module sub-directories in your new app. Check if your version control system supports this easier. The old app config file(s) would need to be "translated" into the respective module's config file(s) and some of the info would go into the new app's top dir config file.
The module URL routing should allow transparent URL mapping, but note that the URLs will actually be <module>-dot-<appname>.appspot.com and the only way to get exactly the same URLs would be to delete all older app versions before deploying the new one (due to conflicting URLs: <module>-dot-<appname> vs <appversion>-dot-<appname>, not sure if you'll get the old or the new code serving or if it's even possible to deploy the new code without error). You could use a new appname at first, just to get all ducks lined up before the switchover (possibly a new staging story you might consider going forward).
You might find helpful complementing URL routing with a dispatch file if you didn't have one before.
Finally, if you have identical files shared across modules you may consider a single per-app copy of the file, symlinked into the respective modules, if that's easier or makes sense from your source code management prospective.
Related
I'm finding inspiration from the twelve-factor app approach to organize the deployment process of small applications. I'm stuck on the release stage on guidelines that sound contradictory.
The twelve factor app says that the config should not be stored in files but in the environment, like environment variables. (I imagine that files sitting somewhere on the host can also serve as "config stored in the environment", such as a ssh private key in .ssh/private_key that will give access to some protected resource through ssh.)
I thus imagine just setting up my various hosts manually by setting those environment variables by hand (or in .bashrc or similar so I don't have to do it again every time they reboot). I only usually have 2 hosts: my laptop for development and a server for showing my work to others. If I had more hosts, I could think of a way to automate this, but this is out of the scope of my question.
Then the twelve factor app guidelines define the release stage as producing a release that contains both the build and the config. This could simply mean sending your build (for example docker images of your app) to the target host. The built app and the target host configuration being at the same place (on the same host), they are de facto combined.
I don't however have any way to uniquely identify a release or have the possibility to rollback. In order to do that, I would have to store the config with the build somewhere so that I can get back to them if I need to. That's where I'm stuck: I can't figure out how one approaches this in practice.
What sounds contradictory is that config should be read from the environment and the possibility to rollback to a previous release, which implies a previous config.
Perhaps the following workflow would be an answer, but maybe convoluted:
send the build to the host,
read the host config (environment variables, etc.) and copy them to make a snapshot of this host's config at that moment,
store both the build and the config copy in a uniquely identified place
Such that when you want to run a particular release on a given host, you :
apply that release config to the host environment
run the build which will read the config from the environment
The step of making a snapshot of the environment's config to apply it again seem somewhat convoluted and I'd like to know if there is a more sensible way to think about the release stage.
I see that there is already an answer to this question but I feel its outdated. Many things have changes since then. There are now modules, cloud_endpoints and webapp2. What should be the good directory structure for my project which allows me to add/modify features easily.
For example I should be able to manage:
Modules.
Cron jobs.
Task queues.
Cloud endpoints.
I'd first take a look at modules, at least for these reasons:
modules really are in many respects (almost) equivalents to entire (single-module) apps in older docs/references, so once a module's position in the app's hierarchy is clarified various posts referencing an app-context can usually be extrapolated to just a module-only context.
nowadays an app can use different languages/sandboxes for different modules (see Run both Java and PHP on google app engine project) or even for different versions of the same module (see Google App Engine upgrading part by part)
Personally I'd stick with the recommended multi-module app structure - each module having its own directory, one level below the app's directory:
The app's top dir would hold the per-app configs (which aren't applicable to a particular module): dispatch.yaml, cron.yaml, index.yaml, and queue.yaml. Note that the cron jobs and task queues definitions belong here (but nothing stops you from routing/dispatching various cron jobs to various modules based on the requested paths).
I'd also place in the app's top level dir any files/directories I'd like to share across multiple modules in the DRY way. These files/dirs would be shared by a module by symlinking them inside the respective modules so that the module gets its own copy at deployment. Almost anything that can exist as a separate file or directory can be shared this way:
templates, images, scripts, CSS, macros, datastore models definitions, python modules - whatever you need
3rd party libs, for example How do I access a vendored library from a module in Python Google App Engine?
even portions of the module's .yaml configuration! for example: Do I need to copy `skip_files` across multiple YAML files?
Finally the recommended files/dir structure of a particular module may further depend on the module language/sandbox, the framework(s) used, the developer's style/preferences, etc. I don't think it's possible to provide a one-size-fits-all recommendation which would be effective/acceptable in all cases.
Endpoints are just RPC (strongly typed) versions of basic REST url's with the added advantage that they can be used to Generated client side libraries. So the endpoint config and definitions belong in the SAME directory as the module (ie mobile-backend) as their REST counterpart would. In other words, if you have (or would have) a REST endpoint in Module1 for "user login", then you should put the "user login" Endpoint in the module1 directory. Further, if you don't like the symlink approach, you can move your module1.yaml file UP one level and then that whole module can import from a "common" directory.
I am migrating from Eclipse to Android Studio and have a Android App connected to AppEngine.
I have split the Server side into two modules (default module for Endpoints and user facing requests) and "admin" module for backend stuff.
Now both these modules need to use the Entities. (backend module usually is responsible for saving these entities to DB, while the frontend default module is the one who returns data back to Android using these Entities).
What is the best way to share these Entity classes between these two modules in Android Studio? (also making sure these classes get enhanced etc). I do not wish to have duplicate classes, both in the default module as well as admin.
Maybe have a common "java" module shared between the two (but not sure class enhancing would work). Or should the admin module NOT use the Entities and instead use other ways of persistence?
Appreciate your thoughts.
While there may be reasons for not sharing the code, personally I prefer DRY.
I solved the issue in DRY spirit with the Python backend by placing the models definition file in the app dir app/models.yaml and sym-linking it into each of the modules subdirs app/module_blah/models.yaml, thus ensuring all modules see the same models definitions. At deployment time the symlinks are automatically replaced with the actual content of the file being symlinked. From appcfg.py update:
The command follows symlinks and recursively uploads all files to the
server. Temporary or source control files, such as foo~, .svn/* are
skipped.
Care may be needed to deploy all modules at the same time.
I used the same technique to also share entire libraries with common code across modules, by symlinking app/lib/libX subdirs into the desired app/module_blah/lib/libX as needed.
Not sure if this technique is usable in Java, tho.
I have a complex appengine service that was written in PHP, now I want to migrate it to Python part by part.
Let's say that my service has 2 parts: /signIn/.... and /data/.... I just want to migrate /signIn/ part first, then /data/ later.
However, since my service is big, so I want to build new /signIn/ part in Python, then use Traffic Splitting to make some A/B Testing on this part.
My problem is that Traffic Splitting can be applied on versions only, so my old and new versions have to be in same module, and same module means that they have to written in same language (I was wrong here, see updated part). But I am migrating from PHP to Python.
What is the best solution for me?
Thanks,
Solution
With Dan Cornilescu's helping, this is what I do:
Split the app into 2 modules: default and old-version.
Dispatch /signIn/ into default module, the rest to old-version module.
Make another version of /signIn/ (default module) in Python
Configure Traffic Splitting to slowly increase requests percent into Python part. This will allow us to test and make sure there is no serious bug happen.
Note:
The /signIn/ part must be default module, since GAE's traffic splitting works at default module only.
I confirmed that we can make 2 versions in different language for a module.
One possible approach is to split your PHP app in modules in a 1st step. It's not a completely wasted effort, most of that will be needed anyways to just allow your app to work in multiple modules, not related to the language change. I suspect this is actually why you can't use A/B testing - mismatch between the modules. Unavoidable.
Once the split in modules is done then you can go on with your 2nd step - switching the language for selected module(s), with A/B testing as you intended.
A more brave approach is to mix the 2 and write the /signin/ module directly in python. On the PHP side you'd just remove the /signin/ portion (part of the earlier mentioned 1st step). Should work pretty well as long as you're careful to only use app language independent means for inter-module communication/operation: request paths, cookies, datastore/memcache keys, etc. A good module split would almost certainly ensure that.
You have testing options other than A/B, like this one: https://stackoverflow.com/a/33760403/4495081.
You can also have the new code/module able to serve the same requests as the old one, side-by-side/simultaneously and using a dispatch.yaml file to finely control which module actually serves which requests. This may allow a very focused migration, potentially offering higher testing confidence.
I'm also not entirely sure you can't actually have 2 versions of the same module in different languages - the versions should be pretty standalone instances, each serving their own requests in their own way using, at the lowest layer, the language-independent GAE infra services. AFAIK nothing stops a complete app re-write and deployment, with the same version or not - I've done that when learning GAE. But I didn't switch languages, it's true. I'd give it a try, but I don't have time to learn a new language right now :)
I'm trying to divide my solution by three configurations:
Development
Testing
Release
All above will have different publishing location, so users can work with release, do their test in testing and see what is new in development release. All three versions will be build with different name postfixes and icons and installed on each user workstation.
For now I get :
Unable to install this application because an application with the
same identity is already installed. To install this application,
either modify the manifest version for this application or uninstall
the preexisting application."
I can't even install this more than once at one workstation.
So What can I do to achive this?
You can not install the same application multiple times unless you change the deployment. The easiest way to do this is by changing the assembly name. This article explains this.
As time past, I can now see that the solution was quite close, just required me to be able to specify my requirements first.
So, now I can tell that it mostly depends on number of such configurations:
if it is limited and low, i.e. live/test/dev, you can have each as separate project in solution, like AppLive, AppTest, AppDev, this requires refactoring to move everything that is common into separate projects, but it makes code and releases clearer and easier to manage.
if those configurations are unlimited, or number is high, than way to go is to load configurations from file and pick one from the pool based on custom logic.
Currently I'm using mix of both, as I want to be able to release test versions earlier than live, but also my application is used by multiple branches, and each of them has some unique styling, logos and such, so this is applied from embed xml file, and proper set is identified based on Active Directory entries.