Please check the answer and comments of my previous question in order to get a better understanding of my situation. If I use Google DataStore on AppEngine, my application will be tightly coupled and hence loose portability.
I'm working on Android and will be using backend which will reside in the cloud. I need client-cloud communication. How do I build an application maintaining portability. What design patterns, architectural patterns should I be using?
Should I use a broker pattern? I'm perplexed.
Google AppEngine provides JPA based interfaces for its datastore. As long as you are writing your code using JPA APIs, it will be easy to port the same to other datastores (Hibernate for example also implements JPA).
I would ensure that the vendor specific code doesn't percolate beyond a thin layer that sits just above the vendor's APIs. That would ensure that when I have to move to a different vendor, I know exactly which part of code would be impacted.
It u really want to avoid portability issues use google cloud sql instead. If u use the datastore unless its a trivial strucfure you sill not be able to trivially port it eve if you use pure jpa/jdo, because those were really not meant for nosql. Google has particularifies with indexes etc.
Of course sql is more expensive and has size limits
In order to maintain portability for my application, I've chosen Restlet, which offers Restful web apis, over endpoints. Restlet would help me to communicate between server and client.
Moreover, it would not get my application locked in to a particular vendor.
Related
We are currently looking into replacing one of our apps with possibly an ESB or some similar tool and was looking for some insights into how best to approach this.
We currently have a stand alone service that consumes/interact with different external services and data sources, some delivered through SOAP Web Services and others we just use a DB connection. This service is exposed through SOAP and we have other apps that consume this service but are very tightly coupled to it, now we also have other apps that need to consume some of the external services and would like to replace this all together with an ESB or some sort of SOA platform.
What would be the best way to replace this 'external' services integration layer with an ESB? We were thinking of having a 'global' contract/API in which all of the services we consume are exposed as one single contract where all the possible operations and data structures that we use are exposed under one single namespace, would this be the best way of approaching this? and if so are there any tools that could help us automate this process or do we basically have to handcraft this contract/API?. This would also mean that for any changes to the underlying services/API's we will have to update this new API as well.
If not then the other option I see is to basically use the 'ESB' as a 'proxy' layer in which all of our sources are exposed as they are, so we would end up with several different 'contracts' / API endpoints, but I don't really see the value in this.
Also given the above what would be the best tool for the job? is a full blown ESB an overkill or are we much better rolling our own using something like Apache Camel or Spring Integration?.
A few more details:
We are currently integrating over 5 different external services with more to come in the future.
Only a couple of apps consuming our current app at the moment but several other apps/systems in the future will need to consume some of these external services.
We are currently using a single method of communication (SOAP) between these services but some apps might use pub/sub messaging in the future, although SOAP will still be the main protocol used.
I am new to ESB integration so I apologize in advance if I'm misunderstanding a lot of these technologies and the problems they are meant to solve.
Any help/tips/pointers will be greatly appreciated.
Thanks.
You need to put in some design thoughts of what you want to achieve over time.
There are multiple benefits and potential pitfalls with an ESB introduction.
Here are some typical benefits/use cases
When your applications are hard to change or have very different release cycles - then it's convenient to have an ESB in the middle that can adopt the changes quickly. This is very much the case when your organization buys a lot of COTS products and cloud services that might come with an update the next day that breaks the current API.
When you need to adapt data from one master data system to several other systems and they might not support the same interfaces, i.e. CRM system might want data imported via web services as soon as it's available, ERP want data through db/staging tables and production system wants data every weekend in a flat file delivered via FTP. To keep the master data system clean and easy to maintain, just implement one single integration service in the master data system, and adapt this interface to the various other applications within the ESB plattform instead.
Aggregation or splitting of data from various sources to protect your sensitive systems might be a use case. Say that you have an old system that can take a small updates of information at a time and it's not worth to upgrade this system - then an integration solution that can do aggreggation or splitting or throttling can be a good solution.
Other benefits and use cases include the ability to track and wire tap every message passing between systems - which can even be used together with business intelligence tools to gather KPI:s.
A conceptual ESB can also introduce a canonical message format that is used for all services that needs to communicate. If a lot of applications share the same data with several other applications (not only point to point) - then the benefits of a canonical message format can outweight the cost (which is/can be high). An ESB server might be useful to deal with canonical data as it is usually very good at mapping from one format to another.
However, introducing an ESB without a plan what benefits you are trying to achieve is not really a good thing, since it introduces overhead - you need another server to keep alive, you need perhaps another team to understand all data flows. You need particular knowledge with your integration product. Finally, you need to be able to have some governance around it so that your ESB initiative does not drift away from the goals/benefits you have foreseen.
You should choose some technology that you are comfortable with - or think you can be comfortable with. Apache Camel is indeed very powerful and my favorite integration engine - but it's not an ESB as it does not come with a runtime that you can use to deploy/manage/monitor your integration services with. You can use it together with most Java EE application servers or even better - Apache ServiceMix (= Karaf+Camel+ActiveMQ+CXF) which is built for this task.
The same goes with spring integration - you need to run it somewhere, app servers or what not.
There is a large set of different products, both open source and commercial that does these things.
we have a few various applications that stores its data and we need one common service which provides access to these data.
With the applications I mean for example Atlassian Jira, Confluence, SVN, Git, LDAP, few internal mysql databases, etc. Some of them offers you SOAP API, REST API, various command line clients, for some you have to directly access database to get data.
What we want is a common REST API interface, to access all possible data sources. Of course, we have to solve authentication and authorization, caching and many more tasks.
It seems that something like ESB - Enterprise service bus and EIP - Enterprise integration patterns is answer to our needs.
For start, we are playing and actually dig in to Apache Camel - it's not full EIP stack, it's "just" a integration framework. But I guess it's good enough for us right now.
My question is - what you mean about the solution? Are we on the good way?
Thanks!
Camel has a lot of connectors, so that would be a great start.
If you are afraid it is too thin, then take a look at Apache ServiceMix, which provides a deployment (OSGi) container for camel routes (and other things). Camel comes bundled within the standard service mix release out of the box.
The hard task is probably to design the generic API good enough to cover your use cases.
A GIT repo and a Database are very different, is this very generic? Do you only want to access "text" data or something?
I like the approach with camel non the less, since it's rather generic and flexible in these kind of scenarios. That you will need
If I for some reason wanted to use my own private server to run my GAE app instead of deploying it to Google servers, could I simply use AppEngine SDK to do that? Are there any technical or legal limitations or problems that I should know about? I know that the SDK isn't designed to run heavy pages but for small websites it should work right?
The dev_appserver is not built to serve public-facing traffic. It's single-threaded, not built for performance, the backends are likewise limited, and it exposes administrative endpoints (such as /_ah/admin/) which users should not be able to access.
Alternatives exist for hosting your app yourself, such as TyphoonAE and AppScale.
Well, you could try. I'm not aware of any legal issues (with the caveat that I Am Not A Lawyer, and can't offer to interpret the license agreement for you), but I doubt you'd be happy with performance.
For one, the dev_appserver Datastore emulation (in both the Python and Java SDKs) doesn't scale all that well once you're past small amounts of data. It's not designed for performance; it's designed to accurately mimic the semantics of the real Datastore.
What is the current status of grails and google app engine deployment. I am new to app engine but wonder worth exploring it. Some specific qns are
the latest plugin, which has high user rating, has any restrictions? or it work seamlessly with all gorm features
is there any issue with high startup time for grails application. How is it in real world scenario? (with a typical small and large scale application)
what about other grails plugins (like, shiro, joda time, nimble etc). I guess they wont play well. So using those libraries directly is the better option
If decided to give up goole-app as a deployment option, how easy to switch to a normal environment. The JPA support ensures the compatibility with other traditional DBs?
Not sure what else are major issues.. probably, this is the foundation for a good discussion.
thanks.
I got few good response from grails mailing list, and the conclusion shares the comment by David. see the thread here
Couple of relevant responses:
From Tomas Lin:
I would suggest looking into Gaelyk if you really want to build a
project on the App Engine. It is built from the ground up with the App
Engine as the target engine, so it can bypass problems like long
loadtimes due to Spring and Hibernate. The newly introduced plugin
mechanism guarantees that your Gaelyk applications can be extended in
a way guaranteed to work on GAE.
Gaelyk has it's own native entity persistence DSL, which is a little
cleaner that the JPA/JDO abstractions on top of the App Engine.
I currently see many HardDeadlineExceeded exceptions with the App
Engine and Grails. It is just not designed to work well with Spring
right now. Hopefully this will improve with the later releases of
Groovy, Grails and the Spring / Google partnership for GAE for
business, but I wouldn't consider Grails on GAE production ready.
Even with Gaelyk, there are reports of slow performance. So imagine
the difficulties that arise with the much bigger Grails stack.
The app-engine comes with it's own implementation of a user / security
management system based on GMail accounts. If you just want to provide
an admin / non-admin implementation, this is supported in the
appengine configuration. Cannot comment on Shiro.
Be aware that one of the major restrictions of the App Engine is the
inability to write a file, so even basic file uploading in Spring
becomes problematic since the default mechanism writes to a temporary
file. I would imagine that most of the plugins would not work out of
the box without digging into their code and changing it.
I think the biggest issue here is lack of support for native JDBC. JPA
is not as well supported as plain JDBC GORM, things like named queries
would probably not work out of the box without retrofitting. If you
want to use the latest and greatest parts of Grails, it might be
worthwhile to consider other hosting solutions.
From Aaron Eischeid
1.The GAE plugin and the JPA-GORM plugins combined do not get you all GORM features seamlessly. Though you should get basics like .save(), .delete(), and maybe .list() the dynamic finders etc. are going to be out (at least for now). I could be way off here, but I think most/all Hibernate dependent features are out or replaced by something else (since it relies on SQL under the hood and GAE doesn't currently have SQL based DB...) so for example any criteria builders are a no go. It is unclear to me how much of the dot drilling you can do on objects. For example, not sure if you could do something like:
def b = new Book()
def stores = b.authors.publishers.bookstores
One place I could use some pointers is how to use JPA in the domain classes. I am sure there is good info out there, but I just haven't found it yet.
unsure
grails plugins that include domain classes or manipulate your current domain classes are bound to have issues since you have to construct your domain classes differently to play nice with JPA which is necessary because Googles Datastore isn't quite like a relational DB. On the flip side. you can use Google's built in security so you shouldn't necessarily need plugins like Acegi or Shiro.
This probably boils down to the different levels of GORM that you can use in controllers and services and the different ways you define domain classes. Some refactoring seems inevitable unless JPA plays just as nice with SQL DB's as it does with Googles Datastore. If JPA can move like that then transferring should be easy, but by using JPA-GORM you give up some stuff you would probably want if you weren't benefiting from due to being on GAE.
Eager to hear what others have to say,
Aaron
I am thinking of making an (initially) small Web Application, which would eventually have a potential to grow. All things considered Google App Engine seems like a very attractive option. Say, user base and complexity grows and for one or other reason I needed to leave GAE behind. How difficult would it be to migrate away?
1) Does GAE provide a way to export the database? What format would it be? Would it be difficult to put it under MySQL (or similar)?
2) In which areas (ex. database access, others?) would I have to use GAE API? I.e. which parts of implementation would have to be abstracted away / interfaced?
Edit: 3) Alternatively, is it even worth to abstract away GAE API?
For question #1: I don't know if GAE specifically supports exports of a database but you can always roll your own, worst case scenario. If you are in a position where you need to, you'll probably have the resources to do it, too.
For question #2: You can and should always encapsulate those kinds of outside dependencies anyway. It doesn't matter whether or not they provide interfaces. Coupling to those interfaces should be kept to an absolute minimum.
For question #3: This question is not really super-clear so I cannot answer it.
I'm speaking strictly from a java webapp point of view...
Google App Engine for python has a backup/restore utility:
http://code.google.com/appengine/articles/gae_backup_and_restore.html
There is a huge interested in porting this to the java flavor.
You can use the higher level standard database apis (JDO/JPS) to allow you to move your app away from google's database services. I suggest purchasing the data nucleus tools in order to smooth the transition from big tables to something like mysql or oracle.
The packaged services GAE provides are enumerated at
http://code.google.com/appengine/docs/java/javadoc/
The stock JRE should handle porting of the urlfetch, mail, and memcache api packages.
You'll have to find a substitute technology for the users, blobstore, xmpp, and taskqueue packages.