I would like to save hyperspectral images using Python, but I don't know where I can persist the data. I have thought about HDFS. I need to do it on my local server without using cloud providers
Is there a way to make it easy, and do you recommend any particular database?
HDFS generally requires you to build and administer your own Hadoop cluster.
I'd consider cloud object storage such as AWS S3 or Google Cloud Storage for the following reasons:
Relatively cheap
Fully managed
No restriction on file size or number of files
Easy Python APIs
Durable - data can be replicated across multiple regions (all handled automatically for you), so you don't need to worry about losing anything if a server dies.
I'd like to understand more about what DB products/combos-of/platforms are generally used these days to build applications which have both mobile and web clients connected to a centralised DB. Lets just say, for example, I wanted to build a simple "To-do" List app which multiple users can download/use. With the following requirements:
Preferably built using free/open source/community products (understand hosting costs involved) but also happy to hear cloud-based options too (other than Firebase)
Must support both a mobile (ios/android) & web client (web app)
Central server-side DB (to support above) with sync capabilities to update clients (mobile/web), when an update takes place clients sync/update
Offline "read" capability (no offline write necessary)
NoSQL/Doc DB is fine (RDBMS would work too though)
This exercise is to help me (and others interested) understand the options/products/platforms available when attempting to create an application with such requirements. Its intrigued me for a while what options are available for a non-enterprise, non-commercial application such as a to-do list where a users data is sync'd across devices yet available to a web client too, via an architecture that doesn't come with enterprise costs.
I've looked at various offerings from the main vendors but when it comes to synchronising data between devices and a central server DB, generally paid-for products are required. I see free apps daily which must use highly cost-effective (bordering free) components (above basic hosting) in their architectures and just wonder what products they generally use to achieve apps capable of synchronising data between clients from a centralised DB/server.
I have an app which needs a backend which consists of 3 parts:
Database
Front PHP pages to handle request from the app, access DB and return JSON
File-system storage for pictures
Now I know that many people go with Amazon AWS today, and I recently had a quick look around at their services. Now for the database, it seems DynamoDB would suit me fine! But my question is, which product (and how) should I use to store the static php pages, and the basic filesystem (which can potentially get pretty big). For the static php pages, I really need something as simple as services the likes of GoDaddy.com, but it has to be fast and be able to respond to many requests.
For the image file-system storing, this could either be done on the same place as the php files, or anywhere else as long and I can access it with php. What do you recommend?
I would really like to hear which products you think will suit this back-end, as this is a pretty popular setting. If you think of something better than amazon, I am open to suggestions, just keep in mind that the top priorities (by order) are stability, scalability and ease of use, and it need to be VERY scaleable.
Thanks for any replays!
You could use S3 for storage as it is simple, easy to use and scalable. But it's slow to retrieve files from S3. To solve this, retrieve the files from Cloudfront instead of S3. S3 bucket can act as the origin server of your Cloudfront distribution. It has two advantages -
- Retrieval will be very fast, specially of more popular pages/pictures
- It doesn't matter which part of globe your app is being used, Cloudfront will select the nearest edge cache to serve your content.
I am asking this in very general sense. Both from cloud provider and cloud consumer's perspective. Also the question is not for any specific kind of application (in fact the intention is to know which type of applications/domains can fit into which of the cloud slab -SaaS PaaS IaaS).
My understanding so far is:
IaaS: Raw Hardware (Processors, Networks, Storage).
PaaS: OS, System Softwares, Development Framework, Virtual Machines.
SaaS: Software Applications.
It would be great if Stackoverflower's can share their understanding and experiences of cloud computing concept.
EDIT: Ok, I will put it in more specific way -
Amazon EC2: You don't have control over hardware layer. But you can take your choice of OS image, Dev Framework (.NET, J2EE, LAMP) and Application and put it on EC2 hardware. Can you deploy an applications built with Google App Engine or Azure on EC2?
Google App Engine: You don't have control over hardware and OS and you get a specific Dev Framework to build your application. Can you take any existing Java or Python application and port it to GAE? Or vice versa, can applications that were built on GAE be taken out of GAE and ported to any Application Server like Websphere or Weblogic?
Azure: You don't have control over hardware and OS and you get a specific Dev Framework to build your application. Can you take any existing .NET application and port it to Azure? Or vice versa, can applications that were built on Azure be taken out of Azure and ported to any Application Server like Biztalk?
Good question! As you point out, the different offerings fit into different categories:
EC2 is Infrastructure as a Service; you get VM instances, and do with them as you wish. Rackspace Cloud Servers are more or less the same.
Azure, App Engine, and Salesforce are all Platform as a Service; they offer different levels of integration, though: Azure pretty much lets you run arbitrary background services, while App Engine is oriented around short lived request handler tasks (though it also supports a task queue and scheduled tasks). I'm not terribly familiar with Salesforce's offering, but my understanding is that it's similar to App Engine in some respects, though more specialized for its particular niche.
Cloud offerings that fall under Software as a Service are everything from infrastructure pieces like Amazon's Simple Storage Service and SimpleDB through to complete applications like Fog Creek's hosted FogBugz and, of course, StackExchange.
A good general rule is that the higher level the offering, the less work you'll have to do, but the more specific it is. If you want a bug tracker, using FogBugz is obviously going to be the least work; building one on top of App Engine or Azure is more work, but provides for more versatility, while building one on top of raw VMs like EC2 is even more work (quite a lot more, in fact), but provides for even more versatility. My general advice is to pick the highest level platform that still meets your requirements, and build from there.
This is an excellent question. Full disclosure as I am partial to Azure but have experience with the others.
Where I think Azure stands out from the others is the quick transition from on prem to the cloud. For example -
SQL Azure - change connection string, upload DB, go!
Queues work a lot like MSMQ.
Blobs are pretty much blobs any way you shake them but they scale like crazy.
The table storage component is good because it provides incredible scalability for name/value pairs - but takes some getting used to.
Service Bus is my favorite of the services because it allows for a variety of communications paradigms. Two SB endpoints first try to connect to each other, if they cannot, then they route through the cloud - makes for very secure and scalable processing when firewalls tend to get in the way.
Access control list - paired typically with the service bus to make sure the right people access the right things - think SAML in the cloud.
I hope that helps!
My cloud experience is currently limited to Salesforce.com
For standard business operations and automation it provides a significant number of features that allow us to get apps up and running very quickly. We are particularly benefitting from the following:
Security (Administrators can control access to objects and fields)
Workflow & Approvals
Automatic UI generation
Built in reporting and dashboards
Entire system (including our custom changes) is accessible via web services
Ability to make the data in the system available through public sites (e.g. eCommerce)
Large library of third party apps to solve standard problems
The platform does NOT solve every problem.
I would not use the platform to model a nuclear power station or build the next twitter.
The major points of cloud computing is to save on costs by paying for usage and enable immediate deployment of computing resources.
The costs are not purely x amount of cents per instance per hour. The costs include maintenance, development, administration, etc. The huge benefit of cloud, in my mind is to liberate the customers from having to manage anything that is not within the realm of their core business competency. If I am an insurance business, I want my developers to concentrate on my insurance problems that help solve needs of my claims, rates, etc. I would rather avoid dealing with problems of email servers, file servers, document repositories, and administrating OS patches, service packs, etc.
Thus, in my opinion, the biggest benefits are derived from the SaaS and PaaS cloud offerings. One should go to IaaS only when PaaS or SaaS have serious restrictions to specific needs (i.e. I need to install a set of proprietary COM components and Azure does not support them).
SaaS is good for commodity type of applications that are not the core line of business for the client, but are more of a utility. These are your typical Messaging systems, Portals, Document Repositories, Email systems, CRMs, ERP's, Accounting, etc. etc. etc. Why reinvent the wheel by writing your own when you can customize a well supported third party product.
PaaS is great for core line of business software that supports companies' main business offering. Abstracts clients from having to deal with OS management and lets clients concentrate on the business system development - something that noone else can do for the client.
One can also take advantage of the benefits of PaaS (let's say, Google App Engine) and extend it, at times and if necessary, by pulling out some virtual machines from IaaS providers (e.g. Amazon) to do some number crunching then just send back the output to Google App Engine.
This way, you get the best of both worlds -- you can rapidly develop scalable apps in GAE, then you can always augment it by running any program you want from Amazon virtual machines.
This keeps changing, now Windows Azure also supports VM, so it is also an IaaS provider now.
Now how about Free Amazon EC2 for a year to do a better comparision. Check this out.
http://www.buzzingup.com/2010/10/amazon-announces-free-cloud-services-for-new-developers/
I'm working on a Silverlight app that would allow a user to upload a few gigs of files to a hypothetical cloud based file store, then allow the user to view some data about those files later (more functionality than a file store). Ideally I'd like to use a free, per-user store such as SkyDrive but I can't seem to find an API for that service (and read elsewhere on stack overflow that programmatic access violates their TOS). Do any services fit this bill? I've heard of Amazon S3 but I understand that'll cost some money - is anything free?
EDIT: Could Mesh be an option?
What is LiveMesh Object and its connection with Silverlight 3.0
You could look at using Azure as it offers a blob and table storage cloud infrastrucutre and will happily run silverlight applications in an azure web role. Currently there is no cost but this will change once it RTW's.
More info at http://www.azure.com/
AFAIK, nothing in this world is free when you're dealing with gigabytes of storage, plus the bandwith to put them in the cloud.
Amazon S3 is quite reasonable on its pricing.