I've built a django/satchmo ecommerce site which is starting to get some traffic, and I am having a problem because I do not have a smart way to deal with database changes. When I develop the site on my local system, I make changes to the layout and the DB, which manages the product attributes.
When I want to push new developments to the server, I have to overwrite the server database which has information about recent shoppers and purchases.
What I want to do is "merge" the two databases together so that new purchases still stay recorded in the database, but which also allow me to push local changes to the server.
I'd appreciate any advice. Thanks.
are you using south? (if not, you should)
in particular, have a look at data migrations
Related
I am trying to figure out a reliable data architecture for a small business that wants to utilize Shopify for their online store.
My current idea is to have a central database (doesn't matter which, we'll say Postgres). The business users can create products, services, customers, etc in Shopify. Customers could go to their site and order things.
Then I would have a background process to sync all these things from Shopify to this central database.
Some orders would have to be put in a LIMS system so I would have another background process to sync these orders in the central database with the LIMS system.
Some orders come directly from the LIMS system so another background process to sync these orders to the central database.
All the orders in the db need to be synced to Quickbooks so another background process to sync this.
Concerning Shopify, does this seem like a doable strategy? I'm an experienced programmer, but no experience with Shopify or its APIs.
Also, I'm not really a fan of syncing data sources but...is there a better way?
Obviously I could write an admin ui to let them create things in the central db first and then sync them to Shopify.
But isn't the whole point of Shopify to make this part easy by providing them a way to create all these things right out of the box?
One of the other things that they want is to run reports against this central database. In theory Shopify could act as the central database, but not sure what reporting would even look with Shopify API.
My advice to you is to use Shopify till you understand it, and once you do, you may find things that Shopify cannot do for your business. Only then would you marshal any resources to try and fix those problems with custom code. As it stands, your question is one of "how would I..." for problems that do not exist for you. Why bother thinking about problems you do not have? Wait till you actually have a problem, then ask a question about how to solve it.
Imagine a large corp with dozens of companies, each with their own website and each website will have their own unique functional requirements
Most data on each website will be specific to that website
Each website can edit its own data
Some data will be shared across all websites
There will be a central CMS that is allowed to edit this data, but other websites can read and use that data
e.g. say you're planning the infrastructure for a company that owns multiple sub-companies that make different kinds of products, some in the same category (cereal, food), others in completely different categories (books, instruments). Some are marketing websites, some are for CRM, some are online stores
there are a list of regulatory requirements that affect all products
each company should manage the status of compliance of its own products to each requirement
when a new requirement surfaces, details regarding that requirement should only be entered once
How would the multiple databases be coordinated?
edit: added more info per Bob's suggestions
Thanks for the incredibly insightful questions!
compliance data is not shared, silo'd within each site
shared data is only on the one enterprise-wide database, they will mostly be "types of [thing]"
no conclusive list of instances where they'll be used but currently it'd be to populate CMS dropdowns for individual sites.
changes to shared data would occur a few times a year.
Ideally changes would be reflected within a few minutes, but an hour or so should be acceptable
very low volume in shared data.
All DBs will be new, decision on which DB is pending current investigation.
Sub-systems will expose REST api
Here are some ways I have seen this handled, you need to think about the implications of each structure based on the details of your particular business domain. All can work, but all have to be carefully set up if they are going to work.
One database for shared information and one for each client for client-specific information. Set up the overall application so that the first thing you put in the application on log in is the client and it connects to the correct client. People might have to also have a way to change the client if users will handled multiples.
Separate servers for each client if they completely need to be siloed. Database changes are by script (and in source control) and are applied to each server as need be. So the changes to the central database might have a job that runs to push any data changes to the other servers
All the data in one database, but making sure each table has a client_id so that the data is always filtered correctly by client. You can set up separate views by client, so that the users can only see the clients they are supposed to see. This only works if the data for each client is substantially in the same form.
And since you are in a regulatory environment, I strongly urge that you create an audit database that is updated by database triggers (never audit from the application, you will lose changes to the data) for each database.
I agree with Chris that, even after both the sets of questions, there is still a big set of possible solutions. For instance, if the databases were the same technology, and the shared data were stored in the same way in each one, you could do db-level replication from the central db to the others. Is it OK to have 2 separate dbs per application (one with shared stuff and one with not-shared?) - this would influence the kind of replication.
Or you could have a purely code solution, where clicking publish in a GUI that updates the central db calls a set of APIs that also update the other dbs. Or micro-services - updating the central db also creates a message on a shared queue, that is picked up by services that each look after a different db and apply the updates in whatever form makes sense for that db.
It depends on (among the things already mentioned) what your organisation's technology strategy is, what technology and skills you already have in-house, and so on.
So this is as much an architecture question as it is a db question.
I don't think this question is sufficiently clear to get a single answer. However there are a few possibilities.
In many cases, where you have shared data you want to have a single point of ownership of that information. It could be in a database, in an excel file (which can then be turned into csv and periodically loaded on all dbs), or some other form. The specifics depend on what is shared exactly.
Now in this case it sounds like you are going to have some sort of legal department in charge of some shared information and they will manage that data, which will then be shared to the other sites. This might be done with an application they manage which aggregates information from the other companies or it could be data which is pushed to their systems.
A final point:
Software is at its best when it facilitates human solutions to human problems, not when it tries to solve those problems directly. In these cases, you probably want a good human solution in place and then to look at what software can do to support that. A lot of the issues (who owns the information?) will already have been solved and you will be simply automating what is already done.
Are there any resources available that can guide someone on how to 'think' about the various components of a hosted / cloud solution before going ahead and starting to make a hosted application? If that made no sense, what I mean to ask is are there any guidance books/websites on what things need to be considered when making a cloud application?
I am attempting to make a hosted CRM-style software application that will serve many hundreds of customers. The application is powered by a SQL server database with many tables and a ColdFusion, HTML5, CSS, Javascript front-end. If I was installing this application and its components at each client site, then each installation is unique to that customer. But somehow I have to replicate this uniqueness in the cloud which is baffling me.
Only two things have come to mind so far:
The need for a unique database per customer in SQL server
The need to change DB connection strings per customer in the web application
My thought process has come to a block when I am trying to envisage how to design the application to serve so many different customers. Even though the application that all customers use will is the same (same DB tables, same front-end), the data that they store and retrieve will be specific to them. So I was thinking that surely each customer needs a separate database creating for them? Is it feasible to create a replica database for each customer? If I need to update some tables or add a new table, how would I do this for hundreds of different databases?
From the front-end I guess each unique customer log-in would change DB connection strings so that they can only access their database. Other than this I can't think of anything else that needs to change per customer basis.
When a new customer wants to sign up, it needs to be clear to me what I need to create for them to have access to the application. I guess this is ultimately what I need to think of but I'm stuck.
If anyone can suggest some things to think of or if there is a book or website on this kind of thing that someone could point me to I'd really be very thankful.
EDIT:
I was looking at an article about Salesforce.com and it says
"In order to ensure privacy of data for each user and give an effect of each having their own database, the data from different users are securely isolated from one another."
Anyone know how this is achieved or how it may be done?
Found some great information here. It is called multi-tenant database design and seems to be a common topic. Once I get the database designed then the application can sit nicely on top.
https://dba.stackexchange.com/questions/1043/what-problems-will-i-get-creating-a-database-per-customer
I am looking for a decently efficient way to listen for List changes on a SharePoint site using only the Client Object Model. I understand how backwards this idea is, but I am trying to keep from having to push any libraries to the SharePoint servers on install. Everything is supposed to be drop and go on a local machine.
I've thought about a class that just loops a timer and keeps querying the ClientContext from the last date of successful query on, but that seems horribly inefficient.
I know this is a client object model, but is there any way to get notifications from the server on changes from the client only?
I am afraid that this is not possible by using the client object model. If you need to poll too often that the user experience suffers from the slow performance too much, you would need to catch the list changes on the server side. deploy a solution with a feature registering an SPItemEventReceiver to your list.
I understand your reluctance to push server-side code to the SP farm; without it, you can save discussions and explanations to the customer's administrators. However, some tasks are more efficient or even feasible only when run on the server. You can consider Sandbox Solutions for such functionality. They are deployed to SP not by the farm administrator but to a site collection by an site collection administrator by a friendly web UI. This needs less privileges, more relaxed company policies to comply with, and can be better accepted by your customers. You can develop, test and even use your solution in your site collection only without affecting the entire farm. Microsoft recommends even farm-wide solutions to be designed with as much as possible functionality in sandboxed solutions, putting only the necessary minimum to a farm solution.
If deploying the entire application as sandbox solution would not be possible, you could combine a sandboxed solution gathering the changes with an external web site requesting the gathered data from the site collection, or in you case with a client-only application as you are speaking about. (Sandboxed solutions have one big limitation: You cannot make a web request from within the site collection outside; you can only access the site collection from outside.)
--- Ferda
I have a few questions about the architecture of a subscription service I am about to embark on and I am looking for some feedback on how best to set it up.
I won’t have a large amount of customers as Basecamp, maybe a few hundred and was wondering what would be a solid architecture for setting up the customer sites. I’m running SQL Server and .NET on a dedicated machine. Should create a new database for each customer as to have control and isolation of data or keep them all in one database?
I am also thinking of creating a sub-domain for each customer as well so modifications can be made to each site as needed. The customer URLs would look like this:
https://customer1.foobar.com
https://customer2.foobar.com
I am going to have the ability to ‘plug-in’ reports that will be uploaded to the site so each customer can customize as needed. Off the top of my head this necessitates having each sub domain on its own code-base for the uploading of these reports.
So on the main site the customer would sign up for their new subscription and I would programmatically create a new directory for the customer from the main code base and then create a sub domain pointing to the new directory for the customer and then finally their database.
Does this sound about right? Am I on the right track? How do other such sites accomplish the same thing?
Thanks for letting me bend your ear for a bit on this.
From a maintenance perspective, having a virtual directory for each customer scares me. Having done something similar, I would create separate domain pointers as you are intimating. Then you can check the referral headers to see what should be displayed. I would probably create one main site template and dynamically brand it for each customer. You can still create separate folders for customer specific reports or if you really need custom pages unique to that customer. I just wouldn't make each their own site.
The advantage of separate sites (including databases) is that the fate on one client isn't bound to all others. It'd be easier to upgrade (trial) to a sub-set before deploying to everyone else. The big issue here, as Scot points out, is time. You'd want to have things as automated as possible (and well tested), etc. It's also easy when a client leaves. You can always just back-up their database and send it to them (for example).
Auto-provisioning new sites and databases isn't easy, and the account that does that will need plenty of privileges - so your security testing will need to be better than usual.
A multi-tenancy approach is good for minimizing your time but you do have to be careful, you don't want customers data getting mixed up.
One approach that will work, within the one app (and database), is to make use of HttpHandlers (MVC framework, perhaps) so that some sort of client identifier is in part of the URL - but the folder doesn't have to physically exist (or virtually in the IIS sense). That way you don't have to worry about getting folder permission correct; but you do have to be careful about correctly identifying clients, their ids, and making sure clients can't make calls that use an id that isn't theirs.
https://www.foobar.com/[clientid]/subscriptions
The advantage of this is it's relatively straight forward: everything is in the application, and you don't have to worry about adding new DNS records, setting directory and/or database permissions, etc.