Sitecore external database integration - database

What would be a good way, and good practice when "integrating" an external SQL database, in a Sitecore project.
The Sitecore project will get alot of its content from this external database, which is maintained elsewhere, and is constantly updated. (so copying the external database or syncing, is not really prefered, and we dont plan on enriching the data either)
Are there some method of defining some objects and "pipelines" between the Sitecore, and the external database (say without having to use, too many webservices)

What you can do is create a custom Data Provider to connect to your external SQL database. This way you can expose the external data to Sitecore as if it were native data.
See When to Implement Data Providers in the Sitecore ASP.NET CMS for more information.

When we implement architecture like this, we would use a search index (like solr) for maintaining state of the external database and keep enough content to display lists.
I would then request the external source for the display of an "article".
Sitecore would own the web application, url and all page rendering definitions.
If you want to use DMS you might want to extend this further into having some items for referencing elements

Related

exporting data for analytics use in SaaS

We are a SaaS product and we would like to be able have per-user data exports that will be used with various analytical (BI) tools like Tableau or PowerBI. Instead of just managing all those exports manually, we thought of using some cloud database such as AWS Redshift (which will be part of our service). But then, it is not clear how is user will access those databases naturally, unless we do some kind of SSO integration with AWS.
So - what is the best practice for exporting data for analytics use in SaaS products?
In this case you can build your security in to your backend API layer.
First you can set up processes to load your data to Redshift, then make sure that only your backend API server/cluster has access to redshift (e.g. through a vpc with no external ip access to redshift)
Now you have your data, you can validate your user as usual through your backend service, then when a user requests a download through the backend API, the backend can create a query to extract from redshift only the correct data based upon the users security role. In order to make this possible you may need to build some kind of security column into your redshift data model.
I am assuming getting data to redshift is not a problem.
What you are looking for, if I understand correctly is a OEM solutions.
The problem is how does one mimic the security model you have in place for your SaaS offering.
That depends on how complex is your security model.
If it is as simple as just authenticate the user and he has access to all tenant data or the data can be easily filtered for user. Things are simple for you. Trusted authentication will allow you to authenticate that user and user filtering will allow you to show him all that he has access to.
But here is the kicker, if your security is really complex , then it can become really difficult to mimic it within these products.
Here for integrating tableau this link will help:-
https://tableau.github.io/embedding-playbook/#
Power BI, this product am not a fan off. I tried to embed a view in one my applications and data refresh was a big issue.
Its almost like they want you to be a azure shop for real time reporting.( I like GCP more )
If you create the api's and populate datasets then they have crazy restrictions like 1MB/sec etc.
On the other instances datasets can be refreshed only 8 times.
I gave up on them.
Very recently I got a call from Sisense and they seemed promising as well from a OEM perspective. You might was to try them.

Create multiple databases dynamically in Microsoft Azure

I am a newbie in Microsoft Azure platform. I want to create multiple databases dynamically (We are developing multi-tenant model. So, Each organization should have their own database. Whenever an organization is registered with our system, we need to create a new database dynamically). Please provide some insights on this.
By using Azure Resource Manager Templates you can reliably deploy the whole infrastructure required by each organisation. So if they need a webserver, database and middleware servers, you can define the whole thing in a template and reliably deploy that for every client.
(from the above link)
You can deploy, manage, and monitor all of the resources for your solution as a group, rather than handling these resources individually.
You can repeatedly deploy your solution throughout the development lifecycle and have confidence your resources are deployed in a consistent state.
You can use declarative templates to define your deployment.
You can define the dependencies between resources so they are deployed in the correct order.
You can apply access control to all services in your resource group because Role-Based Access Control (RBAC) is natively integrated into the management platform.
You can apply tags to resources to logically organize all of the resources in your subscription.
You can clarify billing for your organization by viewing the rolled-up costs for the entire group or for a group of resources sharing the same tag.
The link above has a lot of resources for learning how to use templates as well as the syntax and usage.
There are a large number of templates at the Azure ARM Template Github page and even some pre-existing templates to get you started deploying SQL Server to Azure (there's also mysql and postgress if you prefer)
Plus many others that you can work through to get accustomed to how they work.
you can use the AZURE SQL Database REST API to do so, its as simple as sending a PUT Request to a URL https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group-name}/providers/microsoft.sql/servers/{server-name}/databases/{database-name}?api-version={api-version}
Check out these links for more details
https://msdn.microsoft.com/en-us/library/azure/mt163571.aspx
https://msdn.microsoft.com/en-us/library/azure/mt163685.aspx

How does writing to a database work when a web app uses multiple databases like Sitecore has

In Sitecore you basically have three databases. The Core, Master and Web database.
Simply put the Core database holds all Sitecore settings. The Master database is the authoring database. So it contains all versions of any content.
Then in Sitecore you can "publish" the contents and it will publish the latest version of each content to the Web database.
So suppose I have a website with a news page. And a user is able to edit a news item from the web site (so not through the CMS). How would the database then get updated when it's set up like this?
It would probably update the Web database, but then when I go into the CMS I don't see the latest changes, since the CMS reads from the Master database, right?
So does that mean that it should write twice? Once to the Web database and once to the Master database?
Can anyone tell me how this works in Sitecore or the like?
The reason I'd like to know this is becasue I'm thinking of creating a similar database setup. And I'm just not sure how to solve this issue.
When you have items that needs to be updated by the website visitor, you need to use the SitecoreService SOAP webservice or create your own custom webservice that runs on the Master-instance and triggers a publish after updating.
Well, Sitecore has a publishing step. When the user publishes in Sitecore, it updates the Web database at that point. If you want to build a similar system, I would simply store all versions of an item in the Master database and only when the user chooses to publish, copy the latest version to the Web database.
If your site
- generates a lot of comments
- generates the comments continuously
- uses multiple content delivery servers
- requires CMS users to manage them
I would not store the comments as content items.
The reason is HTML cache and publishing behavior.
On high volume site you'd most certainly use html caching to achieve best possible performance. If a publish is required to show comments, you'd need frequent publish actions and thus html caches are cleared often.
You don't wan't that :-)
Modeling after the DMS implementation is the safest (not cheapest and Datatables isn't something I recommend these days), storing stuff in a separate database, possibly using queuing to prevent an overload if things get busy..

Grails - switching data sources based on User

We are developing a Grails web application, where different users (customers) need to be pointed at different databases containing only their organization's data. Unfortunately, the separated databases are a requirement, and we are being asked to be able to have only 1 web application for everybody.
However, Grails expects only a single datasource pool connecting to one database.
We want to be able switch database connections, per session, based on the user that is logged in, where the different connections are read in from properties files during the BootStrap init().
So far, we have been unable to find a solution that does not seem to have race conditions, there is no plugin we can find, and it doesn't seem to be a popular issue.
Our most promising was creating a custom dynamic data source, set up in Bootstrap to define a map of organization->dataSource, and utilizing a closure defined in Bootstrap to find the appropriate dataSource before GORM behavior, but this seems to cause race condition when there is latency.
Does anyone have an idea how this switching can legitimately be performed?
Thanks
Considering Grails is built upon Spring your best bet is to develop your own resolvable datasource.
Dynamic datasource routing
Example of datasource routing
It's not clear in your question if you're deploying your application once, and trying to configure the DataSource used by User, or if you just want to configure by deployment.
If it's just per deployment, Grails allow you to externalize the configuration. You can set this to use a file in the classpath or in a static location.

Setup for server side for application which need easy acces to data source

I need to make a couple of mobile applications which will all access a shared online resource using e.g. REST API.
What is the cheapest/easiest setup for the server side resource?
The server should store data as either xml/json/sqlite and expose an API to access this data, preferably in a secure manner.
Is Google App Engine appropriate? Any others?
What would be a recommended way to implement?
What I want to do is to have a database online (not important which format - content will not bee too big, ~5000 records with around 5-10 text fields each), have a simple management console for editing this content and then let mobile devices connect in order to check if they have the latest data and update if required.
The data should not be publicly available but key may be hardcoded into device applications.

Resources