In Short
I am thoroughly confused by the array of SQL Server services available, and am having a hard time finding a brief primer.
The Scoop
I am a long time ASP.Net developer who has happily been churning out ASP.Net applications for years. These have usually been based on SQL server and a range of custom tables and views.
Just recently I have had the need to do some reporting that goes a little beyond the straight forward, and so I have started to look into SQL Server Reporting Services (SSRS). In doing so I feel like i have opened up Pandora's Box.
There is reporting services, integration services, Analysis services, Business intelligence, etc etc. This has got me wondering what they all do, and if I could (or should have been) leveraging these tools to do some of the heavy lifting for .net applications.
My immediate research efforts on Microsoft have stifled me with impenetrable marketing and business speak. My Google searched have turned up no comprehensive overview of what these tools are, or they fit together, or even which tool should be used when. One of the biggest problems i am having comprehending the options is that they all seem kind of the same to me!
So, all of this to say, if some kind sole could set me straight and point me in the right direction, I would be very grateful :)
Very high level overview:
Core services for the database engine itself consist of the SQL Server service itself (or MSSQLSERVER) and there's also the SQL Server Agent and the SQL Server Browser. The SQL Server Agent is a job scheduler for SQL Server and handles some other maintenance tasks and so on, while the Browser service helps with accepting logins and so on by creating an easier way for client applications to connect to SQL Server.
All three of these services though are true 'services' in the sense that they're daemons running on your server. But of these three, you ONLY need MSSQLSERVER running to be able to work with the relational database engine. (The other two services are optional.)
Then, in addition to core database engine functionality, there are a bunch of add-ons or supplemental 'products' and tools that Microsoft Offers to help make SQL Server that much better and more compelling as a platform/offering. As you've seen, these are typically marketed as 'services'.
Reporting Services (or SSRS) is a great solution that uses a native SQL Server database to store information about data sources, report delivery schedules, access permissions, and so on... for a powerful reporting engine that makes it very easy to output great reports from pretty much ANY data source. You just define sources for the reports, wire up datasets and parameters, then use a specialized xml markup (Report Definition Language or RDL) to define look, feel, shape, and so on of your reports. And, Microsoft has created a 'tweaked' version of Visual Studio to make creating/developing/editing these reports much easier. That version of visual studio is called SQL Server Business Intelligence Design Studio (or BIDS).
Likewise, Microsoft also uses BIDS to help developers, DBAs, and analysts create data cubes - multidimensionally 'joined' tables using a specialized schema that allows SQL Server Analysis Services (SSAS) to churn through pretty much any combination of data relations in order to provide 'analytical data'. So... say you have a company selling widgets: you could define relationships between products, customers, sales regions, dates, and so on - and SSAS will take that data (along with specially defined 'cube' definitions that define those relationships, their granularity, and so on) and pre-calculate the outcome to pretty much any combination of those 'dimensions' or angles of data - so that you could say something like: "Show me all x-brand widgets sold by sales people in the northwest region in January of this year.." and it would give you a sum of those parts sold... and even let you drill-down by sales person, store, and so on - assuming you've defined your cube/relationships as needed.
Of course, this functionality is pretty powerful, but you almost never want these cubes running against your production tables... so companies typically create data warehouses or data marts to store their analytic data in. But to copy data on a regular basis from production into these warehouses/marts.. you need some kind of tool that makes it easy to figure out which changes should be picked up, how they should be handled, and what kinds of tweaks/changes you need to make to them to get them out of a highly-normalized production environment into a highly DEnormalized warehouse/data mart (where you're using snowflake or star schemas)...
And this tool/service is none other than SQL Server Integration Services (SSIS) - which has a rich set of connectors, pumps, transforms, monitors, and so on to facilitate data transformations and migrations on a scheduled basis. What's very cool about this tool/service though... is that it doesn't just connect/transform data between SQL Server and SQL Server... but it's able to connect MOST data sources to MOST data sources - so it's a powerful tool for transforming, exporting, importing, modifying, tweaking, and so on all of your data.
And... to define the 'workflows' for your integration/transformation projects, you also use Business Intelligence Design Studio - same as with defining cubes and so on for SQL Server Analysis Services.
Ultimately though, what makes all of these so cool is that you get access to:
SQL Server
SQL Server Reporting Services
SQL Server Integration Services
SQL Server Analysis Services
With SQL Server Standard and Enterprise Editions - so there's no need to buy extra tools/solutions/offerings in order to be able to take advantage of these great features. (The Enterprise Edition of SQL Server which costs about 5x as much as Standard... comes with much better/cooler features in some ways ... but you still get a solid subset of all features using the Standard Edition.)
Here is a basic overview of the services:
Analysis Services: This is Microsoft's data warehousing solution. It allows you to create data cubes and other warehousing features. Provides tools to make it easier to create what-if scenarios. This runs on top of SQL Server. Data warehousing is also referred to as Business Intelligence applications because the warehouses are used to mine and analyze data.
Reporting Services: Allows you to build database reports more easily. It provides a report designer and management functionality. This service is supposed to make it easier and faster to create rich reports. The reporting service also works against non-SQL Server data services.
Integration Services: Provides functionality to build Extract, Transform, Load (ETL) functionality. ETL tools integrate individual data sets and help with extracting, aggregating, cleaning and uploading data. Like the other services, it provides a GUI that is supposed to make these packages easier to create.
You aren't required to use the three services in one bunch but they do provide functionality when used together. For example, you may use Analysis Services to create data cubes, Integration Services to extract the data from an external service and import it into your cube, and the Reporting Services to create reports that contain business metrics from the data contained in the cube.
As to whether you should be/have been using them, as always, it depends. The services provide some great services but don't always fit the project. For example, Reporting Services do make it easier to create reports but, in my experience, it isn't friendly enough for the end-user report authoring as it is advertised. Integration services are great to do some ETL jobs but isn't as flexible as was sometimes required. I used this service in a few projects but it always ended up being more trouble than I would have expected. But definitely, take a look at them.
Related
Im looking through all the options in Visual Studio for database integratiion with Winforms apps. I am just overwhelmed by the options. It's not that I can't read about each of them on MSDN which I have done, it's that most of them seem adequate. Basically I have a fairly large collection of financial data which I need to stuff into a db an run queries on it from a Windows Form. Can someone briefly advice which of Service Based Databas, ADO.Net and all it's varients, Local Database, SQL Server Compact 4.0 Local Databace, Service Based Database etc would be suited to the task? Or just generally if any of them are NOT suited at all?
I would use SQL Server Express, which is a free version of SQL Server.
Why? it has very good tools and will grow and scale-up very easily, whether you wish to add more (large volumes) of data, or support multiple front-ends - ASP.NET, mobile, Winforms, MVC, MS Access.
You may wish to connect to your SQL Server back-end with Access to run ad-hoc queries and generate quick reports off of them and at the same time keep expose your data through services via .NET. SQL Server will support all these front-ends at the same time.
All of these are variants of SQL Server. Just create the database in whatever manner you like, and use Entity Framework (or at worse, classic ADO.NET) to interact with it.
Lacking other specific requirements and assuming your data is less than the 10GB limit for the free SQL Server 2010 Express. If you find that you need to store more than 10GB, or evolve to use other features of SQL Server, the upgrade path is very straightforward.
Entity Framework is generally a very productive way to access the database from WinForms. If you're not familiar with it, I would recommend setting aside some time to learn to use it if you can. You will quickly make up the time invested in productivity gains.
I have been looking into PowerPivot and concluded that for "self service BI" and ahoc buidling of cubes it has its uses. In particular I like the enhanced UI that you get from using PowerPivot rather than just using a PivotTable hooked up to an analysis services datasource.
However it seems that hooking up PowerPivot to an existing analysis services cube is not a solution for "organisational BI". It is not always desireable to suck millions of rows into excel at once and the interface between PowerPivot and analysis services is very poor in my book.
Hence the question is can an existing analysis services solution get the enhanced ui features that power pivot brings, without using powerpivot as the design tool? If powerpivot is aimed at self service/personal BI then it seems bizare that the UI for this is better than for bigger/more costly analysis services solutions.
Although I agree that PowerPivot has a nicer UI than using Analysis Services via standard pivot tables, PowerPivot through the Excel client has some really bad drawbacks when trying to use it in lieu of Analysis Services.
You have to download all the rows into your spreadsheet to "refresh" the data. In large data warehouses, this is equivalent to having users run SELECT * queries directly against your database. It's horribly slow for the user and has a high resource usage cost to your server.
It is extremely easy for someone to either intentionally or unintentionally walk out of the office with your entire data warehouse in a non-secure manner. Ouch!
The end-user machines need to be pretty powerful. I tried using PowerPivot with a few small tables (5 million rows or less) on our standard company machine build and it did not have sufficient memory to refresh PowerPivot. The only way I can see to deploy PowerPivot across the enterprise is to upgrade all of the analyst machines to 64-bit Windows 7 with at least 6GB to 8GB of RAM. Although this can be feasible in a small organization, it is not a reasonable solution in a large enterprise.
You won't have any good metrics on how people are using your data if you hand out PowerPivot with unrestricted access to your data warehouse. Yes, you may have metrics on how frequently people hit the refresh button and you may be able to log which tables they are querying, but you won't see how they use the data unless you audit their spreadsheets directly. And even then, you will only get their final result -- not their path to how they got to the final result.
PowerPivot generates really, really big files. Even if someone drills the data down to a small subset of the total data, it is still difficult to share the files with others since large PowerPivot files generally exceed minimum Exchange server file size limits. I've encountered this at my organization despite never having had this problem with an Analysis Services files.
PowerPivot does not have a very good security model. Sure, you can restrict who gets to the data the first time, but you can't restrict it once it is in the spreadsheet. Analysis Services prevents users from making changes to the spreadsheet if they don't have access to the underlying cube. It's just so easy to compromise the security of your most valuable business data with PowerPivot.
PowerPivot does not currently scale for very large data sources. I have several multi-billion fact tables that just can't be downloaded by PowerPivot unless I pre-aggregate them down to a few hundred million rows. PowerPivot works really well for small data warehouses, but it doesn't elegantly scale to large data warehouses.
Please note my above comments don't apply to PowerPivot via SharePoint. I haven't tried the SharePoint integrated product out, but many of the above concerns seem to have been addressed from the documentation and demonstrations that I've seen of the SharePoint version of the product.
Despite all of the above comments, PowerPivot could work as a replacement for Analysis Services if you have a very small or immature data warehouse. If your largest fact table is a few million rows, then the overhead of building and maintaining a data warehouse may not be cost effective if you are a BI team of 1-2 people. PowerPivot is probably a great new feature for a department that doesn't have a dedicated BI team and only has a handful of Excel junky analysts. It doesn't take much sophistication to put together a virtual data mart from disparate data sources with PowerPivot. But if you want to build a truly professional data warehouse that is secure, scalable, and highly manageable, then I would recommend building cubes in Analysis Services and either use Excel or a 3rd party vendors tools for connecting to the Analysis Services OLAP cubes.
Now that SQL Server 2012 is released, you may want to take a look at using one or more SSAS BISM models, rather than PowerPivot. You get interop with PowerPivot, but you can now build your model using SSDT (in Visual Studio) and can get more control over security and can host on a dedicated server.
I'll be presenting live and online this spring and summer of the BISM - here's my latest deck on slideshare - http://www.slideshare.net/lynnlangit/sql-2012-bism
Now that Office 2013 preview is out, you can check out PowerView inside of Excel (PowerPivot) without the need to have SharePoint. It remains to be seen when MSFT will remove the dependency on Silverlight (i.e. move to HTML5). The preview release of Office 2013 that I got in September still included PowerPivot which required Silverlight. I am looking forward to the release built on HTML5. Here's a deck by Jen Underwood to give you an idea of what PowerView looks like.
WebPivotTable is a pure javascript pivot table and pivot chart component which can be used to pivot csv data and all kinds of OLAP cubes, include microsoft SSAS. It mimics all functionalities of Excel but web based, no dependence on any other plugins, drivers, server side compoenents. It can be easily to integrated into any web application and web sites.
Here is Demo and Documents
I know that Powerpivot is a free download for Excel 2010, but for a better desktop client experience you should look at the ProClarity client.
Also worth looking at Analyzer by Strategy Companion (http://www.strategycompanion.com).
I've found it provides a smooth web-based interface for slicing and dicing in pivot tables (and charts) that is nicer than what is provided by Excel 2007.
ProClarity was the runaway best option until Microsoft bought them and killed the product. Some of the features are making their way into other tools, but the product itself is no longer supported. Panorama or Tableau are probably the best 3rd party options.
This is the best I've found so far that is up-to-date: http://www.varigence.com/products/vivid/videos
Edit: http://silverlight.galantis.com is also a possible solution - WPF version comes out next month that could be used is a VSTO add-in.
Our centralized IT department has suggested two primary ad hoc query tools for our general user base of approximately 200 staff members:
Microsoft SQL Server Management Studio 2008 (SSMS)
Microsoft Access 2003
Environment
The backend database is a read-only Microsoft SQL Server 2005 database.
The schema is 400+ tables; allowing access to the raw data for our general staff would be a disaster.
We will be building an "abstraction layer" over the raw data for our general staff to run ad hoc queries against.
The abstraction layer will most likely contain a number of views.
A number of users have basic knowledge in Microsoft Access; none have used SSMS.
Which of the above tools (or alternative) would be best for a decidedly non-techie user base of approximately 200 people? What are the pros and cons of each?
Also, the IT department has suggested teaching people T-SQL so they may use SSMS. Is this reasonable?
How about this one? i-net Clear Reports (used to be called i-net Crystal-Clear) has a powerful ad-hoc reporting component that is made to be an easy-to-use thing for non-technical users. Your users won't have to know anything about reporting at all. They simply select the kind of report, the data et voila there is a report suiting the needs.
The data abstraction can be done easily by creating so called data-views which can be designed by e.g. your administration. There are various ways to access the ad hoc reporting GUI. We have a web GUI, a Java Applet or a standalone Java program.
The end users will not need any training since the GUI is highly intuitive.
The views can easily be build by drag and drop in addition to setting datatypes, formats and so on.
All reports (depending on security settings) can be accessed via DAV our a report repository gui.
The server supports different security settings on a per user or per group basis.
The standalone report designer is free and fully functional.
Disclosure: Yep. I work for the company who built this.
Your "abstraction layer" is the right approach to take with Access. Create an MDB with the basic views required linked into it and distribute to the users. Allow them to create new queries and reports in their own MDB as required.
Now how you are going to stop them from running a Cartesian join on tables with a million records or more I'm not quite sure.
Microsoft have a free tool for business and end users which called "Report Builder". It supports the full capabilities of SQL Server Reporting Services. The good thing it is provides a Microsoft Office look-like user interface.
You can download latest version "Report Builder 3.0" from here
http://www.microsoft.com/download/en/details.aspx?DisplayLang=en&id=6116
And for more information about MS Report Builder check this link
http://technet.microsoft.com/en-us/library/dd207008.aspx
Attempting to teach "non-techie" people T-SQL to query a schema with 400+ tables probably isn't going to do well, unless they are limited to querying the views only, and the views hide all the ugly complexities of various joins, grouping etc.
Our company was in a similar situation where Access was used early on, and then we switched everyone over to use T-SQL and SSMS. IMO, this is the approach you'd want to take.
Again though, the success of this will depend on the quality of your views, or better yet, reports you provide your end-users.
Randy
I would look more into something like Stonefieldquery.com that is designed for non developers to build reports. Not that the report writer or query builder in Access is bad, but may be too much. I think they also provide a way to centralize reports and queries where they can be shared. Multiple people are not going to be able to open a single access file and create a report (I think query building is OK.).
Most will use the drag and drop capability, but about 5-10%will come thing a need for SQL and then you can take advantage of the "teachable moment" and get them some training.
Cons for Access certainly would be cost; SSMS should be free assuming you're properly licensed for the SQL server.
Depending on the actual needs, some users might actually be better off with Crystal Reports (never thought I'd say that), or Reporting Services.
you could create a series of sql server analysis cubes and have the users conenct to those using excel so that they can use excel's pivot tables.
Being a newbie at ad hoc reporting and doing the work myself, I used Izenda.com ad hoc reporting. It was very straight forward, and I could do it myself versus outsourcing.
Check SQLS*Plus - http://www.sqlsplus.com
I found SQLS*Plus to be a very effective command line SQL server reporting tool - this is a free tool (for personal use) and allows me to generate reports with the titles, headers, in HTML and CSV formats, format columns in custom masks, set report length, pagesize, etc. As I understand it is very similar to very well known Oracle SQL*Plus reporting tool
We currently have our custom application that is being distributed with our database (SQL 2005/2008). It is an easy task, before we release a new version we just pack our database into SQL initialization scripts (these create tables and populate data). We use SQL Management studio to generate these scripts.
As a next step we would like to deploy OLAP cube (along with ETL commands made with Integration Services) that would be used to analyze the data in the original database. .
We know to create and design a cube, but I do not even know how could be generalize all these packages and deploy them as a solution, script or something that our customers could install on their servers. Customers do not have a Visual studio and we need to create "something" in a wizard (with some input required from customer e.g. OLAP cube name, server etc) for them to deploy it.
How do you do that?
From Here:
Microsoft SQL Server 2005 Analysis
Services (SSAS) provides three tools
for deploying an Analysis Services
database onto an Analysis Services
server in the production environment:
Using an XML Script Use SQL Server Management Studio to generate an XML
script of the metadata of an existing
Analysis Services database, and then
run that script on another server to
recreate the initial database.
Using the Analysis Services Deployment Wizard Use the Analysis
Services Deployment Wizard to use the
XMLA output files generated by an
Analysis Services project to deploy
the project’s metadata to a
destination server.
Synchronizing Analysis Services Databases Use the Synchronize
Database Wizard to synchronize the
metadata and data between any two
Analysis Services databases.
In addition to using one of the
deployment tools, you can deploy
Analysis Services by using the backup
and restore functionality. For more
information, see Backing Up and
Restoring an Analysis Services
Database.
The Analysis Services Deployment Wizard can be found in your start menu under SQL 2005, Analysis Services, Deployment Wizard. This takes the asdatabase file in your bin directory and creates an XMLA script that creates the SSAS database.
Links:
Using the Analysis Services Deployment Wizard
Readme for Ascmd Command-line Utility Sample
Or alternatively, you can use a tool to build the Cubes and Schemas that provide a simple mechanism for deploying initial implementations and a smooth upgrade path.
As you know deployment, isn't just a case of implementing a database even an OLAP database in the target environment. There's also the ETL, and tables to consider, which also involves ensuring that at every step of the way you're creating table/SQL scripts, and all this is fine and dandy until you come to provide an upgrade to your product, and need to upgrade the SSIS/DW Relational Schema Tables and SSAS Cube structures.
What you find is MS is no help at all here. It's helpful for initial deployments, but doesn't provide much in the way of in situ upgrades.
This is a problem that we have faced up to and developed a tool to address, so that we're able to do the things that you are trying to do, but do them smoothly. Leaving our technicians to focus on building high quality Data Warehouses, rather than technologies to do mundane, annoying, fraught with danger but necessary things like "upgrades".
Check out http://www.dataacademy.com, this is the product we've developed to do successfully, just what you are trying to do. Drop me a mail, if you'd like to discuss further.
Cheers and the best of luck.
I have to study about data mining using SQL Server. As I know, Business Intelligence in SQL Server supports data mining, but I'm not pretty sure.
Does BI really support data mining?
How can I start with data mining with SQL Server? I mean, resources such as books, blogs,..etc
Thank you all.
I would suggest beginning your research by focusing on SQL Server Analysis Services via the Books Online Documentation, in particular the "Analysis Services Information Worker InfoCenter" as the information presented is mostly for the attention of data analysts.
http://msdn.microsoft.com/en-us/library/ms174577(SQL.90).aspx
Using the reference you can choose further readying into specific subjects such as "Data Mining Concepts", which will subsequently lead you onto the various Data Mining Algorithms that are available to you.
Then to get hands on with the technology, take a look at the Microsoft Data Mining Tutorial:
http://msdn.microsoft.com/en-us/library/ms167167.aspx
Every database supports data mining. Handling large amounts of structured data is what databases do!
First learn SQL which is useful in many applications and databases.
Then, if you find stuff you can't solve with SQL, you can turn to:
Reporting services: to create fancy reports
Analysis services: to analyse truly gigantic amounts of data (if you're thinking in millions of rows, Analysis server is overkill)
Integration Services: import from non Sql Server sources, automate tasks, combine queries graphically
These are Sql Server specific, and not as useful in every BI scenario.
The Microsoft BI stack is composed of:
SQL Server Reporting Services
SQL Server Analysis Services
SQL Services Integration Services
The combination of three will allow you to mine data, perform analytics and display them interactively.
I have bookmarked a few good links in my delicious account:
http://delicious.com/syalam/ssrs
Check thislink out - I learned from those screencasts:
http://sqlblog.com/blogs/denis_gobo/archive/2007/12/13/3937.aspx