I would like to cross-reference construct a distance chart similar to the one here (example is a road-distance cross-reference chart) and, ideally, store the data in SQL Server 2008 (preferably the Express version). It needs these properties / abilities
Every column has a corresponding row with the same name (ie. not misspelled like my example).
Changing the value at one Row-Column intersection would update the mirror intersection (Column-Row) or the mirror data could be ignored.
The distance-values would need to be end-user editable.
The end-user would need to be able to add, delete or rename a column/row pair.
The end-user needs to be able to sort the columns and have the rows move automatically.
There could be hundreds of pairs.
a look-up query needs to find a distance given a start & destination (Row & Column)
The distance chart is reasonably straightforward to implement in Excel. Considering this, am I better off...
Using Excel as the user editing UI and then updating an SQL 'thing' with the new data?
Using Excel as the data-source even if it means performance issues with querying the data?
Using an as-yet undiscovered stroke of genius detailed here in an answer?
Sure looks like an Excel application to me, start to end. (heh)
I can't imagine your users typing enough data in to make performance an issue. Excel will only take 32757 rows by ditto columns. If that's enough, I'd say you're golden.
Related
I have a SQL Table with close to 2 Million rows and I am trying to export this data into an Excel file so the stakeholders can manipulate data, see charts, so on...
The issue is, when I hit refresh, it fails after getting all the data saying the number of rows exceed max rows limitation in Excel. This table is going to keep growing every day.
What I am looking for here is a way to refresh data, then add rows to Sheet 1 until max rows limitation is reached. Once maxed out, I want the rows to start getting inserted into Sheet 2. Once maxed out, move to 3rd sheet, all from the single SQL table, from a single refresh.
This does not have to happen in Excel (Data -> Refresh option), I can have this as a part of the SSIS package that I am already using to populate rows in the SQL table.
I am also open to any alternate ways to export SQL table into a different format that can be used by said stakeholders to create charts, analyze data, and whatever else pleases them.
Without sounding too facetious, you are suggesting a very inefficient method.
The best way of approaching this method is not to use .xlsx files at all for the data storage.
Assuming your destination stakeholders don't have read access to the SQL server, export the data to .csv and then use Power Query in some sort of 'Dashbaord.xlsx' type file to load the .csv to the data model which can handle hundreds of millions of rows instead of just 1.05m.
This will allow for the use of Power Pivot and DAX for analysis and the data will also be visible in the data model table view if users do want raw rows (or they can refer to the csv file..).
If they do have SQL read access then you can query the server directly so you don't need to store any rows whatsoever as it will read directly.
Failing all that and you decide to do it your way, I would suggest the following.
Read your table into a Pandas df and iterate over each row and cell of the dataframe, writing to an your xlsx[sheet1] using openpyxl then once the row number reaches 1,048,560 simply iterate to xlsx[sheet2].
In short: openpyxl allows you to create workbooks, worksheets, and write to cells directly.
But depending on how many columns you have it could take incredibly long.
Product Limitations
Excel 2007+ 1,048,576 rows by 16,384 columns
A challenge with your suggestion of filling a worksheet with the max number of rows and then splitting is "How are they going to work with that data?" and "Did you split data that should have been together to make an informed choice?"
If Excel is the tool the users want to use and they must have access to all the data, then you're going to need to put the data into a Power Pivot data model (and yes, that's going to impact the availability of some data visualizations). A Power Pivot model is an in-memory tabular data set. What the means is that the data engine, xVelocity, is going to use a bunch of memory but can get over the 1 million row limitation. Depending on how much memory is required, you might need to switch from the default 32 bit Office install and go with a 64 bit install (and I've seen clients have to max RAM out on old, low end desktops because they went cheap for business users).
Power Pivot will have a connection to your SQL Server (or other provider). When it refreshes data, it's going to fire off queries and determine the unique values in columns and then create a dictionary of unique values. This allows it to compress the data with low cardinality really well - sales dates are likely going to be repeated heavily within your set so the compression is good. Assuming your customers are typically not-repeat customers, a customer surrogate key would have high cardinality and thus not compress well since there's little to no repeat. The refresh is going to be dependent on your use case and environment. Maybe the user has to manually kick it off, maybe you have SharePoint with Excel services installed and then you can have it refresh the data on various intervals.
If they're good analysts, you might try turning them on to Power BI. Same-ish engine behind the scenes but built from the ground up to be an response reporting tool. If they're just wading through tables of data, they're not ready for PBI. If they are making visuals out of the data, PBI is likely a better fit.
In my SSAS tabular model I have a calendar dimension, and a wave dimension for half-year data only. The data flows to these tables as such:
Fact Table ---> Wave Dim <---> Calendar
No matter what I do I cannot hide the wave table from Excel users. The table shows hidden in Visual Studio, but in Excel it shows the table with no fields in it. I have tried deleting the table and reloading it with no avail. My assumption is that it has to do with the way it's connected to the Calendar dimension, but I can't seem to find anything on my issue. Any help would be much appreciated.
This is a tough one to answer without viewing your model.bim file. Based on your information I have two guesses on what could be the issue, although this may very well not be it:
1) It sounds like you've marked all fields in the Wave table as hidden, but not the Wave table itself. Could this be the reason?
2) Perhaps you are using perspectives, and only hid the Wave table in a perspective rather than in the Model (default perspective)?
The relationships in your model should have no impact on whether a table is displayed in client tools or not.
Feel free to upload your model.bim file if the above does not help.
I have a SQL View that I'm working on that spits out some important information for my boss's boss's boss. The view includes a field called Item ID, which can be in several different formats.
Here are some examples (that may or many not be made up to protect the innocent):
ATS-LC-PLN-RT-RH-0.3125-18-3X2.125X1.5-1
012345.012345
01234567.0123
123456789012
000000.000000
000000.000002
I'd like to take the view and use it to (eventually) produce an excel spreadsheet, but I'm not confident that there's a way to format this column in a way that will work for all of these different Item ID's.
When playing around with Excel, these numbers drop their trailing zeroes and switch to scientific notation, among other shenanigans. I just need to format this column in a way that will preserve the Item ID.
If you know of a way to programmatically create an excel spreadsheet in a way that allows me to assign a format based on the data in the cell, that would work great. The problem that I'm mainly suffering from is that this spreadsheet naturally has hundreds of lines, soon to be thousands, and there's no feasible way to hand-format these lines one at a time on a daily or weekly basis.
I've got SQL-Server 2014 and Excel via Microsoft Office Standard 2013, which may offer more options.
Permit me to suggest another way of framing your issue. I don't think you really want to analyze (either manually or programmatically) each item ID and determine whether it is an integer, a decimal, or alphanumeric text. Since your item ID data varies, the only Excel formatting that will work for all of your cases is 'Text.' So my suggestion is look for a way to automate the export of your data to Excel while making sure that the formatting in Excel is set to 'Text' for all cells to contain your item ID data. As you've noticed, if you are pasting data in Excel, if the target cells are not first set to 'Text' formatting, Excel will make its own 'corrections' to each pasted value, including removal of leading and trailing zeros.
The best solution is to use SQL Server Reporting Services (SSRS). You can set the field formatting in SSRS, and then (if you choose) automate the export of your data to Excel by calling the report server by URL with &rs:Format=excel. (There is learning curve for SSRS but if you plan to continue doing things like this, it will be worth it.)
Other options
The easiest manual option is to 1) export the data to .csv format, 2) Open Excel and use the Text Import Wizard, and during Step 3 make sure to click the data column and then choose 'Text' as the data format. (You could automate this somewhat with an Excel VBA macro.)
The most complicated method involves programming using Excel VBA and ADO to automate the connection and querying of the data from your database view, and then rendering that data to a spreadsheet, using VBA to set the formatting to 'Text.'
I have the following problem:
When I process a cube in SSAS and view the data as a normal user without restrictions all is well and the aggregated results show their normal values.
When I choose inside Data Tools (MsSQL 2012) to view the data as a specific role then all the security constraints work normally (For example I view only the Specific country data and no other) but all the corresponding values are (null).
I have tried Visual Totals = true and I have imposed those restrictions on both the dimension and the cube. The CALCULATE in calculation script is still there (everything works ok when I choose to use a normal user).
This is happening on both the measures and the calculated values!!!
Any ideas what could be happening?
Solved.
This problem had to do with cell data!!!
The Data Tools studio automatically checks the enable Cell Data permissions checkboxes in the respective tab.
If no MDX query is defined in these, then it is automatically assumed that you have no permissions to view any aggregated measure data!!!!
Uncheck them or define the proper MDX query and the problem will be fixed!!!!
We have a very sparse cube in SSAS. The size of the cube is only 50M bytes in MOLP storage. When writing your own MDX query, you can use NON EMPTY/NONEMPTY to suppress/filter out empty results. But we have a 3rd party tool (Cognos Analysis Studio) for our Business Users to run ad-hoc analysis against the cube. At first the users drag in dimensions and measures using Cognos Analysis Studio, the initial results come out with empty results. Then the users try to suppress the empty results. It takes an extremely long time to come up with non-empty results.
Since the MDX is generate by Cognos, we don't have control over how the MDX is generated to run against the cube. Our users using this cube are only interested in non-empty results. We'd like to know if there is a behavior control in a SSAS cube to always return non-empty results.
Question:
How can we set (or achieve) a cube-wide behavior to always suppress empty results?
I think the feature you're looking for is called "zero suppression". Click on “Settings”, “Suppress”,
“Zeros and Empty Cells”. This changes the default which will only suppress empty cells and not cells filled with zeros. Or, click on a column in the report. Click on the “Zero Suppression” icon at the top. Any row that is filled with zeroes will disappear.
You can google on "zero suppression cognos" to get more explanations about this.
I don't know if Cognos Analysis Studio allow using sets. If so, try creating non empty of dimension attribute against default cube measure in MdxScript. Then use this set instead of original dimension attribute.
For example
CREATE CURRENTCUBE.[Set1] as NonEmpty([Dim1].[Atr1].allmembers, [Measures].DefaultMember);