(Disclaimer: T-SQL and SSRS expert, about two hours exposure to WIQL)
I have some users that created a Team Foundation Services (TFS) Work Items query using the UI designer that they're in love with, with hard-coded monthly parameters, and have asked me to explore converting this into an SSRS report where they can pass all the parameters they want.
To do that I'm guessing that I'd need to convert the query into T-SQL.
Exporting the report into a .wiq file, then reading the query in the xml within the .wiq file I was able to extract the below WIQL (slightly edited for confidentiality). Problem is, looking at my TFS server, which contains databases TFS_Coniguration, Tfs_CX, Tfs_DefaultCollection, TFS_IT, and TFS_Warehouse, I don't see a table or view named WorkItems anywhere, so I'm at a loss to translate the below WIQL into T-SQL.
If there's some kind of table alias stored in TFS that would give me clues, I'm not seeing it.
So, question: How can I convert the below WIQL into T-SQL?
SELECT
[System.Id], [System.AreaPath], [System.Title], [System.Tags], [System.AssignedTo],
[System.State], [foo.VSTS.Agile.Release], [System.CreatedDate], [Microsoft.VSTS.Scheduling.StoryPoints]
FROM
WorkItems
WHERE
[System.TeamProject] = #project AND
[System.WorkItemType] IN ('User Story') AND
[System.State] = 'Active' AND
[foo.VSTS.Agile.Release] <= '2018-06-10T00:00:00.0000000' AND
[foo.VSTS.Agile.Release] >= '2018-06-30T00:00:00.0000000' AND
[System.AreaPath] UNDER 'AppDev'
ORDER BY
[foo.VSTS.Agile.Release]
Daniel is right, it's not suggest to use SQL to query TFS operational store directly(Tfs_DefaultCollection), which may lose support from MS TFS support team.
One way to do this is using the database view
vw_denorm_WorkItemCoreLatest. In this case "Latest" means, you only
get the latest revision of the workitem without any previous versions
of it. If you need all versions of a workitem, use the view
vw_WorkItemCoreAll.
The example from the WIQL looks like:
SELECT [System.Id], [System.WorkItemType], [System.Title], [System.AssignedTo], [System.State]
FROM WorkItems
WHERE [System.TeamProject] = #project
and [System.WorkItemType] = 'Ticket'
and [System.State] <> 'Closed'
and [System.State] <> 'Removed'
Except the selected columns, a sample of SQL for your reference:
SELECT *
FROM [dbo].[vw_denorm_WorkItemCoreLatest]
WHERE [System.TeamProject] = 'MyTeamProject'
and [System.WorkItemType] = 'Ticket'
and [System.State] <> 'Closed'
and [System.State] <> 'Removed'
For warehouse, please take a look at How to query Work Items using SQL on the Relational Warehouse
I took a different approach. I have used a SSIS package to call the data to an Excel file and build a Data Mart out of it which supports Incremental Load. Then I am using an SSRS to easily use the data within my SQl table to build my report
Related
I'm following a tutorial on Azure Data Factory migration from Azure SQL to Blob through pipelines. While most of the concepts make sense, the 'Copy Data' query is a bit confusing. I have a background in writing Oracle SQL, but Azure SQL on ADF is pretty different and I'm struggling to find specific technical documentation, probably because it's not widely adopted yet.
Pipeline configuration shown below:
Query is posted below:
SELECT data_source_table.PersonID,data_source_table.Name,data_source_table.Age,
CT.SYS_CHANGE_VERSION, SYS_CHANGE_OPERATION
FROM data_source_table
RIGHT OUTER JOIN CHANGETABLE(CHANGES data_source_table,
#{activity('LookupLastChangeTrackingVersionActivity').output.firstRow.SYS_CHANGE_VERSION})
AS CT ON data_source_table.PersonID = CT.PersonID
WHERE CT.SYS_CHANGE_VERSION <=
#{activity('LookupCurrentChangeTrackingVersionActivity').output.firstRow.CurrentChangeTrackingVersion}
Output to the sink Blob as a result of the 'Copy Data' query:
2,name2,14,4,U
7,name7,51,3,I
8,name8,31,5,I
9,name9,38,6,I
Couple questions I had:
There's a lot of external referencing from other activities in the 'Copy Data' query like #{activity('...').output.firstRow.CurrentChangeTrackingVersion. Is there a way to know the appropriate syntax to referencing external activities? Can't find any good documentation the syntax, like what .firstRow is or what the changetable output looks like. I can't replicate this query in SSMS, which makes it a bit of a black box for me.
SYS_CHANGE_OPERATION appears in the SELECT with no table name prefix. Is this directly querying from the table in SourceDataset? (It points to data_source_table, which has table tracking enabled) My main confusion stems from how table tracking information is stored in the enabled tables. Is there a way to show all the table's tracked changes in SSMS? I see some documentation on what the return values, but it's hard for me to visualize it without seeing it on the table, so an output query of some return values would be nice.
LookupLastChangeTracking activity queries in all rows from a table (which when I checked, is just one row), but LookupCurrentChangeTracking activity uses a CHANGE_TRACKING function to pull the version of the data sink in table_store_ChangeTracking_version. Why does it use a function when the data sink's version is already recorded in table_store_ChangeTracking_version?
Sorry for the many questions, but I can't find any way to make this learning curve a bit less steep. Any guides or resources would be awesome!
There is an article to get the same thing done from the UI and it will help you understand it better .
https://learn.microsoft.com/en-us/azure/data-factory/tutorial-incremental-copy-change-tracking-feature-portal .
1 . These are the Lookup activity ,. very straight forward , please read about them here .
https://learn.microsoft.com/en-us/azure/data-factory/control-flow-lookup-activity
2.SYS_CHANGE_OPERATION is a column on data_source_table and so that should be fine . Regarding the details on the how the change tracking (CT) is stored , I am not sure if all the system table are exposed on Azure SQL , but we did had few table on the on-prem version of the SQL which could be queried if needed . But for this exercise I think that will be an over kill .
I am trying to see if there is an easy answer for this. I have done something similar using multiple pick dropdown parameters in SSRS but this appears to be different.
My scenario is this, so maybe there is an even better answer.
I have a production server that I do not want to make any changes to including temp tables or functions. The production server has a table of clients with about 1600 records. I have set up an SSIS package that will allow transfer of data from production to dev based on a clientid. So my sources would have a query similar to Select Field From Table Where ClientId = ?
This works fine. Now I want to load more than one client, based an data in the clients table. It may be Select ClientId From Clients where Field = A and returns multiple ClientIds.
I am able to populate a comma delimited list from an execute sql task to a SSIS variable, so it maybe 1,4,8.
If I change my source query to use ClientId in (?) I get a conversion error.
I have looked at many posts that advocate a temp table or a function which I want to avoid. Select IN using varchar string with comma delimited values
I have contemplated building the entire sql statement into a variable but this don't seem like the right path as I have many tables to query and transfer where using ClientId = ? works well without having to build each individual SQL statement to a variable.
Is there an easy fix I am missing? I will turn my research now to try to find out how I did this in SSRS but I thought that I should try a post here to see if someone has accomplished this before.
I appreciate any info on this, thank you.
EDIT: Key note is that the column on clients is on the dev server, so I cannot just use a select in the where clause as the column does not exist on the production server.
EDIT: I did not mention that I am specifically looking at OLEDB sources mapping a parameter to ? in the sql statement.
EDIT: Narrowing down on this but having trouble relating SSRS and SSIS functionality. In SSRS its called a multi-value parameter in the following link the key line is
WHERE Production.ProductInventory.ProductID IN (#ProductID)
https://msdn.microsoft.com/en-us/library/dn385719(v=sql.110).aspx
This one looks good as well
https://sqlblogcasts.com/blogs/simons/archive/2007/11/22/RS-HowTo---Pass-a-multivalue-parameter-to-a-query-using-IN.aspx
I will keep researching and thank you for the help so far.
I think this sums it up best
This functionality is limited to strictly using embedded SQL.
What SSRS does is transform your SQL column IN (#value) to column IN
(#selectedvalue1,#selectedvalue2) etc.
You need to forget anything you have about the other ways of passing
lists to SQL i.e. building strings etc. and make sure you declare the
data types are correct for the value of your parameter.
You do not need to use the Join(parameters!,",") trick UNLESS
you are passing the list to a stored procedure.
In which case you then need to use some function to turn the delimited
list into a rowset as you have done.
I hope that helps
The core question is if I can get the same functionality in SSIS as in SSRS. It reminds me of macro substitution..
If you dont want to create a function, you can use the following in your t-sql statement.
Declare #ClientIds nvarchar(50) = '123,456'; --<-- Comma delimited list of Client Ids
Select Field
From Table
Where ClientId IN (
SELECT CAST(RTRIM(LTRIM(Split.a.value('.', 'VARCHAR(100)'))) AS INT) ClientIDs
FROM (
SELECT Cast ('<X>'
+ Replace(#ClientIds, ',', '</X><X>')
+ '</X>' AS XML) AS Data
) AS t CROSS APPLY Data.nodes ('/X') AS Split(a)
)
I want to display all audit history data as per MS CRM format.
I have imported all records from AuditBase table from CRM to another Database server table.
I want this table records using SQL query in Dynamics CRM format (as per above image).
I have done so far
select
AB.CreatedOn as [Created On],SUB.FullName [Changed By],
Value as Event,ab.AttributeMask [Changed Field],
AB.changeData [Old Value],'' [New Value] from Auditbase AB
inner join StringMap SM on SM.AttributeValue=AB.Action and SM.AttributeName='action'
inner join SystemUserBase SUB on SUB.SystemUserId=AB.UserId
--inner join MetadataSchema.Attribute ar on ab.AttributeMask = ar.ColumnNumber
--INNER JOIN MetadataSchema.Entity en ON ar.EntityId = en.EntityId and en.ObjectTypeCode=AB.ObjectTypeCode
--inner join Contact C on C.ContactId=AB.ObjectId
where objectid='00000000-0000-0000-000-000000000000'
Order by AB.CreatedOn desc
My problem is AttributeMask is a comma separated value that i need to compare with MetadataSchema.Attribute table's columnnumber field. And how to get New value from that entity.
I have already checked this link : Sql query to get data from audit history for opportunity entity, but its not giving me the [New Value].
NOTE : I can not use "RetrieveRecordChangeHistoryResponse", because i need to show these data in external webpage from sql table(Not CRM database).
Well, basically Dynamics CRM does not create this Audit View (the way you see it in CRM) using SQL query, so if you succeed in doing it, Microsoft will probably buy it from you as it would be much faster than the way it's currently handled :)
But really - the way it works currently, SQL is used only for obtaining all relevant Audit view records (without any matching with attributes metadata or whatever) and then, all the parsing and matching with metadata is done in .NET application. The logic is quite complex and there are so many different cases to handle, that I believe that recreating this in SQL would require not just some simple "select" query, but in fact some really complex procedure (and still that might be not enough, because not everything in CRM is kept in database, some things are simply compiled into the libraries of application) and weeks or maybe even months for one person to accomplish (of course that's my opinion, maybe some T-SQL guru will prove me wrong).
So, I would do it differently - use RetrieveRecordChangeHistoryRequest (which was already mentioned in some answers) to get all the Audit Details (already parsed and ready to use) using some kind of .NET application (probably running periodically, or maybe triggered by a plugin in CRM etc.) and put them in some Database in user-friendly format. You can then consume this database with whatever external application you want.
Also I don't understand your comment:
I can not use "RetrieveRecordChangeHistoryResponse", because i need to
show these data in external webpage from sql table(Not CRM database)
What kind of application cannot call external service (you can create a custom service, don't have to use CRM service) to get some data, but can access external database? You should not read from the db directly, better approach would be to prepare a web service returning the audit you want (using CRM SDK under the hood) and calling this service by external application. Unless of course your external app is only capable of reading databases, not running any custom web services...
It is not possible to reconstruct a complete audit history from the AuditBase tables alone. For the current values you still need the tables that are being audited.
The queries you would need to construct are complex and writing them may be avoided in case the RetrieveRecordChangeHistoryRequest is a suitable option as well.
(See also How to get audit record details using FetchXML on SO.)
NOTE
This answer was submitted before the original question was extended stating that the RetrieveRecordChangeHistoryRequest cannot be used.
As I said in comments, Audit table will have old value & new value, but not current value. Current value will be pushed as new value when next update happens.
In your OP query, ab.AttributeMask will return comma "," separated values and AB.changeData will return tilde "~" separated values. Read more
I assume you are fine with "~" separated values as Old Value column, want to show current values of fields in New Value column. This is not going to work when multiple fields are enabled for audit. You have to split the Attribute mask field value into CRM fields from AttributeView using ColumnNumber & get the required result.
I would recommend the below reference blog to start with, once you get the expected result, you can pull the current field value using extra query either in SQL or using C# in front end. But you should concatenate again with "~" for values to maintain the format.
https://marcuscrast.wordpress.com/2012/01/14/dynamics-crm-2011-audit-report-in-ssrs/
Update:
From the above blog, you can tweak the SP query with your fields, then convert the last select statement to 'select into' to create a new table for your storage.
Modify the Stored procedure to fetch the delta based on last run. Configure the sql job & schedule to run every day or so, to populate the table.
Then select & display the data as the way you want. I did the same in PowerBI under 3 days.
Pros/Cons: Obviously this requirement is for reporting purpose. Globally reporting requirements will be mirroring database by replication or other means and won't be interrupting Prod users & Async server by injecting plugins or any On demand Adhoc service calls. Moreover you have access to database & not CRM online. Better not to reinvent the wheel & take forward the available solution. This is my humble opinion & based on a Microsoft internal project implementation.
I need some help from your side.
For example I have a SQL Server view MyView.
Adb.vs.MyView
with columns:
ID
Name
Address
Email
Phone
Logic behind the view is next
SELECT
Ac.AccountID AS ID,
Ac.AccountName AS Name,
Ad.Main_Adress AS Address,
Em.Main_Email AS Email,
Concat(Ph.Phone_Area_Code,Ph.Mobile_Phone_Number) AS Phone
FROM
Bdb.dbo.Account AS Ac
INNER JOIN
Cdb.dbo.Address AS Ad ON Ac.AccountID = Ad.AccountID
INNER JOIN
Cdb.dbo.Emails AS Em ON Ac.AccountID = Em.AccountID
INNER JOIN
Cdb.dbo.PhoneBook AS Ph ON Ac.AccountID = Ph.AccountID
NOTE:
No KEY relations build between all this tables.
My target to reverse engineer this view to get next kind of result:
Please suggest any kind of tool/tools or scripts to perform this.
Also, if somebody know similar solution but for Rev.En. stored procedures which was used to populating data into tables I will be really appreciated
bec. I will need to reverse tons of such kind views and stored procedures in nearest future.
Thanks in advance for any kind of support !
Take a look at the system views. Most of what you want is probably available in INFORMATION_SCHEMA.VIEW_COLUMN_USAGE. For example:
USE Adb
GO
Select * from INFORMATION_SCHEMA.VIEW_COLUMN_USAGE
where VIEW_NAME='MyView'
GO
Views are just stored SQL scripts that SQL Server adds into your query as a sub select. Consequently the fields used are not actually saved within SQL Server in the same way that table definitions are. Your best bet is to script out all your views using SQL Server Management Studio and plug the files into a tool such as the General SQL Parser which can output the columns and tables that are used in that script.
It isn't perfect but should get you a long way towards what you are trying to achieve. You can try it for free here.
I'm new to SQL queries and need to build a custom report in Microsoft SQL Report Builder 3.0. The data source is a SCCM database. I need help with understanding the best approach to achieve the following:
We need to cross reference if a computer exist in two Views, if so show that name in the report.
InputParameter1 = "Please select a View"
InputParameter2 = "Cross reference with this other View"
If I know the names of the Views beforehand I have a query to get what I need from the SQL server, but I need to create a parameter-based report where you select two Views dynamically and the report cross reference them for you and present which computers exists in both Views.
This is the query I can use for a static result. v_CM_RES_COLL_CMS0020B and v_CM_RES_COLL_CMS000D1 are examples of names of possible Views, until I can solve the parameter issue in Report Builder:
SELECT v_GS_SYSTEM.Name0
FROM v_GS_SYSTEM
WHERE Name0 IN
(SELECT Name from v_CM_RES_COLL_CMS0020B)
AND Name0 IN
(SELECT Name from v_CM_RES_COLL_CMS000D1)
I don't know how to proceed in how to make the above query into a parameter report in Report Builder. Somehow I need to change v_M_RES_COL_CMS00### to what ever the user inputs to the parameters. Does anyone know how? Any help is greatly appreciated.