I have the following issue in a copy activity:
"Code": 23605,
"Message": "ErrorCode=DynamicsOperationFailed,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Dynamics operation failed with error code: -2147088238, error message: A record that has the attribute values SAP Client, SAP System Id, OrderPOS-ID, Order ID, SAP Module, Account Assignment ID already exists. The entity key Purchase Order Atl Key requires that this set of attributes contains unique values. Select unique values and try again..,Source=Microsoft.DataTransfer.ClientLibrary.DynamicsPlugin,''Type=System.ServiceModel.FaultException`1[[Microsoft.Xrm.Sdk.OrganizationServiceFault, Microsoft.Xrm.Sdk, Version=9.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35]],Message=The creator of this fault did not specify a Reason.,Source=Microsoft.DataTransfer.ClientLibrary.DynamicsPlugin,'";
I'm using a dynamic query(with a store procedure) and load that based on config table where i have all entities to load. And Recently we start to get this error messages for some countries(but there's no pattern). The entity in this case is Purchase Orders and the processes load part of the data to dynamics, but some countries are failing. I've already checked all possible duplicates and I didn't find none using the keys presented in message above;
This fields are defined as alternatekeys on dynamics in order to avoid duplicates, and the alternatekey are composed with: SAP Client, SAP System Id, OrderPOS-ID, Order ID, SAP Module, Account Assignment ID;
In order to make our process faster I separate PO(PurchaseOrders) in 2 view: (1)Purchase_Order_master : basically copy to dynamics new records with all keys and after this first view run we have the (2)view that got all attributes and do the upsert of all values that suffered some update on source side.
In this 2 view we have the following field in common, these are the only fields in common
sie_purchaseordersid(uniqueidentifier, null)
Executionid(uniqueidentifier, not null)
Hashkey(varbinary(32), not null)
The fields that compose the hashkey are the same that we can find on the alternate key
,HASHBYTES('SHA2_256', CONCAT(SAPSYSID,SAPMOD,MANDT,EBELN,EBELP,TRY_CONVERT(NVARCHAR(255),TRY_CONVERT(INT,ZEKKN)))) HashKey
I don't have more ideas, and i hope one of you already tackled a similar issue in a passed and can provide some help.
Thanks in advance
I've try to find duplicates based on the fields that were mention in log message, but i didn't find nothing;
Related
I'm trying to think of the easiest way for non-tech users to dump info into a database, without coding my own web application.
Essentially, they are recording subjective phone grading scores for employees.
I linked an Access form to our MS SQL Server database. The only validation I want it -- I want one field, 'employee' - to be validated against a list of employees from say table.employee on SQL Server.
Once the form is submitted it will be written to table.scorecard -- or what have you.
Is this possible in Access? Their standard validation rules don't seem to cover this. Also, is there simply a better way to accomplish this task in general? Thanks
There are two ways to solve this problem.
The simplest is to use a combobox field for your employee information. Use the employee table as the list data source for the combobox and then set the LimitToList property to true. This assumes that you have setup linked table connections for both your employee table and your 'scores' table.
The second solution is to create a foreign key between the employeeId (or whatever the key field is) in the scores table and the employee table on the SQL side. If someone tries to insert an invalid employee, you will get an insert error. Unfortunately, SQL errors tend to be very confusing to most Access users.
If you want to be very through, you could implement both solutions, this would prevent someone going straight to the linked tables and putting in bad data.
I just realized that I am assuming that you are doing proper relational design where the 'scores' table would contain the employeeId rather than a full name. The idea on the form is to have the combobox display the name, but insert the employeeid field.
I'm currently solving this task: some data should be sent from AdventureWorks2012 to Anchor model tables on the same server in MsSQL.
This is my Anchor Model
At this point I have a pretty simple Integration Services project in Visual Studio and it looks like this.
Control flow:
For example Load_territories is:
The main requirement is to fill all tables of Anchor model tables in MsSQL but I'm constantly facing a problem: the amount of attributes in tables are different and some of them are repeating
At this picture in the second table basically TR_ID,TR_GRP_TR_ID, TR_TID_TR_ID, TR_TNM_TR_ID contain the same values from dwh_key but it's impossible to create a one-to-many relation between attributes. My tutor has recommended me to use Lookup but I cannot figure out how to implement them in this project
This may be considered as cheating, but if you insert data into the latest view rather than the separate 6NF tables all of those ID fields will be populated by underlying trigger logic. I suspect that this would defeat the purpose of using SSIS though, since you would effectively be loading attributes sequentially rather than in parallel.
Another option is to leave surrogate key management to the ETL tool. This would require that you switch the data type for your identities from integers to GUID:s. SSIS can then generate a GUID and you can then use that very same GUID to populate all the attributes. Note that the anchor would have to be loaded first, or you will get a foreign key violation.
The most common solution though, is to leave surrogate key management to the database (and use integers). You would have a step in which you populate the metadata column in the anchor with the desired number of new identities to be created. Using the metadata number you can then select the newly generated identities and merge them into your data flow. It doesn't matter which number gets assigned to which row. After that all attributes can be populated in parallel, including their ID columns.
Of course, if this is intended to be used for more than an initial load, you would also have to add steps to detect if the data you are loading is already known or not.
I can also recommend watching the video tutorial referenced in this blog post: https://clinthuijbers.wordpress.com/2013/06/14/ssis-anchor-modeling-example-tutorial/
I think what I am doing is quite simple but hitting an unexpected error.
When I navigate the Salesforce.com Object Database using Power Query, and attempt to access the Activity History table I receive the following error.
DataSource.Error: entity type OpenActivity does not support query
Details:
List
Is there such thing as unqueryable tables from Salesforce Object Library or can I access the table using a different method in PowerQuery? List doesn't seem to work...
let
Source = Salesforce.Data("https://login.salesforce.com/", [CreateNavigationProperties = true]),
SFDC_ActivityHistory = SFDC{[Name="ActivityHistory"]}[Data]
in
SFDC_ActivityHistory
I do my usual login and then provide organization wide security credentials. Other tables work fine like accounts and opportunities.
ActivityHistory Table cannot be accessed directly, but you could get query access to it as the child relationship of a primary object. For instance:
Select Name, (Select Subject, ActivityType from ActivityHistories) from Account
Please refer to the last section, "Usage", of the document: https://developer.salesforce.com/docs/atlas.en-us.api.meta/api/sforce_api_objects_activityhistory.htm
I have a Person database in SQL Server with tables like address, license, relatives etc. about 20 of them. All the tables have id parameter that is unique per person. There are millions of records in these tables. I need to combine theses records of the person using their common id parameter, and convert to a json table file with some column name changes. This json file then gets pushed to kafka through a producer. If I can get the example with the kafka producer as item writer- fine, but real problem is understanding the strategy and specifics on how to utilize spring batch item reader, processor, and item writer to create the composite json file. This is my first Spring batch application so I am relatively new to this.
I am hoping for the suggestions on the implementation strategy using a composite reader or processor to use person id as the cursor, and query each table using the id for each table , convert the resulting records to json and aggregate it to a composite, relational json file with root table PersonData that feeds to kafka cluster.
Basically I have one data source, same database for the reader. I plan to use Person table to fetch id and other records unique for the person, and use id as the where clause for 19 other tables. convert each resultset from the table to json, and composite the json object at the end and write to kafka.
We had such an requirement in a project and solved it with the following approach.
In Splitflow, that run parallel, we had a step for ever table that loaded the data of the table in the file, sorted by common id (this is optional, but it is easier for testing, if you have the data in files).
Then we implemented our own "MergeReader".
This mergereader had FlatFileItemReaders for every file/table (let's call them dataReaders). All these FlatFileItemReaders were wrapped with a SingleItemPeekableItemReader.
The logic for the read method of the MergeReader is as follows:
public MyContainerPerId read() {
// you need a container to store the items, that belong together
MyContainerPerId container = new MyContainerPerId();
// peek through all "dataReaders" to find the lowest actual key
int lowestId = searchLowestKey();
for (Reader dataReader : dataReaders) {
// I assume, that more than one entry in a table can belong to
// the same person id
wihile (dataReader.peek().getId() == lowestId) {
{
container.add(dataReader.read());
}
}
// the container contains all entries from all tables
// belonging to the same person id
return container;
}
If you need restart capability, you have implement ItemStream in a way, that it keeps track of the current readposition for every dataReader.
I used the Driving Query Based ItemReaders usage pattern described here to solve this issue.
Reader: just a default implementation of JdbcCursoritemReader with sql to fetch
the unique relational id (e.g. select id from person -)
Processor: Uses this long id as the input and a dao implemented by me using
jdbcTemplate from spring fetches data through queries against each of
the table for a specific id (e.g. select * from license where id=) and map results in list format to a POJO
of Person - then convert to json object (using Jackson) and then to
string
Writer: either write the file out with json string or publish json string to a
topic in case of kafka use
We went through similar exercise migrating 100mn + rows from multiple tables as a form of JSON so that we can post it to a message bus.
The idea is create a view, de-normalize the data and read from that view using JdbcPagingItemReader.Reading from one source has less overhead.
When you de-normalize the data make sure you do not get multiple rows for master table.
Example - SQL server -
create or alter view viewName as
select master.col1 , master.col2,
(select dep1.col1,
dep1.col2
from dependent1 dep1
where dep1.col3 = master.col3 for json path
) as dep1
from master master;
The above will give you dependent table data in a json String with one row for each master table data. Once you retrieve the data you can use GSON or Jackson to convert it as POJO.
We tried to avoid JdbcCursoritemReader as it will pull all data in memory and read one by one from it. It does not support pagination.
I am trying to work out the best database model for the current setup:
An administrator can create "customer products". This means services/products which customer can attach/subscribe to. The simple cases where the product simply costs a price, or the product subscription should send an e-mail is easy to model in the database.
But how about very specific backend code for a customer product? For example, one product might have very specific code implemented for checking a customer status on a different database. How can I map this relationship in the database so I can turn on/turn off some code based on the product settings.
My intuitive way of handling it would be to have a string column on the CustomerProducts table where a pre-defined set of strings could be set, e.g. "MyCustomCodeHandler", and then the code would check for the existence of this string in order to execute it. But for me it doesn't really feel like a real relationship between the database and code.
Data is data, whereas code is code. I would not recommend storing code in the database.
If you need to allow customers to create product types (in the object-oriented sense of "types") with associated code, I'd choose to deploy that code in the same way you deploy other code.
The custom code may also reference custom data stored in the database. I'd choose to create a dependent table per product subtype, and put the type-specific columns in there. The relationship between this subtype table and the generic product table is one-to-one. That is, the primary key in the subtype table is also a foreign key to the generic product table.