I want to query Salesforce and save the results in a database. The first problem is that I find my payload is a ConsumerIterator. How can I process these results to store in a database?
<flow name="process.accounts" processingStrategy="synchronous">
<sfdc:select config-ref="MySQLDB" doc:name="Select Contact by Email">
<sfdc:parameterized-query><![CDATA[SELECT first_name,last_name,email
FROM Contact WHERE email=#[payload.email]]]></db:parameterized-
query>
</sfdc:select>
</flow>
<flow name="insert.contacts.to.audit" >
<vm:inbound-endpoint exchange-pattern="one-way" path="contact-audit"/>
<db:insert config-ref="Generic_Database_Configuration" doc:name="Save
Contact">
<db:parameterized-query><![CDATA[insert into user_audit v
values(?,?,?)]]></db:parameterized-query>
</db:insert>
</flow>
mulesoft exchange provides many common integration patterns such as the one you seek.
https://www.mulesoft.com/exchange/org.mule.templates/template-sfdc2db-account-bidirectional-sync/
use that template to quickly understand the how
Related
I am using Workato for my Salesforce to NetSuite integration and everything works perfectly with the exception of populating the createdFrom field on the Invoice record. Has anyone been successful in linking the NetSuite Invoice to its associated Sales Order using Web Services? This is critical as the linkage ensures that Revenue Arrangements are not duplicated in NetSuite.
I do not have first hand experience with SOAP Web Services but have seen via SuiteScript 2.x that one cannot simply create the record and try to fill in the createdFrom field. You must rather "transform" a Sales Order into an Invoice for example. The "transformation" is what links the two. Looks like for SOAP the operation is named "initialize / initializeList". The SOAP web services initialize operation emulates the UI workflow by prepopulating fields on transaction line items with values from a related record. Reference Suite Answer 10771 for more information. Suite Answer 91358 also has some sample code.
yes I have created invoices from Sales Order using SOAP Webservices. I am using minimum number of required fields to create the invoice.
Here is the endpoint URL that I used : https://(insert your account Id here).suitetalk.api.netsuite.com/services/NetSuitePort_2020_1
Note 1: You need to make sure that the version is consistent through out your SOAP request (in my case it's 2020_1)
Note 2: You can see that I am passing createdFrom tag as second last tag and I am passing Sales order's internal ID there. That means that invoice is being generated from that particular sales order and it carries over rest of the fields from sales order
Note 3: I am passing externalid of the invoice at the end which will the unique id of this newly created invoice across different systems
Please let me know how this goes!
<soap:Envelope soap:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/" xmlns:messages="urn:messages_2020_1.platform.webservices.netsuite.com" xmlns:accountingLists="urn:accounting_2020_1.lists.webservices.netsuite.com" xmlns:employeesLists="urn:employees_2020_1.lists.webservices.netsuite.com" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:employeesTransactionsTypes="urn:types.employees_2020_1.transactions.webservices.netsuite.com" xmlns:demandplanningTransactions="urn:demandplanning_2020_1.transactions.webservices.netsuite.com" xmlns:common="urn:common_2020_1.platform.webservices.netsuite.com" xmlns:commonTypes="urn:types.common_2020_1.platform.webservices.netsuite.com" xmlns:accountingListsTypes="urn:types.accounting_2020_1.lists.webservices.netsuite.com" xmlns:customizationSetup="urn:customization_2020_1.setup.webservices.netsuite.com" xmlns:inventoryTransactionsTypes="urn:types.inventory_2020_1.transactions.webservices.netsuite.com" xmlns:supportLists="urn:support_2020_1.lists.webservices.netsuite.com" xmlns:filecabinetDocuments="urn:filecabinet_2020_1.documents.webservices.netsuite.com" xmlns:bankTransactionsTypes="urn:types.bank_2020_1.transactions.webservices.netsuite.com" xmlns:communicationGeneralTypes="urn:types.communication_2020_1.general.webservices.netsuite.com" xmlns:customizationSetupTypes="urn:types.customization_2020_1.setup.webservices.netsuite.com" xmlns:supplychainListsTypes="urn:types.supplychain_2020_1.lists.webservices.netsuite.com" xmlns:core="urn:core_2020_1.platform.webservices.netsuite.com" xmlns:coreTypes="urn:types.core_2020_1.platform.webservices.netsuite.com" xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:supplychainLists="urn:supplychain_2020_1.lists.webservices.netsuite.com" xmlns:demandplanningTransactionsTypes="urn:types.demandplanning_2020_1.transactions.webservices.netsuite.com" xmlns:websiteLists="urn:website_2020_1.lists.webservices.netsuite.com" xmlns:salesTransactions="urn:sales_2020_1.transactions.webservices.netsuite.com" xmlns:salesTransactionsTypes="urn:types.sales_2020_1.transactions.webservices.netsuite.com" xmlns:relationshipsLists="urn:relationships_2020_1.lists.webservices.netsuite.com" xmlns:inventoryTransactions="urn:inventory_2020_1.transactions.webservices.netsuite.com" xmlns:employeesListsTypes="urn:types.employees_2020_1.lists.webservices.netsuite.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:customersTransactions="urn:customers_2020_1.transactions.webservices.netsuite.com" xmlns:schedulingActivitiesTypes="urn:types.scheduling_2020_1.activities.webservices.netsuite.com" xmlns:financialTransactions="urn:financial_2020_1.transactions.webservices.netsuite.com" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:relationshipsListsTypes="urn:types.relationships_2020_1.lists.webservices.netsuite.com" xmlns:employeesTransactions="urn:employees_2020_1.transactions.webservices.netsuite.com" xmlns:faults="urn:faults_2020_1.platform.webservices.netsuite.com" xmlns:marketingListsTypes="urn:types.marketing_2020_1.lists.webservices.netsuite.com" xmlns:communicationGeneral="urn:communication_2020_1.general.webservices.netsuite.com" xmlns:faultsTypes="urn:types.faults_2020_1.platform.webservices.netsuite.com" xmlns:supportListsTypes="urn:types.support_2020_1.lists.webservices.netsuite.com" xmlns:websiteListsTypes="urn:types.website_2020_1.lists.webservices.netsuite.com" xmlns:purchasesTransactions="urn:purchases_2020_1.transactions.webservices.netsuite.com" xmlns:financialTransactionsTypes="urn:types.financial_2020_1.transactions.webservices.netsuite.com" xmlns:schedulingActivities="urn:scheduling_2020_1.activities.webservices.netsuite.com" xmlns:bankTransactions="urn:bank_2020_1.transactions.webservices.netsuite.com" xmlns:marketingLists="urn:marketing_2020_1.lists.webservices.netsuite.com" xmlns:customersTransactionsTypes="urn:types.customers_2020_1.transactions.webservices.netsuite.com" xmlns:purchasesTransactionsTypes="urn:types.purchases_2020_1.transactions.webservices.netsuite.com" xmlns:generalTransactions="urn:general_2020_1.transactions.webservices.netsuite.com" xmlns:filecabinetDocumentsTypes="urn:types.filecabinet_2020_1.documents.webservices.netsuite.com">
<soap:Header>
<urn:tokenPassport xmlns:urn="urn:messages_2020_1.platform.webservices.netsuite.com">
<ns8:account xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com">XXX</ns8:account>
<ns8:consumerKey xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com">XXX</ns8:consumerKey>
<ns8:token xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com">XXX</ns8:token>
<ns8:nonce xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com">XXX</ns8:nonce>
<ns8:timestamp xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com">XXX</ns8:timestamp>
<ns8:signature xmlns:ns8="urn:core_2020_1.platform.webservices.netsuite.com" algorithm="HMAC_SHA256">XXX</ns8:signature>
</urn:tokenPassport>
<preferences xmlns="urn:messages_2020_1.platform.webservices.netsuite.com"/>
</soap:Header>
<soap:Body>
<add>
<record xmlns:salesTransactions="urn:sales_2020_1.transactions.webservices.netsuite.com" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="salesTransactions:Invoice">
<itemList>
<item xsi:type="salesTransactions:InvoiceItem">
<orderLine xsi:type="salesTransactions:long">1</orderLine>
<quantity xsi:type="salesTransactions:double">1</quantity>
<rate xsi:type="salesTransactions:string">30000</rate>
<description xsi:type="salesTransactions:string">Annuity test</description>
</item>
</itemList>
<tranDate xsi:type="salesTransactions:dateTime">2020-09-09T00:00:00.000</tranDate>
<createdFrom xmlns:core="urn:core_2020_1.platform.webservices.netsuite.com" xsi:type="core:RecordRef" internalId="33202933"/>
<externalId xmlns:core="urn:core_2020_1.platform.webservices.netsuite.com" xsi:type="core:RecordRef">INV000089099</externalId>
</record>
</add>
</soap:Body>
</soap:Envelope>
I've created an SP in SQL Server that returns as XML. I decided to do this as the information has contacts and addresses in it and I wanted to reduce the amount of data I get.
<Accounts>
<Account>
<Company />
<ContactNumber />
<Addresses>
<Address>
<Line1 />
....
</Address>
<Addresses>
<Contacts>
<Contact>
<Name />
....
</Contact>
<Account>
</Accounts>
I have found SqlCommand.ExecuteXmlReader but I'm confused as to how to serialise this into my POCO. Can someone point me at what my next step is. (The POCO was created by the Insert XML as a class menu item in VS2019).
My Google fu is letting me down as I'm not sure what I should be looking for to help understand how to serialize the XML into something that will allow me to go with Foreach Account in AccountsClass type logic.
Any help is greatly appreciated.
PS The XML above is just a sample to show what I'm doing. The actual data is over 70 fields and with two FK joins the initial 40000 rows is well in excess of 1.8 million once selected as a normal dataset.
EDIT: Just in case someone stumbles on this and are in the same situation I was in.
When preparing a sample record for the Past XML to class make sure you have more than one record if you are expecting something similar to my example above. (The class changes to support more than one record.)
You get very different results when searching for deserialize when doing your research. This small change resulted in me figuring out the issue.
I am running multiple select queries and I want them to run one after the other.
In this example, I select account numbers then use those numbers in the following query. Will the queries run consecutively one after the other i.e. next query only runs after the previous query has finished. Do I need to wrap them in a composite-source and wrap them in a transaction? What would that look like?
<flow name="PopulateAccount">
<db:select config-ref="dsConfig" doc:name="Get Account ID">
<db:paramterized-query><![CDATA[
SELECT ACC_NUM....
</db:select>
<custom-transformer class="com.vf.ListTransformer">
<set-session-variable variableName="messageID" value="#[payload]"
doc:name="Set Account IDs"/>
<!-- The next query depends on the Account IDS from
previous results in the session variable -->
<db:select config-ref="dsConfig" doc:name="Get Account Detail">
<db:paramterized-query><![CDATA[
SELECT ACC_NAME,....
</db:select>
<custom-transformer class="com.vf.AccountsTransformer">
<!-- More database operations --->
</flow>
Will the queries run consecutively one after the other
yes, as long as you do not take any measures to run them in parallel, like for example moving the database component in a separate flow and call it in a asynchronous way.
Do I need to wrap them in a composite-source
no, especially not if you use the result of the first query in the second query (like in your example)
and wrap them in a transaction
why? you are not inserting/updating anything in your example.
What would that look like?
just like in your example. the only thing i would change is the way how you store the result of your first query. although there is nothing wrong with using set-variable i prefer using a enricher to store result of a component in variable instead of changing the payload and setting the variable afterwards.
<flow name="testFlow">
<enricher target="#[flowVars.messageID]" doc:name="Message Enricher">
<db:select config-ref="MySQL_Configuration" doc:name="select ACC_NUM">
<db:parameterized-query><![CDATA[SELECT ACC_NUM ...]]></db:parameterized-query>
</db:select>
</enricher>
</flow>
I am working on a flow that is supposed to take a CSV file from the system and insert the contained data into a database. Some of the records in the file have the wrong format required(Eg: wrong number of columns) and hence must me written into another file of text type for analysis.
I have created a flow that inserts all the good records into the database but it does not input the bad records into a file. I am currently a beginner and hence am not sure how to proceed with it.
The XML code:
<flow name="fileFlow">
<file:inbound-endpoint path="src/main/resources/Input" moveToPattern="#[message.inboundProperties.originalFilename].zip" moveToDirectory="src/main/resources/Output" responseTimeout="10000" metadata:id="b85f6b05-1679-4b60-8bbe-30e6d2c68df7" doc:name="File">
<file:filename-regex-filter pattern=".*csv" caseSensitive="true"/>
</file:inbound-endpoint>
<file:file-to-string-transformer doc:name="File to String"/>
<set-payload value="#[payload.replaceAll(",,", ", ,")]" doc:name="Set Payload"/>
<splitter expression="#[rows=StringUtils.split(message.payload,'\r\n');ArrayUtils.subarray(rows,1,rows.size())]" doc:name="Splitter"/>
<flow-ref name="fileFlow1" doc:name="fileFlow1"/>
<catch-exception-strategy doc:name="Insert the bad record into a file">
<byte-array-to-string-transformer doc:name="Byte Array to String"/>
<set-session-variable variableName="var" value="#[payload+'var']" doc:name="Session Variable"/>
<file:outbound-endpoint path="src/main/resources/Output" outputPattern="BadRecords.txt" responseTimeout="10000" doc:name="File"/>
<flow-ref name="fileFlow1" doc:name="fileFlow1"/>
</catch-exception-strategy>
</flow>
<flow name="fileFlow1">
<expression-transformer expression="#[StringUtils.split(message.payload,',')]" doc:name="Expression"/>
<db:insert config-ref="MySQL_Configuration" doc:name="Database">
<db:parameterized-query><![CDATA[insert into GoodRecords values(#[message.payload[0]], #[message.payload[1]], #[message.payload[2]], #[message.payload[3]], #[message.payload[4]], #[message.payload[5]], #[message.payload[6]], #[message.payload[7]], #[message.payload[8]], #[message.payload[9]], #[message.payload[10]], #[message.payload[11]], #[message.payload[12]], #[message.payload[13]], #[message.payload[14]], #[message.payload[15]], #[message.payload[16]], #[message.payload[17]], #[message.payload[18]], #[message.payload[19]], #[message.payload[20]])]]></db:parameterized-query>
</db:insert>
<logger message="#[payload] " level="INFO" doc:name="Logger"/>
</flow>
The flow structure:
Flow diagram
Personally I think the flow I've created is very inefficient and wrong.
How do I enter the bad records into the file(if the given flow is right)?
I wanted to use bulk mode for the given use case (as there are around 1000 odd records to work with) but am not sure how to proceed with that as well.
In your code you have used a private flow for inserting your record to database. So any exception occurred during insertion will not be catch by the parent flow exception strategy, you can have separate exception strategy for your private flow or you may use sub-flow.
There is another clean solution for that you can use batch processing to process these records and create a separate step to handle all failure records.
I have a mapping that retrieves all active roles for a user. I use the where attribute to filter out roles in the hbm mapping. The mapping look like this:
<map name="Bar" table="Foo_Bar" lazy="true" cascade="all" inverse="false" where="intGroupId Is Null And dtmExpires > getdate()">
<cache usage="read-write"/>
<key column="intUserId"/>
<index column="varRole" type="string"/>
<one-to-many class="Foo.Bar, Foo"/>
</map>
This works great in production on a SQL Server, but in my unit tests where I use SQLite the getdate() function isn't recognized.
How can I modify my mapping so it works in both MS SQL Server and SQLite, but still have the filter?
// Johan
I don't know much about SQLite but you may be able to use
intGroupId Is Null And dtmExpires > CURRENT_TIMESTAMP
CURRENT_TIMESTAMP is the SQL Standard equivalent of getdate(), and I believe it is implemented in SQLite (and I'm positive it is in SQL Server).
You may be able to use the technique discussed here to tweak your mappings at runtime if this doesn't work.