Using Groovy how could I change the Physical layer settings in all ODI maps, including PL / SQL code in Begin Mapping Command & End Mapping Command and sets the Technology for Begin Mapping Command/Location for Begin Mapping Command & Technology for End Mapping Command/Location for End Mapping Command.
Thank you.
The IMappingFinder interface lets you retrieve a collection of mappings (by project for instance).
For each element in the collection you can then use getPhysicalDesign method to retrieve the physical design(s) of the mapping. From there, you can use the following methods to set the code, the technology and the logical schema :
setBeginCmd
setBeginCmdLogicalSchema
setBeginCmdTechnology
setEndCmd
setEndCmdLogicalSchema
setEndCmdTechnology
However, if you need to add some code at the beginning and end of every mapping, it might be a good candidate for a custom Knowledge Module instead. You could duplicate your IKM, add a step at the beginning and one at the end and make sure you use the new KM. That's what KM are used for. Here is an example : https://blogs.oracle.com/dataintegration/odi-12c-components-and-lkmsikms
The full documentation is here : https://docs.oracle.com/en/middleware/fusion-middleware/data-integrator/12.2.1.3/odikd/developing-knowledge-modules-oracle-data-integrator.pdf
I cannot import a database in AWS RDS because of this commands in my sql file:
SET ##SESSION.SQL_LOG_BIN= 0;
SET ##GLOBAL.GTID_PURGED=/*!80000 '+'*/ '';
SET ##SESSION.SQL_LOG_BIN = #MYSQLDUMP_TEMP_LOG_BIN;
Are they important ? whiteout them there is no error.
log_bin_trust_function_creators parameter is set to 1 in a custom parameter.
FYI: MySql 5.7 and 8, same error
ERROR 1227 (42000) at line 20: Access denied; you need (at least one of) the SUPER, SYSTEM_VARIABLES_ADMIN or SESSION_VARIABLES_ADMIN privilege(s) for this operation
SET ##SESSION.SQL_LOG_BIN=0;
This is telling MySQL not to put these INSERT statements into the binary log. If you do not have binary logs enabled (not replicating), then it's not an issue to remove it. As far as I know, there is no way to enable this in RDS; I'm actually trying to figure out a way, which is how I found your question.
SET ##GLOBAL.GTID_PURGED=/*!80000 '+'*/ '';
Did you execute a RESET MASTER on the database where the dump originated from? Check here for an explanation of this value: gtid_purged
SET ##SESSION.SQL_LOG_BIN = #MYSQLDUMP_TEMP_LOG_BIN;
This is setting the ##SESSION.SQL_LOG_BIN variable back to the original setting; you should've seen another line like: SET #MYSQLDUMP_TEMP_LOG_BIN = ##SESSION.SQL_LOG_BIN;
If you're simply recovering this table into a new database that isn't writing to a binary log (for replication), it's safe to remove these lines. Hope this helps!
My odbc connection string for connecting to DB2i looks like this:
Driver={Client Access ODBC Driver (32-bit)};system=xx.xx.xx.xx;dbq=LIB1 LIB2 LIB3 LIB4 LIB5 LIB6 LIB7 LIB8;languageid=ENU;qrystglmt=-1;cmt=0;signon=1
The above connection string specifies multiple libraries/schemas for use. But when I try to access a file/table from a library other than the first one (like from LIB2...LIB8) I get a exception saying "FILE xx not found in LIB1"
Why does not it automatically search for the file/table in other specified libraries, it searches for the file in the first library only?
I need a workaround for this situation.
Use "system naming" mode, by adding naming=1 to your connection string.
In your library list, place a comma before the first library.
Driver={Client Access ODBC Driver (32-bit)};system=systemname;naming=1;
dbq=,LIB1,LIB2,LIB3,LIB4,LIB5,LIB6,LIB7,LIB8;languageid=ENU;cmt=0;signon=1
This works as documented in the manual:
The library list is used for resolving unqualified stored procedure calls and finding libraries in catalog API calls. ...
Note:
The first library listed in this property will also be the default library, which is used to resolve unqualified names in SQL statements.
As stated above, Schema/library list is used to resolve functions/procedure names, not tables.
Let assume you need to read data from lib1.tab1 and lib2.tab2;
Here my personal workarounds (from easy to complex):
a) ask the db admin to have - for each table you need to use - the corresponding schema name, then do "select * from lib1.tab1 join lib2.tab2 on [...]" ;-)
b) ask the db admin to create on schema "MyAlias" several alias (create alias) for each table you want to use. Then do "set current schema=MyAlias" followed by all the SQL statement you need e.g. "select * from tab1 join tab2". Since you’re querying myalias.tab1 which is an alias pointing to table lib1.tab1 it should work.
c) Complex: create your own SQL function that returns the corresponding schema_name for a table (e.g. myfunct('TAB1'). This could be done reading system view “qsys2.systables” where table_name=’TAB1’ and returning TABLE_SCHEMA column, which is a varchar(128). Once you got it, build up a dynamically prepared using the variable you just obtained. E.g.
"set mylib = myfunct('TAB1').
"set mystmt = 'select * from '||table_schema || ‘.tab1’ …”
Prepare mystmt and then execute mystmt.
I did something similar in VBA using ado ibmdrda and it worked.
hope this helps.
f.
Not sure how many of you have used UiPath, but I am having an issue when scrapping within UiPath and trying to insert that scrapped variable into my database. It keeps telling me it is not the right column name. Any suggestions?
To pass the contents of the variable returned by ScrapeActivity to the query you must first declare a parameter in the "Parameters" property of ExecuteNonQuery activity.
Let's say your ScrapeActivity returns the scraped text in scrapeActivityResult variable.
Click on "Edit Query" and afterwards the "Parameters" button.
A new dialog will open where you can add parameters.
Add a new In parameter named companyName of type "String" and set its value to scrapeActivityResult variable.
After that, you have to use the newly created parameter(companyName) into your query
"INSERT INTO tblCompany('CompanyName') VALUES (#companyName)"
That was happening to me too! There is a parameter in the query activity that needs to be set/not set
It's really good to see uipath stuff here...
I want to get the results of a stored procedure and place them into a CSV file onto an FTP location.
The catch though is that I cannot create a local/temporary file that I can then FTP over.
The approach I was taking was to use an SSIS package to create a temporary file and then have a FTP Task within the pack to FTP the file over, but our DBA's do not allow temporary files to be created on any servers.
in reply to Yaakov Ellis
I think we will need to convince the DBA's to let me use at least a share on a server that they do not operate, or ask them how they would do it.
in reply to Kev
I like the idea of the CLR integration, but I don't think our DBA's even know what that is lol and they would probably not allow it either. But I will probably be able to do this within a Script Task in an SSIS package that can be scheduled.
This step-by-step example is for others who might stumble upon this question. This example uses Windows Server 2008 R2 server and SSIS 2008 R2. Even though, the example uses SSIS 2008 R2, the logic used is applicable to SSIS 2005 as well. Thanks to #Kev for the FTPWebRequest code.
Create an SSIS package (Steps to create an SSIS package). I have named the package in the format YYYYMMDD_hhmm in the beginning followed by SO stands for Stack Overflow, followed by the SO question id, and finally a description. I am not saying that you should name your package like this. This is for me to easily refer this back later. Note that I also have two Data Sources namely Adventure Works and Practice DB. I will be using Adventure Works data source, which points to AdventureWorks database downloaded from this link. Refer screenshot #1 at the bottom of the answer.
In the AdventureWorks database, create a stored procedure named dbo.GetCurrency using the below given script.
CREATE PROCEDURE [dbo].[GetCurrency]
AS
BEGIN
SET NOCOUNT ON;
SELECT
TOP 10 CurrencyCode
, Name
, ModifiedDate
FROM Sales.Currency
ORDER BY CurrencyCode
END
GO
On the package’s Connection Manager section, right-click and select New Connection From Data Source. On the Select Data Source dialog, select Adventure Works and click OK. You should now see the Adventure Works data source under the Connection Managers section. Refer screenshot #2, #3 and #4.
On the package, create the following variables. Refer screenshot #5.
ColumnDelimiter: This variable is of type String. This will be used to separate the column data when it is written to the file. In this example, we will be using comma (,) and the code is written to handle only displayable characters. For non-displayable characters like tab (\t), you might need to change the code used in this example accordingly.
FileName: This variable is of type String. It will contain the name of the file. In this example, I have named the file as Currencies.csv because I am going to export list of currency names.
FTPPassword: This variable is of type String. This will contain the password to the FTP website. Ideally, the package should be encrypted to hide sensitive information.
FTPRemotePath: This variable is of type String. This will contain the FTP folder path to which the file should be uploaded to. For example if the complete FTP URI is ftp://myFTPSite.com/ssis/samples/uploads, then the RemotePath would be /ssis/samples/uploads.
FTPServerName: This variable is of type String. This will contain the FTP site root URI. For example if the complete FTP URI is ftp://myFTPSite.com/ssis/samples/uploads, then the FTPServerName would contain ftp://myFTPSite.com. You can combine FTPRemotePath with this variable and have a single variable. It is up to your preference.
FTPUserName:This variable is of type String. This will contain the user name that will be used to connect to the FTP website.
ListOfCurrencies: This variable is of type Object. This will contain the result set from the stored procedure and it will be looped through in the Script Task.
ShowHeader: This variable is of type Boolean. This will contain values true/false. True indicates that the first row in the file will contain Column names and False indicates that the first row will not contain Column names.
SQLGetData: This variable is of type String. This will contain the Stored Procedure execution statement. This example uses the value EXEC dbo.GetCurrency
On the package’s Control Flow tab, place an Execute SQL Task and name it as Get Data. Double-click on the Execute SQL Task to bring the Execute SQL Task Editor. On the General section of the Execute SQL Task Editor, set the ResultSet to Full result set, the Connection to Adventure Works, the SQLSourceType to Variable and the SourceVariable to User::SQLGetData. On the Result Set section, click Add button. Set the Result Name to 0, this indicates the index and the Variable to User::ListOfCurrencies. The output of the stored procedure will be saved to this object variable. Click OK. Refer screenshot #6 and #7.
On the package’s Control Flow tab, place a Script Task below the Execute SQL Task and name it as Save to FTP. Double-click on the Script Task to bring the Script Task Editor. On the Script section, click the Edit Script… button. Refer screenshot #8. This will bring up the Visual Studio Tools for Applications (VSTA) editor. Replace the code within the class ScriptMain in the editor with the code given below. Also, make sure that you add the using statements to the namespaces System.Data.OleDb, System.IO, System.Net, System.Text. Refer screenshot #9 that highlights the code changes. Close the VSTA editor and click Ok to close the Script Task Editor. Script code takes the object variable ListOfCurrencies and stores it into a DataTable with the help of OleDbDataAdapter because we are using OleDb connection. The code then loops through each row and if the variable ShowHeader is set to true, the code will include the Column names in the first row written to the file. The result is stored in a stringbuilder variable. After the string builder variable is populated with all the data, the code creates an FTPWebRequest object and connects to the FTP Uri by combining the variables FTPServerName, FTPRemotePath and FileName using the credentials provided in the variables FTPUserName and FTPPassword. Then the full string builder variable contents are written to the file. The method WriteRowData is created to loop through columns and provide the column names or data information based on the parameters passed.
using System;
using System.Data;
using Microsoft.SqlServer.Dts.Runtime;
using System.Windows.Forms;
using System.Data.OleDb;
using System.IO;
using System.Net;
using System.Text;
namespace ST_7033c2fc30234dae8086558a88a897dd.csproj
{
[System.AddIn.AddIn("ScriptMain", Version = "1.0", Publisher = "", Description = "")]
public partial class ScriptMain : Microsoft.SqlServer.Dts.Tasks.ScriptTask.VSTARTScriptObjectModelBase
{
#region VSTA generated code
enum ScriptResults
{
Success = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Success,
Failure = Microsoft.SqlServer.Dts.Runtime.DTSExecResult.Failure
};
#endregion
public void Main()
{
Variables varCollection = null;
Dts.VariableDispenser.LockForRead("User::ColumnDelimiter");
Dts.VariableDispenser.LockForRead("User::FileName");
Dts.VariableDispenser.LockForRead("User::FTPPassword");
Dts.VariableDispenser.LockForRead("User::FTPRemotePath");
Dts.VariableDispenser.LockForRead("User::FTPServerName");
Dts.VariableDispenser.LockForRead("User::FTPUserName");
Dts.VariableDispenser.LockForRead("User::ListOfCurrencies");
Dts.VariableDispenser.LockForRead("User::ShowHeader");
Dts.VariableDispenser.GetVariables(ref varCollection);
OleDbDataAdapter dataAdapter = new OleDbDataAdapter();
DataTable currencies = new DataTable();
dataAdapter.Fill(currencies, varCollection["User::ListOfCurrencies"].Value);
bool showHeader = Convert.ToBoolean(varCollection["User::ShowHeader"].Value);
int rowCounter = 0;
string columnDelimiter = varCollection["User::ColumnDelimiter"].Value.ToString();
StringBuilder sb = new StringBuilder();
foreach (DataRow row in currencies.Rows)
{
rowCounter++;
if (rowCounter == 1 && showHeader)
{
WriteRowData(currencies, row, columnDelimiter, true, ref sb);
}
WriteRowData(currencies, row, columnDelimiter, false, ref sb);
}
string ftpUri = string.Concat(varCollection["User::FTPServerName"].Value,
varCollection["User::FTPRemotePath"].Value,
varCollection["User::FileName"].Value);
FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(ftpUri);
ftp.Method = WebRequestMethods.Ftp.UploadFile;
string ftpUserName = varCollection["User::FTPUserName"].Value.ToString();
string ftpPassword = varCollection["User::FTPPassword"].Value.ToString();
ftp.Credentials = new System.Net.NetworkCredential(ftpUserName, ftpPassword);
using (StreamWriter sw = new StreamWriter(ftp.GetRequestStream()))
{
sw.WriteLine(sb.ToString());
sw.Flush();
}
Dts.TaskResult = (int)ScriptResults.Success;
}
public void WriteRowData(DataTable currencies, DataRow row, string columnDelimiter, bool isHeader, ref StringBuilder sb)
{
int counter = 0;
foreach (DataColumn column in currencies.Columns)
{
counter++;
if (isHeader)
{
sb.Append(column.ColumnName);
}
else
{
sb.Append(row[column].ToString());
}
if (counter != currencies.Columns.Count)
{
sb.Append(columnDelimiter);
}
}
sb.Append(System.Environment.NewLine);
}
}
}
Once the tasks have been configured, the package’s Control Flow should look like as shown in screenshot #10.
Screenshot #11 shows the output of the stored procedure execution statement EXEC dbo.GetCurrency.
Execute the package. Screenshot #12 shows successful execution of the package.
Using the FireFTP add-on available in FireFox browser, I logged into the FTP website and verified that the file has been successfully uploaded to the FTP website. Refer screenshot #13.
Examining the contents by opening the file in Notepad++ shows that it matches with the stored procedure output. Refer screenshot #14.
Thus, the example demonstrated how to write results from database to an FTP website without having to use temporary/local files.
Hope that helps someone.
Screenshots:
#1: Solution_Explorer
#2: New_Connection_From_Data_Source
#3: Select_Data_Source
#4: Connection_Managers
#5: Variables
#6: Execute_SQL_Task_Editor_General
#7: Execute_SQL_Task_Editor_Result_Set
#8: Script_Task_Editor
#9: Script_Task_VSTA_Code
#10: Control_Flow_Tab
#11: Query_Results
#12: Package_Execution_Successful
#13: File_In_FTP
#14: File_Contents
If you were allowed to implement CLR integration assemblies you could actually use FTP without having to write a temporary file:
public static void DoQueryAndUploadFile(string uri, string username, string password, string filename)
{
FtpWebRequest ftp = (FtpWebRequest)FtpWebRequest.Create(uri + "/" + filename);
ftp.Method = WebRequestMethods.Ftp.UploadFile;
ftp.Credentials = new System.Net.NetworkCredential(username, password);
using(StreamWriter sw = new StreamWriter(ftp.GetRequestStream()))
{
// Do the query here then write to the ftp stream by iterating DataReader or other resultset, following code is just to demo concept:
for (int i = 0; i < 100; i++)
{
sw.WriteLine("{0},row-{1},data-{2}", i, i, i);
}
sw.Flush();
}
}
Is there a server anywhere that you can use where you can create a temporary file? If so, make a web service that returns an array containing the contents of the file. Call the web service from the computer where you can create a temporary file, use the contents of the array to build the temp file and ftp it over.
If there is no where at all where you can create a temporary file, I don't see how you will be able to send anything by FTP.
Try using a CLR stored procedure. You might be able to come up with something, but without first creating a temporary file, it might still be difficult. Could you set up a share on another machine and write to that, and then ftp from there?
Script from the FTP server, and just call the stored proc.
The catch though is that I cannot create
a local/temporary file that I can then FTP over.
This restriction does not make any sense, try to talk to DBA nicely and explain it to him/her. It is totally reasonable for any Windows process or job to create temporary file(s) in appropriate location, i.e. %TEMP% folder. Actually, SSIS runtime itself often creates temporary files there - so if DBA allows you to run SSIS, he is allowing you to create temporary files :).
As long as DBA understands that these temporary files do not create problem or additional workload for him (explain that he does not have to set special permissions, or back them up, etc), he should agree to let you create them.
The only maintenance task for DBA is to periodically clean %TEMP% directory in case your SSIS job fails and leaves the file behind. But he should do this anyway, as many other processes may do the same. A simple SQL Agent job will do this.