I am working in ODI12c Project. I have a Scenario in which i need to load multiple files in a single table Parallelly.
I tried through file List and using Loop in ODI package,but its loading data in serial i.e. 1st file then 2nd file etc.
Please suggest how should i load data Parallelly.
If you want to load multiple files in odi in one go you can merge all those files in one single file and load it in a single go. I'm considering all files have same structure. if you want to load them separately then creating separate model for each file will be headache and re usability of code will also compromised. You can run scenarios in asynchronous mode to made them run independent . I believe this is my best answer till now.
I have an another approach to achieve this:
Create a simple package for loading the file.
Create a another package which is calling the another one.
Load all the files name in a oracle table like:
File_id Name
1 File1
2 File2
Now create a increment variable with pull the name of 1st file and then 2 .
Now call the file loading package inside another package & make it asynchronous.
Pass the increment variable which is having the file name into model created for loading file ( resource Name).
Each time when variable is getting incremented it's gonna pull the next file and create a different session for each file loading . there might be a gap of 1 sec for each file loading because of increment of variables,i think that would be ok.
I hope you got the way in case any help you can add comment.
Is your target table partitioned?
The bottleneck you might be having could be due to the target destination being the same physical storage.
Partitioning the target (or similar techniques) might help
Related
I'm trying to create simple project in which I'd like to download XML files from given website. I have stored files names in DataBase table. What I have done looking at this tutorial: Implementing Foreach Looping Logic in SSIS is:
a. Read all distinct rows from my Table (let's call it XMLTable)
b. Assign result of this query to User variable called: nameOfFileToDownload
c. Created For Each Loop container
d. Configured to assign localy each row with file name to download to: nameFileForeachLoop variable
e. Download files from concate link as a path using HTTPManager with dynamic file name from nameFileForeachLoop variable.
f. Created XMLFlatFile connection for dummy file - I assumed after reading from above tutorial.
The problem is now that this loop container works but doesn't download files separately - still to one file which at the end is empty. My nameFileForeachLoop variable is not updated during each LOOP iteration. What's more I have noticed that during FLAT FILE creation I have only CSV and TXT extension available. I have tried many approaches but without results. Can you help me how to download XML files?
For example I have following link to XML: nbp.pl/kursy/xml/c001z180102.xml What changes here is last part of this link with XML extension which I get from my XMLTable.
I have configured my components as follows:
You are on the right track, but need some amendements.
Do not create and configure Flat File Destination connection manager unless you are creating tables in .CSV or .TXT files. In provided example author selects data with dynamic queries and stores the results in dynamic txt files. As I understand, this is not your case.
Here are some examples how to download and save files with HTTP in SSIS. Sample download script and Review of different approaches to HTTP download.
Got to make my first steps with typo3, now.
Got an Extension, some tables in ...typo3conf\ext\my_extension\ext_tables.sql and would like to put each table Definition in a separate file, because it gets very long.
Is it possible?
The best would still be to put everything into the ext_tables.sql file as many checks are happening with this file like if you add new fields, remove fields, add tables, the DB compare in the Install Tool can handle that.
Have a look at an example in CMS7
/typo3/sysext/install/Classes/Controller/Action/Tool/UpgradeWizard.php::silentCacheFrameworkTableSchemaMigration()
Where given SQL file is used to perform update. fx:
/typo3/sysext/core/Resources/Private/Sql/Cache/Backend/Typo3DatabaseBackendCache.sql
I've chased my tail for a full 12 hours. Haven't found the right solution.
I'm locked into using SSIS. I have a SQL Server table with full paths and filenames already concatenated. Examples:
\\MydevServer1\C$\ABC\App_Data\Sample.pdf
\\MydevServer2\E$\Garth\App_Data\Morefiles.txt
\\MydevServer3\D$\Paths\App_Data\MySS.xlsx
etc.
I need to read each row of the table, get the path and filename and move that file to a new static destination directory.
The rows in the table will remain unchanged. I only use it as a source to locate the file to be moved.
I've tried:
1) Feeding a resultset from an ole db source to a recordset destination then to an Object variable that connects via variable to a foreach loop container holding a files system task. (Very problematic.)
2) Sending the table rows to a .csv file and reading each line of the csv file using a foreach loop container holding a file system task.
3) Reading directly from the table rows using a foreach loop container holding a file system task. (preferred).
and many other scenarios.
I have viewed a hundred examples online, but most of them involve loading a table, or sending results to flat files, or moving files from one folder to another based on extension type, etc. I haven't found anything on configuring a file system task to read a table supplied path and move the file based on the table value as the source.
I'm rambling. :-)
Any insight or help will be appreciated. I'm not new to SSIS, but I sure feel like it right now.
Create two string variables to store source and destination paths
Use an Execute SQL Task to populate a Full Recordset (Variable with Object data type)
Use For Loop container to go through each row of recordset and set those two variables.
Inside For Loop container, use File System Task. You need to specify IsSourcePathVariable = True, IsDestinationPathVariable = True, path variables - DestinationVariable / SourceVariable, and set operation (copy, move, etc.)
It appears I've been tail chasing due to the error, "Source is empty error".
This was caused by a blank first row in my recordset. I was searching for a fix to the Object variable is empty issue, when in reality the issue was that the Object variable couldn't find data right off the bat.
Insert shameful smug here.
Thanks to Anton for the help.
I have several CSV files and have their corresponding tables (which will have same columns as that of CSVs with appropriate datatype) in the database with the same name as the CSV. So, every CSV will have a table in the database.
I somehow need to map those all dynamically. Once I run the mapping, the data from all the csv files should be transferred to the corresponding tables.I don't want to have different mappings for every CSV.
Is this possible through informatica?
Appreciate your help.
PowerCenter does not provide such feature out-of-the-box. Unless the structures of the source files and target tables are the same, you need to define separate source/target definitions and create mappings that use them.
However, you can use Stage Mapping Generator to generate a mapping for each file automatically.
PMy understanding is you have mant CSV files with different column layouts and you need to load them into appropriate tables in the Database.
Approach 1 : If you use any RDBMS you should have have some kind of import option. Explore that route to create tables based on csv files. This is a manual task.
Approach 2: Open the csv file and write formuale using the header to generate a create tbale statement. Execute the formula result in your DB. So, you will have many tables created. Now, use informatica to read the CSV and import all the tables and load into tables.
Approach 3 : using Informatica. You need to do lot of coding to create a dynamic mapping on the fly.
Proposed Solution :
mapping 1 :
1. Read the CSV file pass the header information to a java transformation
2. The java transformation should normalize and split the header column into rows. you can write them to a text file
3. Now you have all the columns in a text file. Read this text file and use SQL transformation to create the tables on the database
Mapping 2
Now, the table is available you need to read the CSV file excluding the header and load the data into the above table via SQL transformation ( insert statement) created by mapping 1
you can follow this approach for all the CSV files. I haven't tried this solution at my end but, i am sure that the above approach would work.
If you're not using any transformations, its wise to use Import option of the database. (e.g bteq script in Teradata). But if you are doing transformations, then you have to create as many Sources and targets as the number of files you have.
On the other hand you can achieve this in one mapping.
1. Create a separate flow for every file(i.e. Source-Transformation-Target) in the single mapping.
2. Use target load plan for choosing which file gets loaded first.
3. Configure the file names and corresponding database table names in the session for that mapping.
If all the mappings (if you have to create them separately) are same, use Indirect file Method. In the session properties under mappings tab, source option.., you will get this option. Default option will be Direct change it to Indirect.
I dont hav the tool now to explore more and clearly guide you. But explore this Indirect File Load type in Informatica. I am sure that this will solve the requirement.
I have written a workflow in Informatica that does it, but some of the complex steps are handled inside the database. The workflow watches a folder for new files. Once it sees all the files that constitute a feed, it starts to process the feed. It takes a backup in a time stamped folder and then copies all the data from the files in the feed into an Oracle table. An Oracle procedure gets to work and then transfers the data from the Oracle table into their corresponding destination staging tables and finally the Data Warehouse. So if I have to add a new file or a feed, I have to make changes in configuration tables only. No changes are required either to the Informatica Objects or the db objects. So the short answer is yes this is possible but it is not an out of the box feature.
I am loading 30 files at a time using a MULTIFLATFILE connection from SSIS into a raw table. These files are constantly coming in and a job loops through to exec the package and pick these up. Some of the files can have bad data every once in a while. I am having trouble when one of the 30 files is bad. I have no way to know which of the files actually failed so I can move it into a Suspect folder, roll the other 29 files back into my Input folder and try again. Can't find any info on the net on this, or in Microsoft's documentation. Any help appreciated. Thanks.
I would rebuild your Connection using the FLATFILE type, then use it within a Foreach Loop Container that sets a Variable to the file name. In the Connection definition, you can use an Expression to set the ConnectionString property to your Variable value.
Then you can use that Variable value to take whatever action you need when your "bad" condition occurs.
Configure the FileNameColumnName property of the Flat File Source component. That will add a column containing the file name.