I have about 500 fixed width columns in a flat file that I want to apply the same logic to to replace an empty column with null before it goes into the database.
I know the command to replace the empty string with null but I really don't want to have to use the gui to input that command for every column.
So is there a tool out there that can do this all on the back end?
You could look at something like the EzAPI to create your data flow. This this answer, I have an example of how one creates a EzDerivedColumn and sets the formula within it.
Automatically mapping columns with EZApi with OLEDBSource
If you can install third party components, I've seen a number of implementations of a Trim-To-Null functionality on codeplex.com
BIML might be an option to generate your package as well. I'd need to play with that to figure the syntax though.
My googlefu worked a little better after lunch.
I as able to modify about the 5th comment down on http://social.msdn.microsoft.com/Forums/sqlserver/en-US/222e70f5-0a21-4bb8-a3fc-3f365d9c701f/ssis-custom-component-derivedcolumn-programmatically-problems?forum=sqlintegrationservices to work for my needs.
My c# code will now loop through all the input columns from a "Flat File Source" object and add a derived column for each.
Related
I'm rewriting an older-than-dirt MFC application, ripping out the old MFC-based DB code and re-working queries to make it run faster. This code works with a MS Access 2003 .mdb file.
The old code used the convenience functions like RFX_Bool, RFX_Long, RFX_Int to read from the records. These are nice, and I am reading about binding in ODBC using SQLBindCol to set the bindings ahead of time to avoid that extra processing time for each row. This is great, but I see SQLBindCol only takes the column number, not the name. What if I want to bind using the column name like with the RFX_* functions? SQLDescribeCol gives the column names, but it doesn't have the "full" name, i.e. [Table/Alias].[Column]. Some of my queries involve JOIN'ing the same table multiple times with aliases, so I can't bind the column by the column name alone. If I plug my query into Access, the Datasheet view shows the alias in the column name. I'm currently using my connection string with Driver={Microsoft Access Driver (*.mdb, *.accdb)}, if it matters.
tl;dr How do I do MFC's RFX_*(fieldExchange, L"[Table].[Column]", &variable) in the modern ODBC API?
OK, I think I understand what the RFX functions are doing now, and I think I know what I need to do.
The MFC ODBC classes construct your query programmatically, so after starting with SELECT, UPDATE, etc, every call to RFX_* simply appends the field name to the query, then ties a reference to your variable to the column index, which it increments after every call. So I just need to append my fields to my queries with a helper function the same way MFC does, in order to bind my pointers in the same way MFC does.
Hopefully this is helpful to somebody.
Right now, I run a stored procedure whose output feeds a "Create CSV Table" Data Operations component. This component, not surprisingly, outputs a comma-delimited list of fields, which is not supported by our remote system. The fields need to be tab-delimited. One would think that the Data Operations component would have a tab (or other character-delimited option). But no, only commas are available, and no other Data Operations component outputs a tab-delimited table.
Using any mechanism for which we'd have to write code is completely the last option, as there's no need for code to use CSV. Also, any mechanism which requires paying for 3rd party components is categorically out, as is using any solution which is in preview mode.
The only option we've thought of is to revamp the stored procedure which outputs a single "column" containing the tab-delimited columns, and then output to a file - ostensibly, a comma-delimited file, but one without commas embedded inside (which is allowed for my system) so that the single column isn't itself enquoted.
Otherwise, I guess Function Apps is the solution. Anyone with ideas?
The easiest way is to use string function and replace comma with other delimiter. If you could accept this way, after creating the csv table I initiate a string variable with this input replace(body('Create_CSV_table_2'),',',' ').
And this is the result.
And if you don't want this way, yes you have to solve it with code and the Function is a choice.
I tried to find documentation in the subject but fell short until now.
I am trying to use Logic Apps in order to update a table when a trigger occurs.
Adding some context:
In many separate excel online file that are located in different area of Sharepoint, I have one Table in each of those files. Anytime the SQL table is updated, I get the following elements:
Name
Age
path_to_doc
doc_id
Name and Age are element I wish to add in those Excel file.
path_to_doc is the path to the Excel file that needs to be updated.
doc_id is the id of the Excel file that needs to be updated.
In the "Add row to a table" action, those are the elements that need to be filled:
Site (Manual no problem, this doesn't change) Document Library
(Manual no problem, this doesn't change)
File (this is where I have a first problem: when I do not click
manually, and try to put either the "path_to_doc" or the "doc_id"
instead, it doesn't work.
Table (It seems that I can force it to be Table1), which is fine
because all my Excel files have the table called Table1
Arguments (that is Azure understands the Table and is componnents and
asks you to fill the ones you need to fill, those elements disappear
when you change from a manual input to an input "path_to_doc" or
"doc_id").
It throws me an error:
ERROR 400
NOTE: When I do it manually, it works.
Anyone has experienced this and found a solution?
Thank you
You don't need to use Expression.
For example, if we want to get tables of the modified Excel, we can do like this:
A similar flow in SharePoint:
Finally found the answer.
I needed to go to the code view and add my dynamic details there for the body.
Thank you for your help.
Here is the solution. I hope it helps others :)
In the designer view, create an action "Add a row into a table" and use the dynamic path that brings you to the excel file that you need to update. It will show an error and you will not be able to add the body arguments.
In the code view, now you can manually add the body of the request to include the element you wish to update in the Table of the excel file.
That's it!
I'm trying to manipulate a column in SSIS which looks like below after i removed unwanted rows with derived column and conditional split in my data flow task. The source for this is a flatfile.
XXX008001161022061116030S1TVCO3057
XXX008002161022061146015S1PUAG1523
XXX009001161022063116030S1DVLD3002
XXX009002161022063146030S1TVCO3057
XXX009003161022063216015S1PUAG1523
XXX010001161022065059030S1MVMA3020
XXX010002161022065129030S1TVCO3057
XXX01000316102206515901551PPE01504
The first three numbers from the left (starting with "008" first row) represent a series, and the next three ("001") represent another number within the series. what i need is to change all of the first three numbers starting from "001" to the end.
The desired reslut would thus look like:
XXX001001161022061116030S1TVCO3057
XXX001002161022061146015S1PUAG1523
XXX002001161022063116030S1DVLD3002
XXX002002161022063146030S1TVCO3057
XXX002003161022063216015S1PUAG1523
XXX003001161022065059030S1MVMA3020
XXX003002161022065129030S1TVCO3057
XXX00300316102206515901551PPE01504
...
My potential solution would be to load the file to a temporary database table and query it with SQL from there, but i am trying to avoid this.
The final destination is a flatfile.
Does anybody have any ideas how to pull this off in SSIS? Other solutions are appreciated also.
Thanks in advance
I would definitely use the staging table approach and use windows functions to accomplish this. I could see a use case if SSIS was on another machine than the database engine and there was a need to offload the processing to the SSIS box.
In that case I would create a script transformation. You can process each row and make the necessary changes before passing the row to the output. You can use C# or VB.
There are many examples out there. Here is MSDN article - https://msdn.microsoft.com/en-us/library/ms136114.aspx
Got to make my first steps with typo3, now.
Got an Extension, some tables in ...typo3conf\ext\my_extension\ext_tables.sql and would like to put each table Definition in a separate file, because it gets very long.
Is it possible?
The best would still be to put everything into the ext_tables.sql file as many checks are happening with this file like if you add new fields, remove fields, add tables, the DB compare in the Install Tool can handle that.
Have a look at an example in CMS7
/typo3/sysext/install/Classes/Controller/Action/Tool/UpgradeWizard.php::silentCacheFrameworkTableSchemaMigration()
Where given SQL file is used to perform update. fx:
/typo3/sysext/core/Resources/Private/Sql/Cache/Backend/Typo3DatabaseBackendCache.sql