Im currently writing to a SQLite db file using Serilog.Sinks.SqLite. This works fine however i would like to create custom columns in this db file. I have managed to Enrich the properties column with various enrichers (ProcessName, EnviromentName etc) but i need to conform to a specific column format.
I cant find any worthwhile documentation on this package. How would one go about creating these custom columns?
Alternative solution:
How can i configure the (much beter documented) MSSQL sink to write to an SQLite file?
Thanks in advance!
Related
The firm I work in has a lot of data sources entering the firm database using the Informatica ETL tool, stored in maplets and other data models (sorry If I'm not using the exact terminology).
The problem is that all the business logic is stored in the 'graphical interface' and nowhere else - Every time I want to see what field goes into the target field I have to trace the inputs through the maplet and that takes a very long time.
The Question is: Is there a tool that can takes all the relationships in the Informatica maplet and somehow export them to a excel table (so I can see it all without tracing)? that way I could try to make proper documentation....
Thanks in Advance.
It's possible to export mappings or whole workflows to XML. Next, you can use this tool - it will create tables with source to target dependency for every mapping.
Keep in mind it will only map input to output, it won't extract the full logic and transformations done along the way - that would've been to complex for simple visualization.
Informatica supports exporting mapping information to Excel - just search the documentation which tells you how to do it.
However, for anything other than the simplest of mappings, what ends up in Excel is not that easy to understand. If your Informatica installation supports it, then using the lineage capabilities is a much better bet.
I have to import about 50 different types of files every day. Some of them with a few columns, some inculde up to 250 columns.
The Flat File connection always defaults all columns to 50 chars.
Some columns can be way longer than 50 chars, and will of course end up in errors.
Currently i am doing a stupid search&replace with notepad++ - Opening all SISS packages, replacing:
DTS:MaximumWidth="50"
by
DTS:MaximumWidth="500"
This is an annoying workaround.
Is there any possibility to set a default length for flatfile string columns to a certain value?
I am developing in Microsoft Visual Studio Professional 2015 and SQL Server Data Tools 14.0.61021.0
Thanks!
I don't think that there is a way to achieve this from SQL Server Data Tools.
But you can do some workaround to achieve this:
Easiest solution, In the Flat file connection manager - Advanced Tab, select all columns (using Ctrl key) and change the data length property for them all in one edit. (detailed in #MikeHoney answer)
You can use BIML (Business Intelligence Markup Language) to create ssis package, if you're new to BIML you can access to BIML Script website for detailed tutorials.
You can create a Small application that loop over .dtsx files in a folder and replace DTS:MaximumWidth="50" with DTS:MaximumWidth="500" using normal String.Replace function or using Regular expressions. (you can read my answer # Automate Version number Retrieval from .Dtsx files to see an exmaple on reading .dtsx file using Regular expressions)
Function To Read and Replace content of dtsx file (Vb.Net)
Public Sub FixDTSX(byval strFile as string)
dim strContent as string = string.empty
Using sr as new Io.StreamReader(strFile)
strContent = sr.ReadToEnd()
sr.Close()
End Using
strContent = strContent.Replace("DTS:MaximumWidth=""50""","DTS:MaximumWidth=""500""")
Using sw as new Io.StreamWriter(strFile,False)
sw.Write(strContent)
sw.Close()
End Using
End Sub
There is a way to achieve what you want using the standard Visual Studio SSDT UI, although it is quite obscure. AFAIK it works in every version of this editor since SQL Server 2005.
With the package open, from the Connection Managers pane, right-click your Flat File Connection and choose Edit. Then navigate to the Advanced page. Then multi-select the columns you want to change (e.g. shift-click a range or ctrl-click a specific set). Now the Properties appearing at the right will be applied to all the selected columns.
In the example shown below, I have set all the selected columns to a width of 255.
Esteban,
I suggest you use the Object Model API which allows you to develop SSIS packages programmatically. Using that, you can make use of any .net code that allows you to gather data/metadata from text files. Also, the assumption is that, since you are using SSIS, you already might be familiar with writing code in C#/VB.Net
Now, if you are just starting with the Object Model API, there would be a huge learning curve (but it is worth learning it if SSIS is your day to day life). If you do not have the time to invest right now, I would recommend you to use a library I wrote (called Pegasus) which greatly simplifies how you can use the Object Model API; you can create your packages in an almost declarative fashion (using C#).
On Github, there is an example that shows how to create a package that loads any number of text files with differing schemas in a given folder. See here; specifically the method GenerateProjectToLoadTextFilesToSqlServerDatabase().
In the example, I use a third party .Net library called lumenworks.framework to probe delimited files and get their metadata. Using this library, I get the names of the columns; and I also infer data types and lengths based on sampling the first 'n' number of rows. (In my code, I am only inferring ints, dates and strings; if you have more data types, add relevant code accordingly). Or you can specify one specific data type and length (looks like you want to use string of 500 chars) for all your columns. [Or (in some cases), you might have this metadata available outside in a excel file/config file.] Then I use this metadata to configure my text file connection managers programmatically.
YOu can download the code from Github and run the DataFlowExample by specifying where your source files are and see how far it gets you.
Another recommendation would be Biml, but I am not sure if you can incorporate your own/third party full fledged C# code (not just snippets) into Biml workflow. If you can, then go with Biml.
Let me know if you have any questions.
Is their any way to create the excel sheet with the data loaded from grid in extjs.
Going through this link,I have not clearly understood the appraoch and how to achieve my goal.Other related questions were e.g here,but could not get anything out of it.Please help me resolve this.Any help is appreciated.Thanks.
The general concept is that an Excel file is actually XML, at least in the more recent versions of Excel. You can create plain XML that just contains the data. Or you can add a lot of formatting, use the rich presentation that Excel provides. I would suggest that you start off as simply as possible, with just the data, and then add complexity.
To learn how to do this, you need to understand how to generate any XML, then the particular XML that Excel requires, and then how to return that mime type to the browser.
Here is one place (of many) that you can find the XML for Excel.
The better option is for export to be handled from the server side rather than from the client-side script. A new request for export can be sent to the server with the currently applied filters and get the file in generated in excel using Apache FOP or similar libraries(http://xmlgraphics.apache.org/fop/).
Export from extjs raises some problems like
1) what data would you expect for a paged grid.
2) what about the filters applied on the grid
Is there a way to automatically generate SSIS packages? I need to create a lot of SSIS packages that just erase data from one table and import data from a text file. The file name matches table name and the column headers are in the first line of the file.
For more detailed information:
I am working on a project in which I have to separate two systems that are currently coupled (one system has direct access to the other's database). After the modifications, one system will provide data through txt files to be loaded in the other database.
We have to use SSIS to load data into the database from the text files.
The text files will be provided in CSV format with column headers in the first line.
The tables from both databases have matching column names, and all we need to do is clear the table and load data from the files.
I have more than one hundred tables with different number of columns. Do I need to create each package manually?
I'm familiar with 2 free options.
EzAPI might be a good place if you're a .NET heavy shop or just really want to geek out with the API. This approach allows you to control the pretty much the entire package generation but at the cost of coding time. I find EzAPI generally easier than working with the base COM/.NET libraries for SSIS.
Biml is an interesting beast. Varigence will be happy to sell you a license to Mist but it's not needed. All you would need is BIDSHelper and then browse through BimlScript and look for a recipe that approximates your needs. Once you have that, click the context sensitive menu button in BIDSHelper and whoosh, it generates packages.
I did this just using vb, I passed in the table names as a command parameter and used vb to generate the insert and clear, worked a charm... I can try and dig it out tomorrow when I'm back in the office but it was pretty simple. There didn't seem to be any other way to say "just get x and export it", "just take y and import it into z" so vb it had to be. In fact come to think of it I think I actually used a small xml file to pass the table info for export and then determined the table name for import from the csv file name. To be clear, this was only one package but it could dynamically choose the number of imports/exports it did. Further clarification this was vb within ssis as a processing step
I'm using a static database that I created with SQLite Database Browser. I put it in my assets folder and built a code to copy the database to a database variable (Does that make sense?) so I could read information from it. Problem is I don't know how - mostly the SQL queries involved - and what are your suggested methods do to that? In other words, what methods should I add to my Database Handler class (Or data adapter?) in order to present the data in a list view, for example.
Thank you for all your help.
Read the Android database documentation.
Copying your database from the assets folder is typically done in the onCreate and/or onUpgrade functions of your SQLiteOpenHelper-derived class.
This tutorial covers the basics:
Using the SQLite Database with ListView
As for naming things: use whatever names make sense in your application.