We are using Netezza in order to store tables. Every day, an updated version of the table will come in as a pipe delimited text file. So I know how to import the text file into Netezza but is there a way I can merge the pipe delimited file with the existing table in Netezza? The schema will always be the same and I would like to do it without having to load the pipe delimited file, doing the merge and than deleting the file that I just loaded because it is a big table so it'll take time.
Thanks in advance for the help!
1) create a external temp table ‘same as Target’ pointing to the file of the Day
2) do an Insert into Target select * from ExternalTempTable
Not much more to it if I understand you correctly
Related
I was working on creating the staging area in Snowflake and creating CSV files inside it. I am stuck up with an issue, hope someone here is experienced enough to help me in the case.
I have created a job in unix to create a CSV file in a staging area from a table added with where conditions to filter the data. But, when at times if there are no rows as output as the result of the select statement from the table, the CSV file is not at all created in the staging area. Is there any way in such cases a CSV file is created with the name of the columns alone with no value rows?
Thanks in advance.
Rahul
Have you tried the header=true option in your COPY INTO?
I have a folder of txt files. The filenames are of the form [integer].txt (like 1.txt, 2.txt and so on).
I have a table, let's say TableA (id int not null, contents varchar(max))
I want a way to mass import the contents of those files into TableA, populating the id column from the filename. Each file will be a single record in the table. It's not a delimited file.
I've looked into SSIS and flat-file source, but I could not find a way to select a folder instead of a single file (this answer claims it can be done, but I could not find out how).
Bulk Insert is my next bet, but I'm not sure how I can populate the id column with the filename.
Any ideas?
For anyone that might need it, I ended up solving this by:
Using a ForEach loop container (Thanks for the hint #Panagiotis
Kanavos)
Using a flat-file source, setting as row delimiter and column
delimiters a sequence I know didn't exist in the file (for example '$$$')
Assigning the filename to a variable, and the full path to a computed
variable (check this great post on how to assign the variables)
Using a derived column to pass the filename in the output (check out
this answer)
I have a task where I need to generate a CSV file from data coming out of two views including a Header having hard coded values and a Trailer at the bottom of the CSV file having These fields- Record_Type = E99, Row_count, and Blank field with 190 length.
I'm able to get the desired output file but I am not able to figure out how to retrieve the NO. of rows coming out of the two vies and write it in between the record type and the blank field at the bottom of the CSV as the whole line is trailer with | delimited.
Please help me figure this out.
Thanks.
My suggestion:
I assume you are using the SSIS Package to solve this problem.
Create a SQL Staging table to store the content which you want to export in CSV file. You may use stored procedure to truncate and refill this staging table by executing it. Execute this store procedure through Execute SQL Task in SSIS Package
Use Data Flow Task to export the data from staging table to CSV file. Input will be SQL Staging table and output will be flat file with Comma(,) delimiter.
I hope it will help you
I need to create table in Netezza from csv file with header. Target table should have the columns as the headers in source file. Target table should have flexible structure changing based on source file headers. Is it possible?
I don't know what your expectations are for this? You will need datatypes as well as column names, and short of using NVARCHAR(100) for everything and crossing your fingers, I don't know how to address that from the csv file alone...
If you can get your source system to provide another csv file with metadata for COLNAME,DATATYPE(Precision) you can certainly change that to a valid 'create table' statement in netezza, and then something like this to get you the rest of the way:
CREATE EXTERNAL TABLE demo_ext SAMEAS emp USING (dataobject
('/tmp/demo.out') DELIMITER '|');
More info here:
https://www.ibm.com/support/knowledgecenter/en/SSULQD_7.2.1/com.ibm.nz.load.doc/c_load_create_external_tbl_expls.html
From the question, im assuming the requirement is you need to create a new table/alter every time based on ur source csv file. Is it?
#Moutusi Das
As mentioned by Lars G Olsen, the option will be having a separate file for table creation which contains column names & datatypes. You can use unix script to get the details of table creation file & create/altee table in netezza.
Then load the table using external table command.
I have to work in a DTS environment in 2005 (too complicated to explain) and I have a comma delimited text file that is to be appended to the main table. I'd like to pull the last column in the text file out for the first record and use it as select criteria for a delete command. But how can I do this in the older DTS environment?
Here's the line of foobar data
9,36,7890432174,2007-12-17 00:00:00.000,21,15.22,99,11,49,28,2009-07-12 00:00:00
what I want to do is create a sql statement that will delete all the records where a certain column is equal to "2009-07-12 00:00:00"
Thanks.
There are at least two ways to implement this in DTS.
The first is to
load the text file into a staging table
select the date value from the temporary table and assign it to a package variable
carry out the delete using the package variable as an input parameter
insert from the staging table into the main table
clear down the staging table
This assumes that there is some way to identify the order of the rows in the text file from the data. If not, you could add an identity column to the staging table definition.
The second is to
extract the value from the input file using a script task and assign it to a package variable
carry out the delete using the package variable as an input parameter
insert from the text file into the main table
EDIT
I believe it's also possible to use the generic text file ODBC driver to access the text file like a database table.