Is it possible that we import csv files of two different structures into a table according to the conditions?
I think it is ok when you adjust the status in import command field. However, if the structures of two tables are not same, especially in field segmentation, is it appropriate when you put them in the same table?
Related
I want export some data from opentsdb,then import it into DolphinDB.
In opentsdb, the metrics are device_id,ssid, the tags are battery_level,battery_status,battery_temperature,bssid,cpu_avg_1min,cpu_avg_5min,cpu_avg_15min,mem_free,mem_used and rssi.
In DolphinDB , I create a table as bellow,
COLS_READINGS = `time`device_id`battery_level`battery_status`battery_temperature`bssid`cpu_avg_1min`cpu_avg_5min`cpu_avg_15min`mem_free`mem_used`rssi`ssid
TYPES_READINGS = `DATETIME`SYMBOL`INT`SYMBOL`DOUBLE`SYMBOL`DOUBLE`DOUBLE`DOUBLE`LONG`LONG`SHORT`SYMBOL
schema_readings = table(COLS_READINGS, TYPES_READINGS)
I find that the csv text file can import into DolphinDB, but I don't know how to export data to csv text file in Opentsdb. Is there a easy way to finish this work?
Assuming you're using an HBase backend, the easiest way would be to access that directly. The OpenTSDB schema describes in detail how to get the data you need.
The data is stored in one big table, but to save space, all metric names, tag keys and tag values are referenced using UIDs. These UIDs can be looked up in the UID table which stores that mapping in both directions.
You can write a small exporter in a language of your choice. The OpenTSDB code comes with an HBase client library, asynchbase, and has some tools to parse the raw data in its Internal class which can make it a bit easier.
The process is one where I would get 28 fixed width files and combine it into one table. In the past, this was done via FoxPro. As I have learned today, there were duplicates for which FoxPro did not reject or have any issues with. I have discovered that I need to write a merge statement in order to import the 28 and not get tripped up by duplicate primary key errors when I try to import each one separately using the Import Wizard.
I use Management Studio with a SQL Server Express front end and therefore can't create SSIS packages.
I am going to break this up into two questions so as to not make this too convoluted. First, I have since converted the fixed width files into tab-delimited text files by using Excel.
First question: Can one construct a merge statement that brings the files (tab-delimited) into SQL Server from the C drive? I could import each using the import wizard but that is cumbersome. I know how to write a merge statement but it demands that the data already exist in SQL Server. Below is an example. The question is how would I bring it in from outside.
Merge Industry as TARGET
Using Table1 as SOURCE
On (TARGET.Primary keys 1-9 = SOURCE.Primary keys 1-9)
No, you can't import data during or as part of a MERGE statement. The MERGE operation is purely for the 'upsert' situation; constructing logic on combining two result sets with criteria for matches and mismatches.
To get data into SQL Server you can either work via the UI (which is pretty boring and error prone when you have 28 files), or you can use some of the built in commands such as BULK INSERT.
Perhaps you could BULK INSERT the files one by one, and merge after each import.
If you wanted to continue using Foxpro but eliminate the duplicate records the first piece of advice would be to quit using the Import Wizard.
Wizards may be convenient to use, but they come with their own set of 'baggage' which can be problematic.
Aside from saying that they are in fixed field length format, you don't indicate which format(s) the 28 import files are in (CSV, SDF, TXT, ect.). Regardless you can farily easily write Foxpro code to handle all of the importing without the use of a 'Wizard'.
Then once all of the records have been imported you can readily eliminate the duplicates with something like the following:
SELECT ImportDBF && Assuming it is used EXCLUSIVELY
DELETE ALL
INDEX ON <primary key> UNIQUE TAG Uniq && Create an Index on only UNIQUE instances of your Primary key field
RECALL ALL && Recall only those UNIQUE records
DELETE TAG Uniq && Eliminate the temporary Index
PACK && PACK out the duplicate records
Now your Foxpro data table should be ready to go.
Good Luck
Actually i am importing EXcel to Data Base using SSIS
But its changing the Row order so is there any way to restrict without changing the row order!
Please don't give hit (-) if its duplicated question in Stack overflow
Thanks (+1)
The order of rows in the database do not matter. Relational tables are un-ordered sets rather than ordered lists (like Excel). Even if you import the data a certain way into the database, the order of the data is random and can change. If you want the data to appear a certain way, simply use a SELECT statement with an ORDER BY of your choice.
I have several CSV files and have their corresponding tables (which will have same columns as that of CSVs with appropriate datatype) in the database with the same name as the CSV. So, every CSV will have a table in the database.
I somehow need to map those all dynamically. Once I run the mapping, the data from all the csv files should be transferred to the corresponding tables.I don't want to have different mappings for every CSV.
Is this possible through informatica?
Appreciate your help.
PowerCenter does not provide such feature out-of-the-box. Unless the structures of the source files and target tables are the same, you need to define separate source/target definitions and create mappings that use them.
However, you can use Stage Mapping Generator to generate a mapping for each file automatically.
PMy understanding is you have mant CSV files with different column layouts and you need to load them into appropriate tables in the Database.
Approach 1 : If you use any RDBMS you should have have some kind of import option. Explore that route to create tables based on csv files. This is a manual task.
Approach 2: Open the csv file and write formuale using the header to generate a create tbale statement. Execute the formula result in your DB. So, you will have many tables created. Now, use informatica to read the CSV and import all the tables and load into tables.
Approach 3 : using Informatica. You need to do lot of coding to create a dynamic mapping on the fly.
Proposed Solution :
mapping 1 :
1. Read the CSV file pass the header information to a java transformation
2. The java transformation should normalize and split the header column into rows. you can write them to a text file
3. Now you have all the columns in a text file. Read this text file and use SQL transformation to create the tables on the database
Mapping 2
Now, the table is available you need to read the CSV file excluding the header and load the data into the above table via SQL transformation ( insert statement) created by mapping 1
you can follow this approach for all the CSV files. I haven't tried this solution at my end but, i am sure that the above approach would work.
If you're not using any transformations, its wise to use Import option of the database. (e.g bteq script in Teradata). But if you are doing transformations, then you have to create as many Sources and targets as the number of files you have.
On the other hand you can achieve this in one mapping.
1. Create a separate flow for every file(i.e. Source-Transformation-Target) in the single mapping.
2. Use target load plan for choosing which file gets loaded first.
3. Configure the file names and corresponding database table names in the session for that mapping.
If all the mappings (if you have to create them separately) are same, use Indirect file Method. In the session properties under mappings tab, source option.., you will get this option. Default option will be Direct change it to Indirect.
I dont hav the tool now to explore more and clearly guide you. But explore this Indirect File Load type in Informatica. I am sure that this will solve the requirement.
I have written a workflow in Informatica that does it, but some of the complex steps are handled inside the database. The workflow watches a folder for new files. Once it sees all the files that constitute a feed, it starts to process the feed. It takes a backup in a time stamped folder and then copies all the data from the files in the feed into an Oracle table. An Oracle procedure gets to work and then transfers the data from the Oracle table into their corresponding destination staging tables and finally the Data Warehouse. So if I have to add a new file or a feed, I have to make changes in configuration tables only. No changes are required either to the Informatica Objects or the db objects. So the short answer is yes this is possible but it is not an out of the box feature.
In general I know how to import tab delimited file or comma delimited file. One of my client sent me delimited file. Example you can see below. And I don't how to import it without quotes.
"Make","Model","ModelYear","Trim","BodyStyle","Mileage","EngineDescription","Cylinders","FuelType","Transmission"
"Dodge","Stealth","1993","ES","Hatchback","107000","V-6","6-Cylinder","Gasoline",""
"GMC","Envoy XL","2003","SLE","Sport Utility","116000","6cyl","6-Cylinder","Gasoline","Automatic"
Could you guide me how to import and what settings do I need to change in order to import it with import wizard?
Thank you
You need to set the "Text qualifier" to ".
People might say it's a bit krufty, but the easiest way might be to open it as a CSV in excel and then copy/paste it right into your table using Management Studio.
If you are trying to do anything fancier than that, this method won't work. If not, you would be hard pressed to find a faster way to do it.
If the data is very simplistic you can use the SSMS import wizard. Right click the database, select tasks, select Import Data, and has been pointed out identify quotes as the text qualifier.
For a csv, tab delimited file you would change Data Source option to: Flat File Source
From there just answer the questions and follow the prompts.
I will warn you however. Unless the data is very simplistic, this method tends to produce a lot of errors. I almost always end up using SSIS which is a little more hassle and complex but a whole lot more powerful and dependable.