I have loaded data into HANA using a CSV file and now I have added a new column to the table by
ALTER TABLE <tablename> ADD (<columnname> <datatype>);
Now I want to import data into this specific column using a CSV file. I could not find any methods to import to a specific column. Any help is appreciated. Thanks in advance.
The CSV import feature doesn't allow for partial loading and "column filling".
What you could do is to load the new data (together with the key columns of course) to a staging table and then update the target table columns from there.
Be aware that the CSV import is not meant to be an ETL solution. For that, the smart data integration features are there.
Related
I am new to AzureSynapse and am technically a Data Scientist whos doing a Data Engineering task. Please help!
I have some xlsx files containing raw data that I need to import into an SQL database table. The issue is that the raw data does not have a uniqueidentifer column and I need to add one before inserting the data into my SQL database.
I have been able to successfully add all the rows to the table by adding a new column on the Copy Data command and setting it to be #guid(). However, this sets the guid of every row to the same value (not unique for each row).
GUID mapping:
DB Result:
If I do not add this mapping, the pipeline throws an error stating that it cannot import a NULL Id into the column Id. Which makes sense as this column does not accept NULL values.
Is there a way to have AzureSynapse analystics read in a raw xlsx file and then import it into my DB with a unique identifier for each row? If so, how can I accomplish this?
Many many thanks for any support.
Giving dynamic content to a column in this way would generate the same value for entire column.
Instead, you can generate a new guid for each row using a for each activity.
You can retrieve the data from your source excel file using a lookup activity (my source only has name column). Give the output array of lookup activity to for each activity.
#activity('Lookup1').output.value
Inside for each, since you already have a linked service, create a script activity. In this script activity, you can create a query with dynamic content to insert values into the destination table. The following is the query I built using dynamic content.
insert into demo values ('#{guid()}','#{item().name}')
This allows you to iterate through source rows, insert each row individually while generating new guid every time
You can follow the above procedure to build a query to insert each row with unique identifier value. The following is an image where I used copy data to insert first 2 rows (same as yours) and inserted the next 2 rows using the above procedure.
NOTE: I have taken Azure SQL database for demo, but that does not affect the procedure.
I'm working on a stock trading project, and I have stock data saved as csv files in this format.
symbol
date
open
high
low
close
vol
AAA
20220627
24.38
24.38
24.365
24.365
500
I'm currently working on the database design, and I'm using SQL Server & SSMS. The issue is with the schema I've created, I don't have a table that shares exactly the same columns as this csv file. Therefore it's not as straight forward as just importing or bulk inserting the data directly into a table.
In my schema I came up with a Stock table
id
symbol
company_name
stock_exchange
And a Stock Data table
id
stock_id
date
open
high
low
close
vol
The csv data ultimately needs to go into my Stock Data table, however I need to figure out a way to convert the stock symbol to the correct id that each stock is being assigned by my Stock table. Is this an issue with my design or is there a simple way to handle this that I cannot seem to find? I had considered simply reading the csv data into a temporary table and then correctly inserting the data into the Stock Data table, but I wasn't sure how to easily accomplish that since I'll be inserting thousands of rows.
You can view my full diagram here - https://lucid.app/lucidchart/28591ceb-6574-4e22-a5ce-284cada1d832/edit?invitationId=inv_d5eb35d3-9bd0-4ba0-aa95-e7d70ca50562#
SSIS can do the lookup before doing the insert. This would be a typical ETL. I would likely use ELT and avoid a complicated SSIS package. There is no extract because you already have the csv file. Bulk load or use simple SSIS package to load the file. Then join the loaded data to the lookup table to do the insert to the StockData table. (You can do the join and insert in the SSIS package as a T-SQL execution. Or create a job to do the bulk insert and T-SQL.)
I have an empty table into which I am trying to import data from a flat file. The only option I seem to have is to create the table, which isn't needed. Why can't I select the other options?
In the screen before the one on your screenshot click where the destination table name is to select an existing table.
When we use the export records function in Filemaker we can see header information in the file (i.e., using the merge format) but the table information is missing. Is there a way of keeping the table info in the exported file?
So for example we have a table named 'T3' but when we export fields from this table the resulting .csv file reads:
__Delirium_DRStotal_score
instead of
T3::__Delirium_DRStotal_score
Any help much appreciated,
Many thanks
Steve
FileMaker can export related fields too and theese preserve their table name on export. At least this happens when you export into XML. So if you self-join the table to itself by a unique ID and export (identical) data from the related copy, it should give you the names you want. (Almost, because you'll have to name that other table differently.)
I have a large (multi-GB) data file exported from an Oracle table. I want to import this data into another Oracle instance, but I want the table name to be different from the original table. Is this possible? How?
Both importing and exporting systems are Oracle 11g. The table includes a BLOB column, if this makes any difference.
Thanks!
UPDATES:
The idea here was to update a table while keeping the downtime on the system that's using it to a minimum. The solution (based on Vincent Malgrat's answer and APC's update) is:
Assuming our table name is A
Make a temp schema TEMP_SCHEMA
Import our data into TEMP_SCHEMA.A
CREATE REAL_SCHEMA.B AS SELECT * FROM TEMP_SCHEMA.A
DROP TABLE REAL_SCHEMA.A Rename REAL_SCHEMA.A to REAL_SCHEMA.A_OLD
Rename REAL_SCHEMA.B to REAL_SCHEMA.A
DROP REAL_SCHEMA.A_OLD
This way, the downtime is only during steps 4 and 5, both should be independent of data size. I'll post an update here if this does not work :-)
If you are using the old EXP and IMP utilities you cannot do this. The only option is to import into a table of the same name (although you could change the schema which owns the table.
However, you say you are on 11g. Why not use the DataPump utility introduced in 10g, which replaces Import and Export. Because in 11g that utility offers the REMAP_TABLE option which does exactly what you want.
edit
Having read the comments the OP added to another response while I was writing this, I don't think the REMAP_TABLE option will work in their case. It only renames new objects. If a table with the original name exists in the target schema the import fails with ORA-39151. Sorry.
edit bis
Given the solution the OP finally chose (drop existing table, replace with new table) there is a solution with Data Pump, which is to use the TABLE_EXISTS_ACTION={TRUNCATE | REPLACE} clause. Choosing REPLACE drops the table whereas TRUNCATE merely, er, truncates it. In either case we have to worry about referential integrity constraints, but that is also an issue with the chosen solution.
I post this addendum not for the OP but for the benefit of other seekers who find this page some time in the future.
I suppose you want to import the table in a schema in which the name is already being used. I don't think you can change the table name during the import. However, you can change the schema with the FROMUSER and TOUSER option. This will let you import the table in another (temporary) schema.
When it is done copy the table to the target schema with a CREATE TABLE AS SELECT. The time it will take to copy the table will be negligible compared to the import so this won't waste too much time. You will need two times the disk space though during the operation.
Update
As suggested by Gary a cleverer method would be to create a view or synonym in the temporary schema that references the new table in the target schema. You won't need to copy the data after the import as it will go through directly to the target table.
Use the option REMAP_TABLE=EXISITNG_TABLE_NAME:NEW_TABLE_NAME in impdp. This works in 11gR2.
Just import it into a table with the same name, then rename the table.
Create a view as select * from ... the table you want to import into, with the view matching the name of the table in the export. Ignore errors on import.