I am a Data Scientist who has been given a Data Engineering task and I am seriously struglling. Please help! I am very new to AzureSynapse and am facing an issue with inserts into my MSSQL DB.
I have a Lookup activity which reads a .json file, followed by a ForEach command which adds some columns and then inserts the rows into my DB.
The issue I am facing is that the foreign text is showing on the DB as ?????. The columns are set as dtype NVARCHAR, which can accept foreign text strings and the output of both the Lookup and ForEach activities have the correct format.
Why is the text showing as ??? on the DB and how can I fix this issue? Maybe my insert statement is incorrect?
I have posted screenshots below. Many thanks for any support.
Pipeline:
Lookup Output:
ForEach Output:
ForEach Activity:
ForEach Script:
DB dtypes:
DB Results:
The solution was hilariously simple. All I had to do was preprend an N before the Values where they were intended to be nvarchar. i.e. instead of '#{item().Jobtitle}' it should be N'#{item().Jobtitle}'.
Prepended the N to all NVARCHAR values and it shows in the DB with the foreign text.
Related
I'm trying to insert into an on-premises SQL database table called PictureBinary:
PictureBinary table
The source of the binary data is a table in another on-premises SQL database called DocumentBinary:
DocumentBinary table
I have a file with all of the Id's of the DocumentBinary rows that need copying. I feed those into a ForEach activity from a Lookup activity. Each of these files has about 180 rows (there are 50 files fed into a new instance of the pipeline in parallel).
Lookup and ForEach Activities
So far everything is working. But then, inside the ForEach I have another Lookup activity that tries to get the binary info to pass into a script that will insert it into the other database.
Lookup Binary column
And then the Script activity would insert the binary data into the table PictureBinary (in the other database).
Script to Insert Binary data
But when I debug the pipeline, I get this error when the binary column Lookup is reached:
ErrorCode=DataTypeNotSupported,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Column: coBinaryData,The data type ByteArray is not supported from the column named coBinaryData.,Source=,'
I know that the accepted way of storing the files would be to store them on the filesystem and just store the file path to the files in the database. But we are using a NOP database that stores the files in varbinary columns.
Also, if there is a better way of doing this, please let me know.
I tried to reproduce your scenario in my environment and got similar error
As per Microsoft document Columns with datatype Byte Array Are not supported in lookup activity is might be the main cause of error.
To workaround this as Follow below steps:
As you explained your case you have a file in which all the Id's of the DocumentBinary rows that need copy in destination are stored. To achieve this, you can simply use Copy activity with the Query where you copy records where the DocumentBinary in column is equal to the Id stored in file
First, I took lookup activity from where I can get Id's of the DocumentBinary rows stored in file
Then I took ForEach I passed the output of lookup activity to ForEach activity.
After this I took Copy activity in forEach activity
Select * from DocumentBinary
where coDocumentBinaryId = '#{item().PictureId}'
In source of copy activity select Use query as Query and pass above query with your names
Now go to Mapping Click on Import Schema then delete unwanted columns and map the columns accordingly.
Note: For this, columns in both tables are of similar datatypes either both uniqueidenntifier or both should be int
Sample Input in file:
Output (Copied only picture id contend in file from source to destination):
I am trying to insert records from Oracle to Postgresql. To do it, I use
QueryDatabaseTableRecord -> PutDatabaseRecord
QueryDatabaseTableRecord -> Fetch from Oracle
Writer -> CSV
PutDatabaseRecord -> to insert record to Postgresql.
Reader -> CSV
A few weeks ago, I faced with the same issue with Postgresql. cloudera question.
This time I made schema to public and Translate field name : false
I have changed postgresql table columns into block letters as I have used in oracle.
I found the solution for this. Its not directly related Apache-NiFi, kind of Postgresql related thing.
Data taken from Oracle comes with Upper-case headers. Headers MUST be converted to Lower-case. Creating postgresql columns with Upper-case wont solve this issue.
To do this, I have used Replace Text processor.
Search Value : MY_COLUMN1,MY_COLUMN2
Replacement Value : my_column1,my_column2
I hope this will help someone who is trying to get data from Oracle and put them back into Postgresql.
I'm using Microsoft SQL Server Management Studio. I have a spreadsheet that I'm manually importing into an existing table in an AZURE database using the SQL Server Import and Export Studio.
The column I'm hitting the issue with is a varchar(55). It's coming from a spreadsheet (.xlsx) where all the cells are formatted as "General" and the data can be a string of numbers "1234567" or a string of text then numbers "companyxyz_1234567."
If the first few rows in this column of the spreadsheet are a string of text then numbers (companyxyz_123456) everything works fine.
However, if the first few rows in this column of the spreadsheet are numeric only (1234567), then every entry in the column that contains text returns as NULL instead of the value once imported into the table.
I'm new to this system and have some basic knowledge in SQL and SQL coding but I'm at a loss here.
Any help is appreciated!
There's no Error return's all records are successfully inserted.
Here's the simulation:
insert into table_3 (sampletext) values ('1234567')
Record has been inserted.
insert into table_3 (sampletext) values (1234567)
Also inserted with the following sampletext varchar(55)
My suggestion is:
Kindly check the data of your excel file by:
Copy and paste the specific cell into notepad, maybe it had some
characters include on it.
I have been using ISQL (SQLAnywhere 12) to import data from CSVs into existing tables using INPUT INTO and never ran into a problem. Today I needed to import data into a table containing an auto-increment column, however, and thought I just needed to leave that column blank, so I tried it with a file containing only 1 row of data (to be safe). Turns out it imported with a 0 in the auto-increment field instead of the next integer value.
Looking at the Sybase documentation, it seems like I should be using LOAD TABLE instead, but the examples look a bit complex.
My questions are the following...
The documentation says the CSV file needs to be on the database server and not the client. I do not have access to the database server itself - can I load the file from within ISQL remotely?
How do I define the columns of the table I'm loading into? What if I am only loading data into a few columns and leaving the rest as NULL?
To confirm, this will leave existing data in the table as-is and simply add to it using whatever is in the CSV?
Many thanks in advance.
Yes. Check out the online documentation for LOAD TABLE - you can use the USING CLIENT FILE clause.
You can specify the column names in parens after the table name, i.e. LOAD TABLE mytable (col1, col2, col3) USING CLIENT FILE 'mylocalfile.txt'. Any columns not listed here will be set to NULL if the column is nullable or the equivalent to an empty string if it's not - this is why your autoincrement column was set to 0. You can use the DEFAULTS ON clause to get what you want.
Yes, existing data in the table is not affected.
I'm trying to use a BULK INSERT statement to populate a large (17-million-row) table in SQL Server from a text file. One column, of type nchar(17) has a UNIQUE constraint on it. I've checked (using some Python code) that the file contains no duplicates, but when I execute the query I get this error message from SQL Server:
Cannot insert duplicate key row in object 'dbo.tbl_Name' with unique index 'IX_tbl_Name'.
Could Server be transforming the text in some way as it executes BULK INSERT? Do SQL Server databases forbid any punctuation marks in nchar columns, or require that any be escaped? Is there any way I can find out which row is causing the trouble? Should I switch to some other method for inserting the data?
Your collation settings on the column could be causing data to be seen as duplicate, whereas your code may see it as unique. Things like accents and capitals can cause issues under certain collation settings.
Another thought too would be that empty or null values count as duplicates, so your Python code may not have found any text duplicates, but what about the empties?