(Using postgres 9.4beta2)
I have a dump that I want to import. I have done this with the 'psql' command, as elsewhere it is noted that this is required when using COPY FROM stdin:
psql publishing < publishing.dump.20150211160001
I get this syntax error:
ERROR: syntax error at or near "fbc61bc4"
LINE 1: fbc61bc4-2875-4a3a-8dec-91c8d8b60bcc root
The offending line in the dump file is the one after the COPY statement, here are both those lines together:
COPY content_fragment (id, content, name, content_item_id, entity_version) FROM stdin;
fbc61bc4-2875-4a3a-8dec-91c8dcontent Content for root content fbc61bc4-2875-4a3a-8dec-91c8d8b60bcc 0
The items in the data appear to be tab separated. I am wondering given that the error message says at or near "fbc61bc4", but the full string is "fbc61bc4-2875-4a3a-8dec-91c8dcontent", is psql not liking the '-' character?
This kind of error happens when the COPY itself fails because the table doesn't exist, or one of the mentioned columns doesn't, or the user lacks the permission to write into it, etc...
As COPY fails, the SQL interpreter continues to the next line and interpret it as if was an SQL statement, although it's actually data meant to be fed to COPY. Generally this leads to a syntax error, preceded by the error telling why the COPY failed (and often followed by tons of errors if there are many lines of data).
See another question: psql invalid command \N while restore sql, which shares the same root cause and has some useful comments.
Related
I am having a heck of a time using an environment variable with a semicolon in a properties file read by WildFly (24) in Linux. One like:
DATABASE_JDBC_URL=jdbc:sqlserver://sqlserver.c3klg5a2ws.us-east-1.rds.amazonaws.com:1433;DatabaseName=ejbca;encrypt=false
The issue is that its truncating things at the semicolon if I don't use quotes so I end up with it trying to write to master since it thinks no database is specified.
I have it setup so that variable is in a file called datasource.properties that gets read from standalone.conf where this variable sits:
JAVA_OPTS="$JAVA_OPTS -DDATABASE_JDBC_URL=${DATABASE_JDBC_URL}"
It's read in with the following in standalone.conf:
set -a
. /opt/wildfly_config/datasource.properties
set +a
That in turn gets populated in standalone.xml with:
<connection-url>${env.DATABASE_JDBC_URL}</connection-url>
I try putting it in quotes and oddly enough it doesn’t start at all. Standalone.sh is no longer able to parse it:
/opt/wildfly/bin/standalone.sh: line 338: --add-exports=java.desktop/sun.awt=ALL-UNNAMED: No such file or directory
So I then escape it in quotes like this:
DATABASE_JDBC_URL="jdbc:sqlserver://sqlserver.c3klg5a2ws.us-east-1.rds.amazonaws.com:1433\;DatabaseName=ejbca\;encrypt=false"
Startup looks good in the log output this way:
-DDATABASE_JDBC_URL=jdbc:sqlserver://sqlserver.c3klg5a2ws.us-east-1.rds.amazonaws.com:1433;DatabaseName=ejbca;encrypt=false
But then java doesn’t like it, for some reason it sees the escape ticks:
Caused by: com.microsoft.sqlserver.jdbc.SQLServerException: The port number 1433\ is not valid.
I can use sed to change the value in the standalone.xml, but all of the other properties I am doing work fine with the exception of this one and:
<check-valid-connection-sql>${env.DATABASE_CONNECTION_CHECK}</check-valid-connection-sql>
Where that value is "SELECT 1;" which it also does not like. That one worked with "'SELECT 1;'" but this one does not. I tried single quotes as well. That also gives the parsing error above. Is there any way to read in this environment variable that keeps wildfly happy?
You can enclose the characters you want to escape in { and } braces.
From the SQL Server documentation:
For example, {;} escapes a semicolon.
Just to note: Different database vendors will most likely have different ways of escaping characters in their connection URLs. The above approach works for SQL Server. But just to give one different example, MySQL uses URL encoding.
A alternate solution was to change how the variables were created. Part of the problem I had was that sourcing them from a properties file made them properties and not variables. I ended up creating a /opt/wildfly/bin/start.sh script with:
#!/bin/bash
export DATABASE_JDBC_URL="jdbc:sqlserver://sqlserver.c3klg5a2ws.us-east-1.rds.amazonaws.com:1433;DatabaseName=ejbca;encrypt=false"
/opt/wildfly/bin/standalone.sh
I then changed the wildfly service point to that new start.sh script. No longer have any parsing issues as long as the variables are started in memory.
No escaping was needed after that.
In a SSIS package, I want to send data from several instance to a flat files. To do so I create a dynamic connection string made of 3 variables:
".txt"
a Network path
The file name (which is the instance Name variable (string) that i use elsewhere in my package)
When i evaluate my expression at this point i receive :
For
TRIM(#[User::FileName]+REPLACE(#[User::ServerName],"\\","")+#[User::ExtensionFile])
I receive
\\test-01\TEMP\SQL01MyInstance.txt
But, when i run the job, it's unable to create the SQL01MyInstance.txt, and i receive as error :
[Flat File Destination [11]] Error: Cannot open the datafile "\\test-01\TEMP\SQL01MyInstance
.txt".
[SSIS.Pipeline] Error: Flat File Destination failed the pre-execute phase and returned error code 0xC020200E.
There's a unwanted space at the end filename, when i copy paste the error message elsewhere it appear to be a line return (before the .txt)
Does anybody know how can i get rid of it that line return (which i'm assuming is making the job fail) ?
Edit 1:
Rights on the destination folder are ok, because there's another flat file that I create in case of errors and it's created normally after that failure; but not with a dynamic name (normal behavior)
To remove line return you can use REPLACE() function with \r\n
REPLACE(REPLACE(TRIM(#[User::FileName]+REPLACE(#[User::ServerName],"\\","")+#[User::ExtensionFile]),"\r",""),"\n","")
Where
\r : carriadge return
\n : line feed
The TRIM function only trims the space character (versus other functions which trim all white space):
TRIM does not remove white-space characters such as the tab or line feed characters. Unicode provides code points for many different types of spaces, but this function recognizes only the Unicode code point 0x0020. When double-byte character set (DBCS) strings are converted to Unicode they may include space characters other than 0x0020 and the function cannot remove such spaces. To remove all kinds of spaces, you can use the Microsoft Visual Basic .NET Trim method in a script run from the Script component.
https://learn.microsoft.com/en-us/sql/integration-services/expressions/trim-ssis-expression
You can try this first to see if it works (Trim first then concatenate):
TRIM(#[User::FileName]) + TRIM(REPLACE(#[User::ServerName],"\","")) + TRIM(#[User::ExtensionFile]))
If not then you'll have to do the recommended String.Trim() function using a Script Task/Component that the MSDN article recommends (again, Trim each variable first, then concatenate)
I have a package that giving me a very confusing "Text was truncated or one or more characters had no match in the target code page" error but only when I run the full package in the control flow, not when I run just the task by itself.
The first task takes CSV files and combines them into one file. The next task reads the output of the previous file and begins to process the records. What is really odd is the truncation error is thrown in the flat file source in the 2nd step. This is the exact same flat file source which was the destination in the previous step.
If there was a truncation error wouldn't that be thrown by the previous step that tried to create the file? Since the 1st step created the file without truncation, why can't I just read that same file in the very next task?
Note - Only thing that makes this package different from the others I have worked on is I am dealing with special characters and using code page 65001 UTF-8 to capture the fields that have special characters. My other packages were all referencing flat file connection managers with code page 1252.
The problem was caused by the foreach loop and using the ColumnNamesInFirstDataRow expression where I have the formula "#[User::count_raw_input_rows] < 0". I have a variable initialized to -1 and I assign it to the ColumnNamesInFirstDataRow for the flat file. When in the loop I update the variable with a row counter on each read of a CSV file. This puts the header in the first time (-1) but then avoids repeating on all the other CSV files. When I exit the loop and try to read the input file it treats the header as data and blows up. I only avoided this on my last package because I didn't tighten the column definitions for the flat file like I did with this package. Thanks for the help.
I try to do a bulk copy from a file into a table in sqlserver 2008. Its only a part of the file that I want to copy (in the BCP command example bellow it is only line number 3 I will copy).
Sometimes I receive the "Unexpected EOF encountered" error message.
The command is:
BCP tblBulkCopyTest in d:\bc_file.txt -F3 -L3 -c -t, -S(local) -Utest -Ptest
When the file looks like the following, the copy works fine (the line number 3 is inserted into my table):
44261233,207,0,0,2168
44261234,207,0,0,2570
44261235,207,0,0,2572
When the file looks like the following, I receive "Unexpected EOF encountered" error message:
Test
44261233,207,0,0,2168
44261234,207,0,0,2570
44261235,207,0,0,2572
It seems like when the file starts with something not in the correct format, the BCP command fails, even though it is not the line I want to copy (in the bcp command I have specified line number 3).
When the file looks like the following, the copy works fine:
44261233,207,0,0,2168
44261234,207,0,0,2570
44261235,207,0,0,2572
Test
So it is only when the file has some incorrect data "before" the lines I want to copy, that I receive the error.
Any suggestions how the BCP command will ignore the lines, not in question?
The way I solve that kind of errors every day:
1) Create a table with single column tblRawBulk(RawRow varchar(1000)).
2) Insert all of the rows there.
3) Remove unnecessary unformatted rows (e.g. 'test' in your example) with WHERE-parameter. Or even remove all unnecessary rows (except of rows, that you need to load) to simplify point 5.
4) Upload this table with BCP to some workfolder.
5) Download this new file.
It's no exact what you want, but may help if you'll write corresponding stored procedure, that can automatically do all this things.
I'm working on a project where I have to parse a bunch of .csv files, all of different formats and containing different kinds of data through some C++ functions. After that I extract data from the files and create a .sql file that can be imported in psql to insert the data into a PostgreSQL database at a later stage.
But I am not able to figure out the correct syntax for the .sql file. Here is a sample table and a sample .sql file reproducing the same errors I am getting:
Table Creation Code:
CREATE TABLE "Sample_Table"
(
"Col_ID" integer NOT NULL,
"Col_Message" character varying(50),
CONSTRAINT "Sample_Table_pkey" PRIMARY KEY ("Col_ID" )
)
insertion.sql (after the copy line, fields separated by a single tab character)
copy Sample_Table (Col_ID, Col_Message) from stdin;
1 This is Spaaarta
2 Why So Serious
3 Baazinga
\.
Now if I execute the above sql file, I get the following error:
ERROR: syntax error at or near "1"
LINE 2: 1 This is Spaaarta
^
********** Error **********
If it can help, I'm running a PostgreSQL 9.1 release, and all the above queries were executed through PGAdmin III Software.
PgAdmin doesn't support executing COPY commands in the same way that psql does (or at least, it didn't the last time I tried it with version 1.14). Use psql to execute the script, or use INSERT statements.
Three things to check:
Is there actually exactly one tab characters between the columns? Spaces are a no-go.
Are there more error messages? I'm missing at least one. (See below)
When you force case sensitive table and column names you have to do this consequently. Therefore you must write this:
copy "Sample_Table" ("Col_ID", "Col_Message") from stdin;
Otherwise you will get theese errors:
psql:x.sql:1: ERROR: relation "sample_table" does not exist
psql:x.sql:5: invalid command \.
psql:x.sql:5: ERROR: syntax error at or near "1"
LINE 1: 1 This is Spaaarta
^
With these things in place I can use your example data successfully.
EDIT Bug change: The questioner now has
ERROR: invalid input syntax for integer: "1 'This is Spaaarta'"
So something with the 1 is not OK.
My guess is, that this is an encoding problem. Windows with it's UTF-16 stuff might be the culprit here.
Debugging these kind of problems other the web is not easy, because to many semi-intelligent programs are in the line, most of them like to adjust "a few" things.
But first check a few things in psql:
\encoding
show client_encoding;
show server_encoding;
According to the pastebin data these should be the same and one of "SQL_ASCII", "LATIN1" or "UTF-8".
If they already are or if adjusting them does not help: Unix/Linux/cygwin has a hexdump -C x.sql program, post its output to pastebin. DO NOT USE the hexdump from any Windows editor like ultraedit - they have fooled me several times. When transferring the file to Linux be sure to use binary transfer.