So I'm running a Linux based embedded system and using Busybox syslogd for system logging. When I have a system log statement:
syslog("testing \r\n");
the log will output
testing ^M
but the ^M is the actual literal characters ^(5E) and M(4D)
Is there any way to fix this without removing '\r'?
I tried removing '\r' which works but I can't do that for every log statement. I believe it has something to do with the way Busybox syslogd interprets '\r'.
I am using following postgres command in terminal to output very large query result into CSV format:
psql -d ecoprod -t -A -F"," -f queries/query.sql > exports/output.csv
It works just fine except its not valid CSV format. Text values should be wrapped in quotes "". Its not and its causing many problems parsing the CSV when there are commas in the text and so on.
Of course I could use another delimiter like semicolon however its the similar problem. In addition some text values contain line break characters which also breaks the parsing.
Didnt find any way to modify the command in documentation. Hope you will help me. Thank you.
-F doesn't promise to generate valid CSV. There is a --csv option you could use instead, which is at least intended for this purpose. But it seems like COPY or \copy would be more suited.
I'm using BCP to download data from SQL Server, using queryout option.
However, I notice that if the data content in any columns contain '\n', the content exported from BCP will be treated as newline.
For example, if the data in SQL Server is:
COLUMN_1 COLUMN_2
AAA NAME\nSURNAME
BBB NAMESURNAME
The exported file be like:
AAA NAME
SURNAME
BBB NAMESURNAME
Refer to BCP document, as I understand, the -c should not treat \n as newline.
-c
Performs the operation using a character data type. This option does not prompt for each field; it uses char as the storage type, without prefixes and with \t (tab character) as the field separator and \r\n (newline character) as the row terminator. -c is not compatible with -w.
I'm not sure what I misunderstood.
Here is the command I use:
bcp "select [col_name] from [table_name] where [condition]" queryout test.dat -U[username] -P[password] -S[serverip.port] -c
Thank you.
If your data contains newline or crlf control characters then those characters WILL, naturally, be included in the data that is copied out.
Are the control characters supposed to be there? If so, then leave them and they will be imported into whatever your destination is. Just because your text view shows a "broken" line, does not mean that SQL Server cannot ingest that line again and keep the control character tucked into the data (again, if those control characters belong there... i've seen plenty of cases where they would be).
If the newline character "\n" (or any control character for that matter) is NOT desired, then it's just a matter of doing as you commented in Martin's answer. Just clean the data either before you query it ("update") or during query (as you commented with "select/replace") or after you've copied the data out.
I've used "file cleaner" applications in the past to "clean" a file of unwanted characters (this can be an issue with long-lived data or data that has traversed various platforms or has been touched by humans!!! Yuck!).
I assume that your text includes the actual \n control character rather than simply the characters \ and n next to each other?
Where this exists then your options are to either use native mode or change the row terminator to be something other than \n so that it recognises the correct pattern.
I'd suggest using native mode and test whether that re-imports the data correctly with the \n in place.
My BCP command looks like this:
BCP azuredatabase.dbo.rawdata IN "clientPathToCSVFile" -S servername -U user#servername -P pw -c -t,-r\n
My CSV file is in {cr}{lf} format.
My CSV looks like this
125180896918,20,9,57.28,2020-01-04 23:02:21,282992,1327,4,2850280,49552
125180896919,20,10,57.82,2020-01-04 23:02:21,282992,1298,4,2850280,48881
125180896920,16,11,58.20,2020-01-04 23:02:21,282992,1065,4,2850280,48612
125180896921,20,12,69.10,2020-01-04 23:02:21,282992,515,4,2850280,10032
125180896922,20,13,70.47,2020-01-04 23:02:21,282992,1280,4,2850280,48766
125180896923,1,1,105.04,2020-01-04 23:02:21,,1296,4,2969398,49161
As you can see there are also empty fields.
My output looks like this
Starting copy...
0 rows copied.
Network packet size (bytes): 4096
Clock Time (ms.) Total : 547
So how do I correctly setup my command for BCP?
You stated that your data is in CRLF format (that means \r\n). But your bcp command is told to look for a line terminator of \n (using the -r option).
I would have expected to see the first half of your actual CRLF line terminator of "\r\n" be split in half with the \r being included in your last column and the \n being found as the line terminator, but it look like BCP loaded no rows because it found no \n in your file.
I have not worked with Azure/BCP much, so maybe someone else knows how BCP for Azure would handle this, but the SQL Server version of BCP would still find your \n and then load the \r into your last column.
Either that or your line terminator is now what you think it is. Have you viewed the file with a text editor (not notepad, not wordpad... something that will show hidden characters like line terminators and tabs and such).
Usually, when BCP loads no rows (and there are rows in the file to load), it could be a mixup with line terminators.
EDIT: Fixed! Changing grep to find fixed the newline issue.
I'm trying to output newline characters to a results file when I'm running a script to log test results. After every test result, I want the script to output a newline character for proper formatting.
The command I'm sending is:
tester.bat | grep "Passed all trials" > results.txt
It works properly, but it outputs like this inside results.txt:
Test #1: Passed all trialsTest #2: Passed all trials
and so on.
I'd like it to output like this inside results.txt:
Test #1: Passed all trials (newline)
Test #2: Passed all trials
Can it be done in a single line? If not, I'm open to longer solutions. If this is not possible, I guess I could write a separate script that will update the text file with appropriate newlines between "trialsTest," but this route is not preferable. Thanks in advance!
why using external utilities, when there are built-in function that do exactly what you Need?
tester.bat|find "Passed all trials"
or
tester.bat|findstr /c:"Passed all trials"