Related
I have a table in Excel. This table I load to MSSQL through SSIS. In the table is column TEXT that contains multiple sentences in one cell separated by enter spaces (alt + enter). My table looks like this:
NAME
ID
TEXT
JOHN SMITH
125
TEXT TEXT TEXT
But sometimes I have in column TEXT more enter spaces without text, like this:
NAME
ID
TEXT
JOHN SMITH
125
TEXT TEXT TEXT
I need to remove excess line breaks but keep enter space after each TEXT.
In basic case you can use REPLACE function for replace duplicates
UPDATE T SET T.T = REPLACE(T.T, CHAR(10) + CHAR(10), CHAR(10))
WHERE T.T LIKE('%' + CHAR(10) + CHAR(10) + '%')
But this query replace ONLY duplicates, so in case 3 or more duplicates you need repeat the query. For automate this you can use PROCEDURE like next:
CREATE PROCEDURE cleanDupes
AS
BEGIN
DECLARE #count INT
SET #count = 1
WHILE #count > 0
BEGIN
UPDATE T SET T.T = REPLACE(T.T, CHAR(10) + CHAR(10), CHAR(10))
WHERE T.T LIKE('%' + CHAR(10) + CHAR(10) + '%')
SET #count = (SELECT COUNT(*) FROM T WHERE T.T LIKE('%' + CHAR(10) + CHAR(10) + '%'))
END
SELECT * FROM T
END
GO
Test this solution on https://sqlize.online/sql/mssql2019/3ce3f73e8d27862fd7d268523a4a4a14/
I would like to replace (or remove) a newline character in a TSQL-string.
Any Ideas?
The obvious
REPLACE(#string, CHAR(13), '')
just won't do it...
Actually a new line in a SQL command or script string can be any of CR, LF or CR+LF. To get them all, you need something like this:
SELECT REPLACE(REPLACE(#str, CHAR(13), ''), CHAR(10), '')
REPLACE(#string, CHAR(13) + CHAR(10), '')
I may be a year late to the party, but I work on queries & MS-SQL every day, and I got tired of the built-in functions LTRIM() & RTRIM() (and always having to call them together), and of not catching 'dirty' data that had newlines at the end, so I decided it was high time to implement a better TRIM function. I'd welcome peer feedback!
Disclaimer: this actually removes (replaces with a single whitespace) extended forms of whitespace (tab, line-feed, carriage-return, etc.), so it's been renamed as "CleanAndTrim" from my original answer. The idea here is that your string doesn't need such extra special-whitespace characters inside it, and so if they don't occur at the head/tail, they should be replaced with a plain space. If you purposefully stored such characters in your string (say, your column of data that you're about to run this on), DON'T DO IT! Improve this function or write your own that literally just removes those characters from the endpoints of the string, not from the 'body'.
Okay, now that the disclaimer is updated, here's the code.
-- =============================================
-- Description: TRIMs a string 'for real' - removes standard whitespace from ends,
-- and replaces ASCII-char's 9-13, which are tab, line-feed, vert tab,
-- form-feed, & carriage-return (respectively), with a whitespace
-- (and then trims that off if it's still at the beginning or end, of course).
-- =============================================
CREATE FUNCTION [fn_CleanAndTrim] (
#Str nvarchar(max)
)
RETURNS nvarchar(max) AS
BEGIN
DECLARE #Result nvarchar(max)
SET #Result = LTRIM(RTRIM(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
LTRIM(RTRIM(#Str)), CHAR(9), ' '), CHAR(10), ' '), CHAR(11), ' '), CHAR(12), ' '), CHAR(13), ' ')))
RETURN #Result
END
Cheers!
Another Disclaimer:
Your typical Windows line-break is CR+LF, so if your string contains those, you'd end up replacing them with "double" spaces.
UPDATE, 2016:
A new version that gives you the option to replace those special-whitespace characters with other characters of your choice! This also includes commentary and the work-around for the Windows CR+LF pairing, i.e. replaces that specific char-pair with a single substitution.
IF OBJECT_ID('dbo.fn_CleanAndTrim') IS NULL
EXEC ('CREATE FUNCTION dbo.fn_CleanAndTrim () RETURNS INT AS BEGIN RETURN 0 END')
GO
-- =============================================
-- Author: Nate Johnson
-- Source: http://stackoverflow.com/posts/24068265
-- Description: TRIMs a string 'for real' - removes standard whitespace from ends,
-- and replaces ASCII-char's 9-13, which are tab, line-feed, vert tab, form-feed,
-- & carriage-return (respectively), with a whitespace or specified character(s).
-- Option "#PurgeReplaceCharsAtEnds" determines whether or not to remove extra head/tail
-- replacement-chars from the string after doing the initial replacements.
-- This is only truly useful if you're replacing the special-chars with something
-- **OTHER** than a space, because plain LTRIM/RTRIM will have already removed those.
-- =============================================
ALTER FUNCTION dbo.[fn_CleanAndTrim] (
#Str NVARCHAR(MAX)
, #ReplaceTabWith NVARCHAR(5) = ' '
, #ReplaceNewlineWith NVARCHAR(5) = ' '
, #PurgeReplaceCharsAtEnds BIT = 1
)
RETURNS NVARCHAR(MAX) AS
BEGIN
DECLARE #Result NVARCHAR(MAX)
--The main work (trim & initial replacements)
SET #Result = LTRIM(RTRIM(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
LTRIM(RTRIM(#Str)) --Basic trim
, NCHAR(9), #ReplaceTabWith), NCHAR(11), #ReplaceTabWith) --Replace tab & vertical-tab
, (NCHAR(13) + NCHAR(10)), #ReplaceNewlineWith) --Replace "Windows" linebreak (CR+LF)
, NCHAR(10), #ReplaceNewlineWith), NCHAR(12), #ReplaceNewlineWith), NCHAR(13), #ReplaceNewlineWith))) --Replace other newlines
--If asked to trim replacement-char's from the ends & they're not both whitespaces
IF (#PurgeReplaceCharsAtEnds = 1 AND NOT (#ReplaceTabWith = N' ' AND #ReplaceNewlineWith = N' '))
BEGIN
--Purge from head of string (beginning)
WHILE (LEFT(#Result, DATALENGTH(#ReplaceTabWith)/2) = #ReplaceTabWith)
SET #Result = SUBSTRING(#Result, DATALENGTH(#ReplaceTabWith)/2 + 1, DATALENGTH(#Result)/2)
WHILE (LEFT(#Result, DATALENGTH(#ReplaceNewlineWith)/2) = #ReplaceNewlineWith)
SET #Result = SUBSTRING(#Result, DATALENGTH(#ReplaceNewlineWith)/2 + 1, DATALENGTH(#Result)/2)
--Purge from tail of string (end)
WHILE (RIGHT(#Result, DATALENGTH(#ReplaceTabWith)/2) = #ReplaceTabWith)
SET #Result = SUBSTRING(#Result, 1, DATALENGTH(#Result)/2 - DATALENGTH(#ReplaceTabWith)/2)
WHILE (RIGHT(#Result, DATALENGTH(#ReplaceNewlineWith)/2) = #ReplaceNewlineWith)
SET #Result = SUBSTRING(#Result, 1, DATALENGTH(#Result)/2 - DATALENGTH(#ReplaceNewlineWith)/2)
END
RETURN #Result
END
GO
The Newline in T-SQL is represented by CHAR(13) & CHAR(10) (Carriage return + Line Feed). Accordingly, you can create a REPLACE statement with the text you want to replace the newline with.
REPLACE(MyField, CHAR(13) + CHAR(10), 'something else')
To do what most people would want, create a placeholder that isn't an actual line breaking character. Then you can actually combine the approaches for:
REPLACE(REPLACE(REPLACE(MyField, CHAR(13) + CHAR(10), 'something else'), CHAR(13), 'something else'), CHAR(10), 'something else')
This way you replace only once. The approach of:
REPLACE(REPLACE(MyField, CHAR(13), ''), CHAR(10), '')
Works great if you just want to get rid of the CRLF characters, but if you want a placeholder, such as
<br/>
or something, then the first approach is a little more accurate.
In SQL Server 2017 & later, use Trim
Select Trim(char(10) + char(13) from #str)
it trims on starting and ending, not in the middle
the order of \r and \n does not matter
I use it to trim special characters for a file name
Select Trim(char(10) + char(13) + ' *<>' from #fileName)
If your column data type is 'text' then you will get an error message as
Msg 8116, Level 16, State 1, Line 2 Argument data type text is
invalid for argument 1 of replace function.
In this case you need to cast the text as nvarchar and then replace
SELECT REPLACE(REPLACE(cast(#str as nvarchar(max)), CHAR(13), ''), CHAR(10), '')
Sometimes
REPLACE(myString, CHAR(13) + CHAR(10), ' ')
won't work. In that case use the following snippet code:
REPLACE(REPLACE(myString, CHAR(13),''), CHAR(10), ' ')
If you have an issue where you only want to remove trailing characters, you can try this:
WHILE EXISTS
(SELECT * FROM #ReportSet WHERE
ASCII(right(addr_3,1)) = 10
OR ASCII(right(addr_3,1)) = 13
OR ASCII(right(addr_3,1)) = 32)
BEGIN
UPDATE #ReportSet
SET addr_3 = LEFT(addr_3,LEN(addr_3)-1)
WHERE
ASCII(right(addr_3,1)) = 10
OR ASCII(right(addr_3,1)) = 13
OR ASCII(right(addr_3,1)) = 32
END
This solved a problem I had with addresses where a procedure created a field with a fixed number of lines, even if those lines were empty. To save space in my SSRS report, I cut them down.
If you have have open procedure with using sp_helptext then just copy all text in new sql query and press ctrl+h button use regular expression to replace and put ^\n in find field replace with blank .
for more detail check image.enter image description here
To #Cerebrus solution: for H2 for strings "+" is not supported. So:
REPLACE(string, CHAR(13) || CHAR(10), 'replacementString')
I was wanting to sanitize the contents of a column to generate a csv file, so want to get rid of the comma (,) within the varchar as well as newline and carrage-return.
I also wanted to eventually use the generated csv to create another script (to insert rows into another db) so also needed to change ' within the varchar to '' so ended up with this...
REPLACE(REPLACE(REPLACE(REPLACE(ErrorMessage, CHAR(13), ''), CHAR(10), ''),',',''),'''','''''')
There may be other nicer ways but it got the job done.
The answer posted above/earlier that was reported to replace CHAR(13)CHAR(10) carriage return:
REPLACE(REPLACE(REPLACE(MyField, CHAR(13) + CHAR(10), 'something else'), CHAR(13), 'something else'), CHAR(10), 'something else')
Will never get to the REPLACE(MyField, CHAR(13) + CHAR(10), 'something else') portion of the code and will return the unwanted result of:
'something else''something else'
And NOT the desired result of a single:
'something else'
That would require the REPLACE script to be rewritten as such:
REPLACE(REPLACE(REPLACE(MyField, CHAR(10), 'something else'), CHAR(13), 'something else'), CHAR(13) + CHAR(10), 'something else')
As the flow first tests the 1st/Furthest Left REPLACE statement, then upon failure will continue to test the next REPLACE statement.
I have one column for comment and I need to show this for one report.
Here what happen some time, users uses multiple enters in comment box. I can not access code part I need to manage this thing in SQL only.
So I have removed unwanted
1 /r/n
2 /n/n
from using
REPLACE(REPLACE(Desc, CHAR(13)+CHAR(10), CHAR(10)),CHAR(10)+CHAR(10), CHAR(10)) as Desc,
Now I want to remove any \r or \n from starting or ending of the string if any
By the way you meant in your question:(Remove char(10) or char(13) from specific string)
Note: You should see the output result by switching your resultset output to Results to Text(Ctrl+T).
Results to Text
Results to Grid
Use TRIM check here
Example : UPDATE tablename SET descriptions = TRIM(TRAILING "<br>" FROM descriptions)
if you want to replace newline then use something like below
SELECT REPLACE(REPLACE(#str, CHAR(13), ''), CHAR(10), '')
or
DECLARE #testString varchar(255)
set #testString = 'MY STRING '
/*Select the string and try to copy and paste into notepad and tab is still there*/
SELECT testString = #testString
/*Ok, it seems easy, let's try to trim this. Huh, it doesn't work, the same result here.*/
SELECT testStringTrim = RTRIM(#testString)
/*Let's try to get the size*/
SELECT LenOfTestString = LEN(#testString)
/*This supposed to give us string together with blank space, but not for tab though*/
SELECT DataLengthOfString= DATALENGTH(#testString)
SELECT ASCIIOfTab = ASCII(' ')
SELECT CHAR(9)
/*I always use this like a final solution*/
SET #testString = REPLACE(REPLACE(REPLACE(#testString, CHAR(9), ''), CHAR(10), ''), CHAR(13), '') SELECT #testString
/*
CHAR(9) - Tab
CHAR(10) - New Line
CHAR(13) - Carriage Return
*/
Reference
select dbo.trim('abc','c') -- ab
select dbo.trim('abc','a') -- bc
select dbo.trim(' b ',' ') -- b
Create a user-define-function: trim()
trim from both sides
trim any letter: space, \r, \n, etc
Create FUNCTION Trim
(
#Original varchar(max), #letter char(1)
)
RETURNS varchar(max)
AS
BEGIN
DECLARE #rtrim varchar(max)
SELECT #rtrim = iif(right(#original, 1) = #letter, left(#original,datalength(#original)-1), #original)
return iif( left(#rtrim,1) = #letter, right(#rtrim,datalength(#rtrim)-1),#rtrim)
END
We have issues with extended ascii codes getting in our database (128-155)
Is there anyway to search the entire database and display the results of any of these characters that may be in there and where they are located within the tables and columns.
Hope that makes sense.
I have the script to search entire DB, but having trouble with opening line.
DECLARE #SearchStr nvarchar(100)
SET #SearchStr != between char(32) and char(127)
I have this originally that works, but I need to extend the range I'm looking for.
SET #SearchStr = '|' + char(9) + '|' + char(10) + '|' + char(13)
Thanks
It's very unclear what your data looks like, but this might help you to get started:
declare #TestData table (String nvarchar(100))
insert into #TestData select N'abc'
insert into #TestData select N'def'
insert into #TestData select char(128)
insert into #TestData select char(155)
declare #SearchPattern nvarchar(max) = N'%['
declare #i int = 128
while #i <= 155
begin
set #SearchPattern += char(#i)
set #i += 1
end
set #SearchPattern += N']%'
select #SearchPattern
select String
from #TestData
where String like #SearchPattern
Of course you'll need to add some code to loop over every table and column that you want to query (see this question), and it's possible that this code will behave differently on different collations.
... where dodgyColumn is your column with questionable data ....
WHERE(patindex('%[' + char(127) + '-' + char(255) + ']%', dodgyColumn COLLATE Latin1_General_BIN2) > 0)
This works for us, to identify extended ASCII characters in our otherwise normal ASCII data (characters, numbers, punctuation, dollar and percent signs, etc.)
I need to ensure that a given field does not have more than one space (I am not concerned about all white space, just space) between characters.
So
'single spaces only'
needs to be turned into
'single spaces only'
The below will not work
select replace('single spaces only',' ',' ')
as it would result in
'single spaces only'
I would really prefer to stick with native T-SQL rather than a CLR based solution.
Thoughts?
Even tidier:
select string = replace(replace(replace(' select single spaces',' ','<>'),'><',''),'<>',' ')
Output:
select single spaces
This would work:
declare #test varchar(100)
set #test = 'this is a test'
while charindex(' ',#test ) > 0
begin
set #test = replace(#test, ' ', ' ')
end
select #test
If you know there won't be more than a certain number of spaces in a row, you could just nest the replace:
replace(replace(replace(replace(myText,' ',' '),' ',' '),' ',' '),' ',' ')
4 replaces should fix up to 16 consecutive spaces (16, then 8, then 4, then 2, then 1)
If it could be significantly longer, then you'd have to do something like an in-line function:
CREATE FUNCTION strip_spaces(#str varchar(8000))
RETURNS varchar(8000) AS
BEGIN
WHILE CHARINDEX(' ', #str) > 0
SET #str = REPLACE(#str, ' ', ' ')
RETURN #str
END
Then just do
SELECT dbo.strip_spaces(myText) FROM myTable
This is somewhat brute force, but will work
CREATE FUNCTION stripDoubleSpaces(#prmSource varchar(max)) Returns varchar(max)
AS
BEGIN
WHILE (PATINDEX('% %', #prmSource)>0)
BEGIN
SET #prmSource = replace(#prmSource ,' ',' ')
END
RETURN #prmSource
END
GO
-- Unit test --
PRINT dbo.stripDoubleSpaces('single spaces only')
single spaces only
update mytable
set myfield = replace (myfield, ' ', ' ')
where charindex(' ', myfield) > 0
Replace will work on all the double spaces, no need to put in multiple replaces. This is the set-based solution.
It can be done recursively via the function:
CREATE FUNCTION dbo.RemSpaceFromStr(#str VARCHAR(MAX)) RETURNS VARCHAR(MAX) AS
BEGIN
RETURN (CASE WHEN CHARINDEX(' ', #str) > 0 THEN
dbo.RemSpaceFromStr(REPLACE(#str, ' ', ' ')) ELSE #str END);
END
then, for example:
SELECT dbo.RemSpaceFromStr('some string with many spaces') AS NewStr
returns:
NewStr
some string with many spaces
Or the solution based on method described by #agdk26 or #Neil Knight (but safer)
both examples return output above:
SELECT REPLACE(REPLACE(REPLACE('some string with many spaces'
, ' ', ' ' + CHAR(7)), CHAR(7) + ' ', ''), ' ' + CHAR(7), ' ') AS NewStr
--but it remove CHAR(7) (Bell) from string if exists...
or
SELECT REPLACE(REPLACE(REPLACE('some string with many spaces'
, ' ', ' ' + CHAR(7) + CHAR(7)), CHAR(7) + CHAR(7) + ' ', ''), ' ' + CHAR(7) + CHAR(7), ' ') AS NewStr
--but it remove CHAR(7) + CHAR(7) from string
How it works:
Caution:
Char/string used to replace spaces shouldn't exist on begin or end of string and stand alone.
Here is a simple function I created for cleaning any spaces before or after, and multiple spaces within a string. It gracefully handles up to about 108 spaces in a single stretch and as many blocks as there are in the string. You can increase that by factors of 8 by adding additional lines with larger chunks of spaces if you need to. It seems to perform quickly and has not caused any problems in spite of it's generalized use in a large application.
CREATE FUNCTION [dbo].[fnReplaceMultipleSpaces] (#StrVal AS VARCHAR(4000))
RETURNS VARCHAR(4000)
AS
BEGIN
SET #StrVal = Ltrim(#StrVal)
SET #StrVal = Rtrim(#StrVal)
SET #StrVal = REPLACE(#StrVal, ' ', ' ') -- 16 spaces
SET #StrVal = REPLACE(#StrVal, ' ', ' ') -- 8 spaces
SET #StrVal = REPLACE(#StrVal, ' ', ' ') -- 4 spaces
SET #StrVal = REPLACE(#StrVal, ' ', ' ') -- 2 spaces
SET #StrVal = REPLACE(#StrVal, ' ', ' ') -- 2 spaces (for odd leftovers)
RETURN #StrVal
END
Method #1
The first method is to replace extra spaces between words with an uncommon symbol combination as a temporary marker. Then you can replace the temporary marker symbols using the replace function rather than a loop.
Here is a code example that replaces text within a String variable.
DECLARE #testString AS VARCHAR(256) = ' Test text with random* spacing. Please normalize this spacing!';
SELECT REPLACE(REPLACE(REPLACE(#testString, ' ', '*^'), '^*', ''), '*^', ' ');
Execution Time Test #1: In ten runs of this replacement method, the average wait time on server replies was 1.7 milliseconds and total execution time was 4.6 milliseconds.
Execution Time Test #2: The average wait time on server replies was 1.7 milliseconds and total execution time was 3.7 milliseconds.
Method #2
The second method is not quite as elegant as the first, but also gets the job done. This method works by nesting four (or optionally more) replace statements that replace two blank spaces with one blank space.
DECLARE #testString AS VARCHAR(256) = ' Test text with random* spacing. Please normalize this spacing!';
SELECT REPLACE(REPLACE(REPLACE(REPLACE(#testString,' ',' '),' ',' '),' ',' '),' ',' ')
Execution Time Test #1: In ten runs of this replacement method, the average wait time on server replies was 1.9 milliseconds and total execution time was 3.8 milliseconds.
Execution Time Test #2: The average wait time on server replies was 1.8 milliseconds and total execution time was 4.8 milliseconds.
Method #3
The third method of replacing extra spaces between words is to use a simple loop. You can do a check on extra spaces in a while loop and then use the replace function to reduce the extra spaces with each iteration of the loop.
DECLARE #testString AS VARCHAR(256) = ' Test text with random* spacing. Please normalize this spacing!';
WHILE CHARINDEX(' ',#testString) > 0
SET #testString = REPLACE(#testString, ' ', ' ')
SELECT #testString
Execution Time Test #1: In ten runs of this replacement method, the average wait time on server replies was 1.8 milliseconds and total execution time was 3.4 milliseconds.
Execution Time Test #2: The average wait time on server replies was 1.9 milliseconds and total execution time was 2.8 milliseconds.
This is the solution via multiple replace, which works for any strings (does not need special characters, which are not part of the string).
declare #value varchar(max)
declare #result varchar(max)
set #value = 'alpha beta gamma delta xyz'
set #result = replace(replace(replace(replace(replace(replace(replace(
#value,'a','ac'),'x','ab'),' ',' x'),'x ',''),'x',''),'ab','x'),'ac','a')
select #result -- 'alpha beta gamma delta xyz'
You can try this:
select Regexp_Replace('single spaces only','( ){2,}', ' ') from dual;
Just Adding Another Method-
Replacing Multiple Spaces with Single Space WITHOUT Using REPLACE in SQL Server-
DECLARE #TestTable AS TABLE(input VARCHAR(MAX));
INSERT INTO #TestTable VALUES
('HAPPY NEWYEAR 2020'),
('WELCOME ALL !');
SELECT
CAST('<r><![CDATA[' + input + ']]></r>' AS XML).value('(/r/text())[1] cast as xs:token?','VARCHAR(MAX)')
AS Expected_Result
FROM #TestTable;
--OUTPUT
/*
Expected_Result
HAPPY NEWYEAR 2020
WELCOME ALL !
*/
Found this while digging for an answer:
SELECT REPLACE(
REPLACE(
REPLACE(
LTRIM(RTRIM('1 2 3 4 5 6'))
,' ',' '+CHAR(7))
,CHAR(7)+' ','')
,CHAR(7),'') AS CleanString
where charindex(' ', '1 2 3 4 5 6') > 0
The full answer (with explanation) was pulled from: http://techtipsbysatish.blogspot.com/2010/08/sql-server-replace-multiple-spaces-with.html
On second look, seems to be just a slightly different version of the selected answer.
Please Find below code
select trim(string_agg(value,' ')) from STRING_SPLIT(' single spaces only ',' ')
where value<>' '
This worked for me..
Hope this helps...
With the "latest" SQL Server versions (Compatibility level 130) you could also use string_split and string_agg.
string_split can return an ordinal column when provided with a third argument. (https://learn.microsoft.com/en-us/sql/t-sql/functions/string-split-transact-sql?view=sql-server-ver16#enable_ordinal). So we can preserve the order of the string_split.
Using a common table expression:
with cte(value) as (select value from string_split(' a b c d e ', ' ', 1) where value <> '' order by ordinal offset 0 rows)
select string_agg(value, ' ') from cte
a b c d e results in a b c d e
I use FOR XML PATH solution to replace multiple spaces into single space
The idea is to replace spaces with XML tags
Then split XML string into string fragments without XML tags
Finally concatenating those string values by adding single space characters between two
Here is how final UDF function can be called
select dbo.ReplaceMultipleSpaces(' Sample text with multiple space ')