SQL Server - pattern matching a string - sql-server

Question

First post here. What a great resource. Hoping someone can help....
I have a character field that contains mostly numeric values but not all. The field, lets call it diag, is formatted as varchar(8). It contains diagnosis codes and they have been entered inconsistently at times. So I might see 29001 in the diag field. Or I might see 290.001. Sometimes people will code it as 290.00 other times 29000 and yet other times 290. To make it more complicated, I may have alpha characters in that field so the field could contain something like V700.00 or H601. Using these as examples, but it's indicative of what's in the field.
I am trying to find a range of values....for instance diagnosis codes between 29001 to 29999. Taking into account the inconsistencies in coding entry, I also want to return any records that have a diag value of 290.01 to 299.99 I am just at a loss. Searched here for hours and found a lot of info... but couldn't seem to answer my question. I am somewhat new to SQL and can't figure out how to return records that match the range of values I am looking for. There are 40-some million records so it is a lot of data. Trying to pare it down to something I can work with. I am using an older version of SQL Server...2005 in case it matters.
Any help would be most appreciated. I really don't even know where to start.
Thank you!

Answer 1

you can use this T-SQL to remove all non wanted characters in your numbers.
declare #strText varchar(50)
--set #strText = '23,112'
--set #strText = '23Ass112'
set #strText = '2.3.1.1.2'
WHILE PATINDEX('%[^0-9]%', #strText) > 0
BEGIN
SET #strText = STUFF(#strText, PATINDEX('%[^0-9]%', #strText), 1, '')
END
select #strText
on your case I suggest you to create a function
CREATE Function CleanNumbers(#strText VARCHAR(1000))
RETURNS VARCHAR(1000)
AS
WHILE PATINDEX('%[^0-9]%', #strText) > 0
BEGIN
SET #strText = STUFF(#strText, PATINDEX('%[^0-9]%', #strText), 1, '')
END
return #strText
END
Then you'll have to create a normal query calling the function.
WITH CTE as
(
SELECT dbo.CleanNumbers(yourtable.YourFakeNumber) as Number, yourtable.*
FROM yourtable
WHERE YourCriteria = 1
)
Select * from CTE where CAST(Number as int) between 29001 and 29999
Or easier
Select * from yourtable where CAST(dbo.CleanNumbers(YourFakeNumber) as int) between 29001 and 29999
I hope I haven't done any spelling mistakes ;)

Answer 2

It sounds like you have a little bit of a mess. If you know the rules for the variances, then you could build an automated script to update. But it sounds like it's pretty loose, so you might want to start by deciding what are valid values for the fields, making a table of them to validate against, and then identifying and classifying the invalid data.
First step, you need to get a list of valid diagnosis codes and get them into a table. Something like:
CREATE TABLE [dbo].[DiagnosticCodes](
[DiagnosticCode] [varchar](8) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
[DiagnosticDescription] [varchar](255) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
CONSTRAINT [PK_DiagnosticCodes] PRIMARY KEY CLUSTERED
(
[DiagnosticCode] ASC
)
)
Then get a list of the valid codes and import them into this table.
Then you need to find data in your table that is invalid. Something like this query will give you all the invalid codes in your database:
CREATE TABLE [dbo].[DiagnosticCodesMapping](
[Diag] [varchar](8) COLLATE SQL_Latin1_General_CP1_CI_AS NOT NULL,
[NewCode] [varchar](8) COLLATE SQL_Latin1_General_CP1_CI_AS NULL,
CONSTRAINT [PK_DiagnosticCodesMapping] PRIMARY KEY CLUSTERED
(
[Diag] ASC
)
)
insert into [dbo].[DiagnosticCodesMapping]
Select distinct MyDataTable.Diag, null NewCode
from MyDataTable
left join DiagnosticCodes
on MyDataTable.Diag = DiagnosticCodes.DiagnosticCode
where DiagnosticCodes.DiagnosticCode is null
This creates a table of all the invalid codes and also includes a field called NewCode, which you will populate a mapping from the invalid code to a new valid code. Hopefully this list will not be ridiculously long. Then you hand it over to someone for review and to enter the NewCode field to be one of the valid codes. Once you have your DiagnosticCodesMapping table completely filled in, you can then do an update to get all your fields to have valid codes:
update MyDataTable
set Diag=NewCode
from MyDataTable
join DiagnosticCodesMapping
where MyDataTable.Diag = DiagnosticCodesMapping.Diag
Doing it this way has the added advantage that you can now start validating all data entry in the future and you'll never have to do this cleanup again. You can create a constraint that ensures only valid codes from the DiagnosticCode table can be entered into the Diag field of your data table. You should check your interface to use the new lookup table as well. You'll also have to create a data maintenance interface to the DiagnosticCode table if you need to have super users with the ability to add new codes.