I would like some help making my SQL Server 2016 stored procedure more efficient. I got it to work and that is 50% of my battle but I know that many (if not most) of you folks have much more experience with SQL Server stored procedures than I do.
My code so far:
DECLARE #U1A nvarchar(50), #U2A nvarchar(50),
#U3A nvarchar(50), #U4A nvarchar(50),
#U5A nvarchar(50), #U6A nvarchar(50),
#U7A nvarchar(50), #U8A nvarchar(50),
#U9A nvarchar(50)
DECLARE #Jsonstring nvarchar(max)
DECLARE #recCount int
SELECT
#recCount = COUNT(*)
FROM
[dbo].[Staging_PersonalInformation]
WHERE
jsondata IS NULL
WHILE #recCount > 0
BEGIN
SELECT TOP 1
#U1A = [FirstName], #U2A = [MiddleName],
#U3A = [LastName], #U4A = [EmailAddress],
#U5A = eraCommons, #U6A = [PositionTitle],
#U7A = [MyNCBILink], #U8A = [UniqueID],
#U9A = [ReferenceID]
FROM
[dbo].[Staging_PersonalInformation]
WHERE
jsondata IS NULL
SET #Jsonstring = '[{"name":"FirstName","value":"'+isnull(#U1A, '')+'"},{"name":"Middlename","value":"'+ISNULL(#U2A, '')+'"},{"name":"LastName","value":"'+isnull(#U3A, '')+'"},{"name":"emailaddress","value":"'+isnull(#U4A, '')+'"},{"name":"eRACommons","value":"'+ISNULL(#U5A, '')+'"},{"name":"positionTitle","value":"'+ISNULL(#U6A, '')+'"},{"name":"MyNCBILink","value":"'+ISNULL(#U7A, '')+'"},{"name":" uniqueid","value":"'+ISNULL(#U8A, '')+'"},{"name":"ReferenceID","value":"'+ISNULL(#U9A, '')+'"}]'
UPDATE Staging_PersonalInformation
SET JsonData = #Jsonstring
WHERE (EmailAddress = #U4A);
SET #recCount = #recCount - 1
END
The purpose of this is to take the individual column values and make a string that my sterilized JavaScript form can repopulate. I would rather store the string than to make it on the fly each time.
Thanks for your help
Well the biggest issue is that looping is horribly inefficient. And since you are always going to update this column based on values already in the table you could use a computed column and avoid all this work entirely.
I would suggest that in the future you give your variable names something meaningful instead of just numbering them.
Here is how you could make this a computed column. You can read more about computed columns here. https://technet.microsoft.com/en-us/library/ms191250.aspx
alter table [dbo].[Staging_PersonalInformation]
add jsondata as '[{"name":"FirstName","value":"' + isnull(FirstName, '')
+ '"},{"name":"Middlename","value":"' + ISNULL(MiddleName, '')
+ '"},{"name":"LastName","value":"' + isnull(LastName, '')
+ '"},{"name":"emailaddress","value":"'+isnull(EmailAddress, '')
+ '"},{"name":"eRACommons","value":"'+ISNULL(eraCommons, '')
+ '"},{"name":"positionTitle","value":"'+ISNULL(PositionTitle, '')
+ '"},{"name":"MyNCBILink","value":"'+ISNULL(MyNCBILink, '')
+ '"},{"name":" uniqueid","value":"'+ISNULL(UniqueID, '')
+ '"},{"name":"ReferenceID","value":"'+ISNULL(ReferenceID, '')
+ '"}]'
The answer by Sean Lange is a great answer, but I am curious as to why you are not taking advantage of SQL Server 2016 support of for json.
I realize that the format is not the same as you specified, so I suppose that could be the reason. Perhaps this format would also work?:
select *
from Staging_PersonalInformation
for json auto, include_null_values
dbfiddle.uk demo: http://dbfiddle.uk/?rdbms=sqlserver_2016&fiddle=d533c6d3b82fdd7865b3817fba94037d
returns:
[{"Id":1,"FirstName":"Sean","MiddleName":null,"LastName":"Lange","EmailAddress":null,"PositionTitle":null,"MyNCBILink":null,"UniqueID":"6E6732A9-9FC9-4B6E-8695-AF6BB2DA2152","ReferenceID":0}
,{"Id":2,"FirstName":"Sql","MiddleName":null,"LastName":"Zim","EmailAddress":null,"PositionTitle":null,"MyNCBILink":null,"UniqueID":"FA33808B-E8BE-41B5-AA89-DA8A37503F8F","ReferenceID":0}]
Reference:
JSON support in SQL Server 2016 - Robert Sheldon
JSON Data
Include Null Values in JSON - include_null_values Option
Related
I have a problem I think it's something simple but I'm just getting started on this, I have a .txt file that contains
Kayle;Osvorn;35;4399900433
What would be these my columns: First name;Last name;Age;Phone
I need to separate them through the process of transformation of the derived column into ETL but for now only the first and last name I have been able to extract and the rest I do not know how to continue.
I have this for the first two columns
Name = SUBSTRING(CustomerData,1,FINDSTRING(CustomerData,";",1) - 1)
Last Name = SUBSTRING(CustomerData,FINDSTRING(CustomerData,";",1) + 1,LEN(CustomerData))
Age = ?
Phone = ?
Does anyone have any idea how the expression would go?
There's no need to use a Derived Column transformation in an SSIS package. Instead, in your Flat File Connection Manager, define your field separator as the semicolon ; instead of the default comma ','. Indicate that it should ... identify columns and now your single column of CustomerData goes away and you have nice delimited columns.
If you have column headers, it should pull that out. Otherwise, you will need to specify no header and then go into the advanced tab and give them friendly names.
Please use this below logic to achieve your requirement-
Demo Here
DECLARE #T VARCHAR(200) = 'Kayle;Osvorn;35;4399900433'
DECLARE #index_1 INT
DECLARE #index_2 INT
DECLARE #index_3 INT
DECLARE #name VARCHAR(100)
DECLARE #last_name VARCHAR(100)
DECLARE #age VARCHAR(100)
DECLARE #phone VARCHAR(100)
SELECT #index_1 = CHARINDEX(';',#T,0) + 1
SELECT #index_2 = CHARINDEX(';',#T,#index_1 + 1) + 1
SELECT #index_3 = CHARINDEX(';',#T,#index_2 + 1) + 1
SELECT
#name = SUBSTRING(#T,0,#index_1 - 1),
#last_name = SUBSTRING(#T, #index_1 ,#index_2 - #index_1 - 1),
#age = SUBSTRING(#T,#index_2, #index_3 - #index_2 - 1),
#phone = SUBSTRING(#T,#index_3,LEN(#T))
SELECT #name,#last_name, #age,#phone
There is one simple way by doing the same operation on the REVERSEd string:
[Name] = SUBSTRING(#CustomerData,1,FINDSTRING(#CustomerData,";",1) - 1)
[Last Name] = SUBSTRING(#CustomerData, FINDSTRING(#CustomerData, ";",1) + 1,
FINDSTRING(SUBSTRING(#CustomerData, FINDSTRING(#CustomerData, ";",1)+1, LEN(#CustomerData)),";",1)-1)
Age = REVERSE(SUBSTRING(REVERSE(#CustomerData), FINDSTRING(REVERSE(#CustomerData),";",1)+1,
FINDSTRING(SUBSTRING(REVERSE(#CustomerData), FINDSTRING(";",REVERSE(#CustomerData),1) + 1, LEN(#CustomerData)),";",1)-1))
Phone = REVERSE(SUBSTRING(REVERSE(#CustomerData),1,FINDSTRING(REVERSE(#CustomerData),";",1) - 1))
If you need to do that using a transformation, why not using the TOKEN() function?
Name = TOKEN(CustomerData,";",1)
Last Name = TOKEN(CustomerData,";",2)
Age = TOKEN(CustomerData,";",3)
Phone = TOKEN(CustomerData,";",4)
I received help previously on creating a query for listing total sales by month for a list of jobs. However, I cannot put this into a view as a variable needs declared.
I thought I could use a Table Valued Function (which I've never used before) but apparently I need to define the columns of the table which I cannot do if they are to be dynamic (the columns are yyyy-mm).
Can someone suggest how I could approach this problem please?
I am not familiar with creating anything other than views so this is completely new to me. Obviously if using the TVF isn't the best method please advise.
The query:
Declare #SQL varchar(max) = Stuff((Select Distinct ',' + QuoteName(YearMonth) From v_JobSalesByMonth Order by 1 For XML Path('')),1,1,'')
Select #SQL = '
Select [JobID], [TotalJobSales],' + #SQL + '
From (
Select JobID
,TotalJobSales = sum(SalesForMonth) over (Partition By JobID)
,YearMonth
,SalesForMonth
From v_JobSalesByMonth A) A
Pivot (sum(SalesForMonth) For [YearMonth] in (' + #SQL + ') ) p'
Exec(#SQL);
I started to create the function but am confused about how to define the dynamic columns.
CREATE FUNCTION db.tvfnJobSalesByMonthPivot (#JobID INT)
RETURNS #JobSalesByMonth TABLE
(
JobID int PRIMARY KEY NOT NULL,
TotalJobSales decimal(6,2) NULL,
2016-12 (ermmmmm?)
)
Many thanks in advance.
Paul
I'm trying to merge a very wide table from a source (linked Oracle server) to a target table (SQL Server 2012) w/o listing all the columns. Both tables are identical except for the records in them.
This is what I have been using:
TRUNCATE TABLE TargetTable
INSERT INTO TargetTable
SELECT *
FROM SourceTable
When/if I get this working I would like to make it a procedure so that I can pass into it the source, target and match key(s) needed to make the update. For now I would just love to get it to work at all.
USE ThisDatabase
GO
DECLARE
#Columns VARCHAR(4000) = (
SELECT COLUMN_NAME + ','
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'TargetTable'
FOR XML PATH('')
)
MERGE TargetTable AS T
USING (SELECT * FROM SourceTable) AS S
ON (T.ID = S.ID AND T.ROWVERSION = S.ROWVERSION)
WHEN MATCHED THEN
UPDATE SET #Columns = S.#Columns
WHEN NOT MATCHED THEN
INSERT (#Columns)
VALUES (S.#Columns)
Please excuse my noob-ness. I feel like I'm only half way there, but I don't understand some parts of SQL well enough to put it all together. Many thanks.
As previously mentioned in the answers, if you don't want to specify the columns , then you have to write a dynamic query.
Something like this in your case should help:
DECLARE
#Columns VARCHAR(4000) = (
SELECT COLUMN_NAME + ','
FROM INFORMATION_SCHEMA.COLUMNS
WHERE TABLE_NAME = 'TargetTable'
FOR XML PATH('')
)
DECLARE #MergeQuery NVARCHAR(MAX)
DECLARE #UpdateQuery VARCHAR(MAX)
DECLARE #InsertQuery VARCHAR(MAX)
DECLARE #InsertQueryValues VARCHAR(MAX)
DECLARE #Col VARCHAR(200)
SET #UpdateQuery='Update Set '
SET #InsertQuery='Insert ('
SET #InsertQueryValues=' Values('
WHILE LEN(#Columns) > 0
BEGIN
SET #Col=left(#Columns, charindex(',', #Columns+',')-1);
IF #Col<> 'ID' AND #Col <> 'ROWVERSION'
BEGIN
SET #UpdateQuery= #UpdateQuery+ 'TargetTable.'+ #Col + ' = SourceTable.'+ #Col+ ','
SET #InsertQuery= #InsertQuery+#Col + ','
SET #InsertQueryValues=#InsertQueryValues+'SourceTable.'+ #Col+ ','
END
SET #Columns = stuff(#Columns, 1, charindex(',', #Columns+','), '')
END
SET #UpdateQuery=LEFT(#UpdateQuery, LEN(#UpdateQuery) - 1)
SET #InsertQuery=LEFT(#InsertQuery, LEN(#InsertQuery) - 1)
SET #InsertQueryValues=LEFT(#InsertQueryValues, LEN(#InsertQueryValues) - 1)
SET #InsertQuery=#InsertQuery+ ')'+ #InsertQueryValues +')'
SET #MergeQuery=
N'MERGE TargetTable
USING SourceTable
ON TargetTable.ID = SourceTable.ID AND TargetTable.ROWVERSION = SourceTable.ROWVERSION ' +
'WHEN MATCHED THEN ' + #UpdateQuery +
' WHEN NOT MATCHED THEN '+#InsertQuery +';'
Execute sp_executesql #MergeQuery
If you want more information about Merge, you could read the this excellent article
Don't feel bad. It takes time. Merge has interesting syntax. I've actually never used it. I read Microsoft's documentation on it, which is very helpful and even has examples. I think I covered everything. I think there may be a slight amount of tweaking you might have to do, but I think it should work.
Here's the documentation for MERGE:
https://msdn.microsoft.com/en-us/library/bb510625.aspx
As for your code, I commented pretty much everything to explain it and show you how to do it.
This part is to help write your merge statement
USE ThisDatabase --This says what datbase context to use.
--Pretty much what database your querying.
--Like this: database.schema.objectName
GO
DECLARE
#SetColumns VARCHAR(4000) = (
SELECT CONCAT(QUOTENAME(COLUMN_NAME),' = S.',QUOTENAME(COLUMN_NAME),',',CHAR(10)) --Concat just says concatenate these values. It's adds the strings together.
--QUOTENAME adds brackets around the column names
--CHAR(10) is a line break for formatting purposes(totally optional)
FROM INFORMATION_SCHEMA.COLUMNS
--WHERE TABLE_NAME = 'TargetTable'
FOR XML PATH('')
) --This uses some fancy XML trick to get your Columns concatenated into one row.
--What really is in your table is a column of your column names in different rows.
--BTW If the columns names in both tables are identical, then this will work.
DECLARE #Columns VARCHAR(4000) = (
SELECT QUOTENAME(COLUMN_NAME) + ','
FROM INFORMATION_SCHEMA.COLUMNS
--WHERE TABLE_NAME = 'TargetTable'
FOR XML PATH('')
)
SET #Columns = SUBSTRING(#Columns,0,LEN(#Columns)) -- this gets rid off the comma at the end of your list
SET #SetColumns = SUBSTRING(#SetColumns,0,LEN(#SetColumns)) --same thing here
SELECT #SetColumns --Your going to want to copy and paste this into your WHEN MATCHED statement
SELECT #Columns --Your going to want to copy this into your WHEN NOT MATCHED statement
GO
Merge Statement
Especially look at my notes on ROWVERSION.
MERGE INTO TargetTable AS T
USING SourceTable AS S --Don't really need to write SELECT * FROM since you need the whole table anyway
ON (T.ID = S.ID AND T.[ROWVERSION] = S.[ROWVERSION]) --These are your matching parameters
--One note on this, if ROWVERSION is different versions of the same data you don't want to have RowVersion here
--Like lets say you have ID 1 ROWVERSION 2 in your source but only version 1 in your targetTable
--If you leave T.ID =S.ID AND T.ROWVERSION = S.ROWVERSION, then it will insert the new ROWVERSION
--So you'll have two versions of ID 1
WHEN MATCHED THEN --When TargetTable ID and ROWVERSION match in the matching parameters
--Update the values in the TargetTable
UPDATE SET /*Copy and Paste #SetColumnss here*/
--Should look like this(minus the "--"):
--Col1 = S.Col1,
--Col2 = S.Col2,
--Col3 = S.Col3,
--Etc...
WHEN NOT MATCHED THEN --This says okay there are no rows with the existing ID, now insert a new row
INSERT (col1,col2,col3) --Copy and paste #Columns in between the parentheses. Should look like I show it. Note: This is insert into target table so your listing the target table columns
VALUES (col1,col2,col3) --Same thing here. This is the list of source table columns
I am stuck on converting a varchar column UserID to INT. I know, please don't ask why this UserID column was not created as INT initially, long story.
So I tried this, but it doesn't work. and give me an error:
select CAST(userID AS int) from audit
Error:
Conversion failed when converting the varchar value
'1581............................................................................................................................' to data type int.
I did select len(userID) from audit and it returns 128 characters, which are not spaces.
I tried to detect ASCII characters for those trailing after the ID number and ASCII value = 0.
I have also tried LTRIM, RTRIM, and replace char(0) with '', but does not work.
The only way it works when I tell the fixed number of character like this below, but UserID is not always 4 characters.
select CAST(LEFT(userID, 4) AS int) from audit
You could try updating the table to get rid of these characters:
UPDATE dbo.[audit]
SET UserID = REPLACE(UserID, CHAR(0), '')
WHERE CHARINDEX(CHAR(0), UserID) > 0;
But then you'll also need to fix whatever is putting this bad data into the table in the first place. In the meantime perhaps try:
SELECT CONVERT(INT, REPLACE(UserID, CHAR(0), ''))
FROM dbo.[audit];
But that is not a long term solution. Fix the data (and the data type while you're at it). If you can't fix the data type immediately, then you can quickly find the culprit by adding a check constraint:
ALTER TABLE dbo.[audit]
ADD CONSTRAINT do_not_allow_stupid_data
CHECK (CHARINDEX(CHAR(0), UserID) = 0);
EDIT
Ok, so that is definitely a 4-digit integer followed by six instances of CHAR(0). And the workaround I posted definitely works for me:
DECLARE #foo TABLE(UserID VARCHAR(32));
INSERT #foo SELECT 0x31353831000000000000;
-- this succeeds:
SELECT CONVERT(INT, REPLACE(UserID, CHAR(0), '')) FROM #foo;
-- this fails:
SELECT CONVERT(INT, UserID) FROM #foo;
Please confirm that this code on its own (well, the first SELECT, anyway) works for you. If it does then the error you are getting is from a different non-numeric character in a different row (and if it doesn't then perhaps you have a build where a particular bug hasn't been fixed). To try and narrow it down you can take random values from the following query and then loop through the characters:
SELECT UserID, CONVERT(VARBINARY(32), UserID)
FROM dbo.[audit]
WHERE UserID LIKE '%[^0-9]%';
So take a random row, and then paste the output into a query like this:
DECLARE #x VARCHAR(32), #i INT;
SET #x = CONVERT(VARCHAR(32), 0x...); -- paste the value here
SET #i = 1;
WHILE #i <= LEN(#x)
BEGIN
PRINT RTRIM(#i) + ' = ' + RTRIM(ASCII(SUBSTRING(#x, #i, 1)))
SET #i = #i + 1;
END
This may take some trial and error before you encounter a row that fails for some other reason than CHAR(0) - since you can't really filter out the rows that contain CHAR(0) because they could contain CHAR(0) and CHAR(something else). For all we know you have values in the table like:
SELECT '15' + CHAR(9) + '23' + CHAR(0);
...which also can't be converted to an integer, whether you've replaced CHAR(0) or not.
I know you don't want to hear it, but I am really glad this is painful for people, because now they have more war stories to push back when people make very poor decisions about data types.
This question has got 91,000 views so perhaps many people are looking for a more generic solution to the issue in the title "error converting varchar to INT"
If you are on SQL Server 2012+ one way of handling this invalid data is to use TRY_CAST
SELECT TRY_CAST (userID AS INT)
FROM audit
On previous versions you could use
SELECT CASE
WHEN ISNUMERIC(RTRIM(userID) + '.0e0') = 1
AND LEN(userID) <= 11
THEN CAST(userID AS INT)
END
FROM audit
Both return NULL if the value cannot be cast.
In the specific case that you have in your question with known bad values I would use the following however.
CAST(REPLACE(userID COLLATE Latin1_General_Bin, CHAR(0),'') AS INT)
Trying to replace the null character is often problematic except if using a binary collation.
This is more for someone Searching for a result, than the original post-er. This worked for me...
declare #value varchar(max) = 'sad';
select sum(cast(iif(isnumeric(#value) = 1, #value, 0) as bigint));
returns 0
declare #value varchar(max) = '3';
select sum(cast(iif(isnumeric(#value) = 1, #value, 0) as bigint));
returns 3
I would try triming the number to see what you get:
select len(rtrim(ltrim(userid))) from audit
if that return the correct value then just do:
select convert(int, rtrim(ltrim(userid))) from audit
if that doesn't return the correct value then I would do a replace to remove the empty space:
select convert(int, replace(userid, char(0), '')) from audit
This is how I solved the problem in my case:
First of all I made sure the column I need to convert to integer doesn't contain any spaces:
update data set col1 = TRIM(col1)
I also checked whether the column only contains numeric digits.
You can check it by:
select * from data where col1 like '%[^0-9]%' order by col1
If any nonnumeric values are present, you can save them to another table and remove them from the table you are working on.
select * into nonnumeric_data from data where col1 like '%[^0-9]%'
delete from data where col1 like '%[^0-9]%'
Problems with my data were the cases above. So after fixing them, I created a bigint variable and set the values of the varchar column to the integer column I created.
alter table data add int_col1 bigint
update data set int_col1 = CAST(col1 AS VARCHAR)
This worked for me, hope you find it useful as well.
I have a stored procedure that I have scaled down considerably for the purpose of this question but in essence the issue I need assistance with is this.
If a row in table xyz is updated I need the ID's to be appended to each other and output back to the calling application. The update works as expected, the problem is in the manner in which I am building the output #IPV_ID_Found (see commented section at the bottom of the code).
#IPV_Status varchar (50),
#IPV_ID_Found varchar(500) = 'A' OUTPUT
IF (#IPV_Status ='closed')
BEGIN
UPDATE TEST_TBL
SET
Status = 'xyz',
WHERE
ID = #IPV_ID
-- this works for one ID
SELECT #IPV_ID_Found = (CAST(#IPV_ID AS VARCHAR(500)))
-- this does not work for multiple IDs
SELECT #IPV_ID_Found = #IPV_ID_Found + (CAST(#IPV_ID AS VARCHAR(500))) + ','
-- neither does this
SET #IPV_ID_Found = #IPV_ID_Found + (CAST(#IPV_ID AS VARCHAR(500))) + ','
SELECT #IPV_ID_Found
END
See the changes:
-- this does not work for multiple IDs
SELECT #IPV_ID_Found = #IPV_ID_Found + ',' + (CAST(#IPV_ID AS VARCHAR(500)))
then you will get a concatenated list of values, like 1,2,3,4,5
BUT if you need to return a recordset, then you need not OUTPUT parameter, use the table variable instead:
declare #IPV_ID_Found table(Item varchar(500))
IF (#IPV_Status ='closed')
BEGIN
UPDATE TEST_TBL
SET
Status = 'xyz',
WHERE
ID = #IPV_ID
-- this works for one ID
insert #IPV_ID_Found
VALUES (CAST(#IPV_ID AS VARCHAR(500)))
-- this does not work for multiple IDs
INSERT #IPV_ID_Found
VALUES (CAST(#IPV_ID AS VARCHAR(500)))
SELECT Item FROM #IPV_ID_Found
END