I have some records in my table like:
Prefix Column
...
54664300
54664301
54664302
54664303
546643040
546643041
546643042
546643043
546643044
546643045
546643046
546643047
546643048
546643049
54664305
54664306
54664307
54664308
54664309
...
54665100
54665101
54665102
54665103
54665105
54665106
54665109
...
If the 0-9 series are complete for a certain prefix, I simplify them. The following records above will become:
Prefix Column
...
54664300
54664301
54664302
54664303
54664304
54664305
54664306
54664307
54664308
54664309
...
54665100
54665101
54665102
54665103
54665105
54665106
54665109
...
And I can further simplify them because the 0-9 series has been completed. Which will result to:
Prefix Column
...
5466430
...
54665100
54665101
54665102
54665103
54665105
54665106
54665109
...
But some records will not get simplified because they are incomplete.
I achieved this process using WHILE loop:
DECLARE #length INT = (SELECT MAX(LEN(CardPrefix)) FROM #Card) - 1;
IF OBJECT_ID('tempdb..#GroupedCard') IS NOT NULL DROP TABLE #GroupedCard;
CREATE TABLE #GroupedCard (CardPrefix NVARCHAR(20));
WHILE (#length > 6) -- minimum 6 digits only
BEGIN
TRUNCATE TABLE #GroupedCard;
INSERT INTO #GroupedCard
SELECT LEFT(CardPrefix, #length) CardPrefix
FROM #Card
WHERE LEN(CardPrefix) > #length
GROUP BY LEFT(CardPrefix, #length)
HAVING SUM(CAST(ISNULL(RIGHT(CardPrefix, 1), 0) AS INT)) = 45; --sum of 0 to 9 is 45
IF NOT EXISTS (SELECT 1 FROM #GroupedCard) BREAK;
DELETE #Card
FROM #Card C
INNER JOIN #GroupedCard GC ON GC.CardPrefix = LEFT(C.CardPrefix, 9);
INSERT INTO #Card
SELECT CardPrefix FROM #GroupedCard;
END
I am just checking if there is a more efficient way to do this because our records are getting huge in our live environment. There will also be a need to execute this process more frequently.
Reading between the lines, but therefore are you not after...
SELECT Prefix
FROM dbo.YourTable
WHERE Prefix <= 99999999 --If this is alphanumeric, use LEN, but this will come at a cost
UNION ALL
SELECT DISTINCT LEFT(Prefix,8)
FROM dbo.YourTable
WHERE Prefix > 99999999; --If this is alphanumeric, use LEN, but this will come at a cost
Related
I currently have a stored procedure in MSSQL where I execute a SELECT-statement multiple times based on the variables I give the stored procedure. The stored procedure counts how many results are going to be returned for every filter a user can enable.
The stored procedure isn't the issue, I transformed the select statement from te stored procedure to a regular select statement which looks like:
DECLARE #contentRootId int = 900589
DECLARE #RealtorIdList varchar(2000) = ';880;884;1000;881;885;'
DECLARE #publishSoldOrRentedSinceDate int = 8
DECLARE #isForSale BIT= 1
DECLARE #isForRent BIT= 0
DECLARE #isResidential BIT= 1
--...(another 55 variables)...
--Table to be returned
DECLARE #resultTable TABLE
(
variableName varchar(100),
[value] varchar(200)
)
-- Create table based of inputvariable. Example: turns ';18;118;' to a table containing two ints 18 AND 118
DECLARE #RealtorIdTable table(RealtorId int)
INSERT INTO #RealtorIdTable SELECT * FROM dbo.Split(#RealtorIdList,';') option (maxrecursion 150)
INSERT INTO #resultTable ([value], variableName)
SELECT [Value], VariableName FROM(
Select count(*) as TotalCount,
ISNULL(SUM(CASE WHEN reps.ForRecreation = 1 THEN 1 else 0 end), 0) as ForRecreation,
ISNULL(SUM(CASE WHEN reps.IsQualifiedForSeniors = 1 THEN 1 else 0 end), 0) as IsQualifiedForSeniors,
--...(A whole bunch more SUM(CASE)...
FROM TABLE1 reps
LEFT JOIN temp t on
t.ContentRootID = #contentRootId
AND t.RealEstatePropertyID = reps.ID
WHERE
(EXISTS(select 1 from #RealtorIdTable where RealtorId = reps.RealtorID))
AND (#SelectedGroupIds IS NULL OR EXISTS(select 1 from #SelectedGroupIdtable where GroupId = t.RealEstatePropertyGroupID))
AND (ISNULL(reps.IsForSale,0) = ISNULL(#isForSale,0))
AND (ISNULL(reps.IsForRent, 0) = ISNULL(#isForRent,0))
AND (ISNULL(reps.IsResidential, 0) = ISNULL(#isResidential,0))
AND (ISNULL(reps.IsCommercial, 0) = ISNULL(#isCommercial,0))
AND (ISNULL(reps.IsInvestment, 0) = ISNULL(#isInvestment,0))
AND (ISNULL(reps.IsAgricultural, 0) = ISNULL(#isAgricultural,0))
--...(Around 50 more of these WHERE-statements)...
) as tbl
UNPIVOT (
[Value]
FOR [VariableName] IN(
[TotalCount],
[ForRecreation],
[IsQualifiedForSeniors],
--...(All the other things i selected in above query)...
)
) as d
select * from #resultTable
The combination of a Realtor- and contentID gives me a set default set of X amount of records. When I choose a Combination which gives me ~4600 records, the execution time is around 250ms. When I execute the sattement with a combination that gives me ~600 record, the execution time is about 20ms.
I would like to know why this is happening. I tried removing all SUM(CASE in the select, I tried removing almost everything from the WHERE-clause, and I tried removing the JOIN. But I keep seeing the huge difference between the resultset of 4600 and 600.
Table variables can perform worse when the number of records is large. Consider using a temporary table instead. See When should I use a table variable vs temporary table in sql server?
Also, consider replacing the UNPIVOT by alternative SQL code. Writing your own TSQL code will give you more control and even increase performance. See for example PIVOT, UNPIVOT and performance
This question already has answers here:
TSQL Generate 5 character length string, all digits [0-9] that doesn't already exist in database
(6 answers)
Closed 7 years ago.
i want to create a coupon code generator by using SQL database but i don't know how to generate 1000 of random number without repeating them. so can someone help me it's important. thanks
Records in a relational database tables are unordered by nature.
therefor, you can simply create a table that has all the values between #First and #Last (0 and 9999 in your case), and then use a random order by when selecting from that table. you can also use a simple int in the database table and just format it when you select the data from the table.
Since my main database is Sql server, and I have no experience with sqlite, I will use Sql Server syntax in my code example, and leave it up to you to find the sqllite equivalent.
First, create the table:
CREATE TABLE Tbl
(
IntValue int PRIMARY KEY,
IsUsed bit NOT NULL DEFAULT 0
)
Then, populate it with numbers between 0 and 9999:
;With CTE AS (
SELECT 0 As IntValue
UNION ALL
SELECT IntValue + 1
FROM CTE
WHERE IntValue + 1 < 10000
)
INSERT INTO Tbl (IntValue)
SELECT IntValue
FROM CTE
OPTION(MAXRECURSION 0)
Then, you want to select multiple values each time, so I would write a stored procedure like this:
CREATE PROCEDURE stp_GetCouponCodes
(
#Number int = 5 -- or whatever number is default
)
AS
BEGIN
DECLARE #UsedValues AS TABLE
(
IntValue int
)
BEGIN TRY
BEGIN TRANSACTION
INSERT INTO #UsedValues
SELECT TOP(#Number) IntValue
FROM Tbl
WHERE IsUsed = 0
ORDER BY NEWID()
UPDATE Tbl
SET IsUsed = 1
FROM Tbl
INNER JOIN
#UsedValues uv ON(Tbl.IntValue = uv.IntValue)
SELECT RIGHT('00000' + CAST(IntValue as varchar), 5)
FROM #UsedValues
COMMIT TRANSACTION
END TRY
BEGIN CATCH
IF ##TRANCOUNT > 0
ROLLBACK TRANSACTION
END CATCH
END
Then, when ever you want to generate coupons, simply execute the stored procedure with the number of coupons you want:
EXEC stp_GetCouponCodes 10;
See working fiddle example here.
The code below uses a quick method to generate 100 random 5-character strings based on the alphabet provided. You'll still need to perform duplicate checking, but this should get you started.
DECLARE #Quantity INT = 1000
DECLARE #Alphabet VARCHAR(100) = '0123456789'
DECLARE #Length INT = LEN(#Alphabet)
DECLARE #Top INT = SQRT(#Quantity) + 1
;WITH CTE AS (
SELECT TOP (#Top) *
FROM sys.objects
)
SELECT TOP (#Quantity)
SUBSTRING(#Alphabet, ABS(CHECKSUM(NEWID())) % #Length + 1, 1)
+ SUBSTRING(#Alphabet, ABS(CHECKSUM(NEWID())) % #Length + 1, 1)
+ SUBSTRING(#Alphabet, ABS(CHECKSUM(NEWID())) % #Length + 1, 1)
+ SUBSTRING(#Alphabet, ABS(CHECKSUM(NEWID())) % #Length + 1, 1)
+ SUBSTRING(#Alphabet, ABS(CHECKSUM(NEWID())) % #Length + 1, 1)
AS [Code]
FROM CTE X
CROSS JOIN CTE Y
My Table structure is
id type no amount
1 type1 a1 1000
2 type1 a2 2000
3 type2 b1 3000
4 type3 c1 4000
5 type1 a3 5000
6 type2 b2 6000
7 type2 b3 7000
8 type3 c2 8000
now i wants to increment the no field data based on the type.
for example for type1 the next no is a4
and
for numeric only I am using the following code
SELECT ISNULL(Max(No),0)+1 AS No FROM table
but how to do it for with Alphabets in SQL Server 2005
Assuming that prefixes are of single character length, you may try following:
;with cte as (
select type, typePrefix = left(no, 1), typeNum = right(no, len(no) - 1)
from TableName
)
select typePrefix + cast(isnull(max(typeNum), 0) + 1 as varchar(10))
from cte
where type = 'type1'
group by typePrefix
But it will not work if you try to generate next no for a type which is not in table (e.g. 'type4'). To allow it, you may need a separate table, where prefix for each type is specified:
create table TypePrefixes (type varchar(50), prefix varchar(10))
insert into TypePrefixes values ('type1', 'a')
insert into TypePrefixes values ('type2', 'b')
insert into TypePrefixes values ('type3', 'c')
insert into TypePrefixes values ('another_type', 'd')
--etc.
In this case, statement to get next no will look as:
select tp.prefix + cast(isnull(max(cast(right(t.no, len(t.no) - len(tp.prefix)) as int)), 0) + 1 as varchar(20))
from TableName t
right join TypePrefixes tp on tp.type = t.type
where tp.type = 'type4'
group by tp.prefix
Also, you may just wish to calculate no for each record on the fly, like:
;with cte as (
select *,
typeNum = row_number() over (partition by type order by id),
typePrefix = char(dense_rank() over (order by type) + ascii('a') - 1)
from TableName
)
select *, No2 = typePrefix + cast(typeNum as varchar(10))
from cte
However, the latter is limited in number of distinct types in your table, which should not exceed 26 (so that we not go beyond 'z').
try something like
SELECT ISNULL(Max(No),0)+1 AS No FROM table group by type
First, you need an UNIQUE index on No column:
CREATE UNIQUE INDEX IUN_MyTable_On
ON MySchema.MyTable(On);
GO
This unique index will prevent duplicate values but, also, will help the query below.
Second, you could use this script to generate the next No for a given letter:
DECLARE #Chr CHAR(1);
SET #Chr='A';
BEGIN TRY
BEGIN TRANSACTION;
DECLARE #LastId INT;
DECLARE #NewNo VARCHAR(...); -- Fill with No's max. length
-- Previous index will help this query
SELECT #LastId=MAX( CONVERT(INT,SUBSTRING(#LastNo,2,8000)) )
FROM MySchema.MyTable x WITH(UPDLOCK) -- It locks the rows to prevent a concurent session to generate the same value (No)
WHERE x.No LIKE #Chr+'%';
SET #NewNo=#Chr+CONVERT(VARCHAR(11),ISNULL(#LastId,0)+1);
-- Do whatever you want with the new value: ex. INSERT
INSERT INTO ... (No,...)
VALUES (#NewNo,...);
COMMIT TRANSACTION;
END TRY
BEGIN CATCH
DECLARE #ErrMsg NVARCHAR(2000);
SET #ErrMsg=ERROR_MESSAGE();
IF ##TRANCOUNT>0
BEGIN
ROLLBACK;
END
RAISERROR(#ErrMsg,16,1);
END CATCH
Note #1: This solution should be safe if this is the only way to generate the new values (#NewNo).
Note #2: If that SELECT query acquires at least 5000 locks then SQL Server will escalate locks at table/partition level.
I have a periodic check of a certain query (which by the way includes multiple tables) to add informational messages to the user if something has changed since the last check (once a day).
I tried to make it work with checksum_agg(binary_checksum(*)), but it does not help, so this question doesn't help much, because I have a following case (oversimplified):
select checksum_agg(binary_checksum(*))
from
(
select 1 as id,
1 as status
union all
select 2 as id,
0 as status
) data
and
select checksum_agg(binary_checksum(*))
from
(
select 1 as id,
0 as status
union all
select 2 as id,
1 as status
) data
Both of the above cases result in the same check-sum, 49, and it is clear that the data has been changed.
This doesn't have to be a simple function or a simple solution, but I need some way to uniquely identify the difference like these in SQL server 2000.
checksum_agg appears to simply add the results of binary_checksum together for all rows. Although each row has changed, the sum of the two checksums has not (i.e. 17+32 = 16+33). This is not really the norm for checking for updates, but the recommendations I can come up with are as follows:
Instead of using checksum_agg, concatenate the checksums into a delimited string, and compare strings, along the lines of SELECT binary_checksum(*) + ',' FROM MyTable FOR XML PATH(''). Much longer string to check and to store, but there will be much less chance of a false positive comparison.
Instead of using the built-in checksum routine, use HASHBYTES to calculate MD5 checksums in 8000 byte blocks, and xor the results together. This will give you a much more resilient checksum, although still not bullet-proof (i.e. it is still possible to get false matches, but very much less likely). I'll paste the HASHBYTES demo code that I wrote below.
The last option, and absolute last resort, is to actually store the table table in XML format, and compare that. This is really the only way you can be absolutely certain of no false matches, but is not scalable and involves storing and comparing large amounts of data.
Every approach, including the one you started with, has pros and cons, with varying degrees of data size and processing requirements against accuracy. Depending on what level of accuracy you require, use the appropriate option. The only way to get 100% accuracy is to store all of the table data.
Alternatively, you can add a date_modified field to each table, which is set to GetDate() using after insert and update triggers. You can do SELECT COUNT(*) FROM #test WHERE date_modified > #date_last_checked. This is a more common way of checking for updates. The downside of this one is that deletions cannot be tracked.
Another approach is to create a modified table, with table_name (VARCHAR) and is_modified (BIT) fields, containing one row for each table you wish to track. Using insert, update and delete triggers, the flag against the relevant table is set to True. When you run your schedule, you check and reset the is_modified flag (in the same transaction) - along the lines of SELECT #is_modified = is_modified, is_modified = 0 FROM tblModified
The following script generates three result sets, each corresponding with the numbered list earlier in this response. I have commented which output correspond with which option, just before the SELECT statement. To see how the output was derived, you can work backwards through the code.
-- Create the test table and populate it
CREATE TABLE #Test (
f1 INT,
f2 INT
)
INSERT INTO #Test VALUES(1, 1)
INSERT INTO #Test VALUES(2, 0)
INSERT INTO #Test VALUES(2, 1)
/*******************
OPTION 1
*******************/
SELECT CAST(binary_checksum(*) AS VARCHAR) + ',' FROM #test FOR XML PATH('')
-- Declaration: Input and output MD5 checksums (#in and #out), input string (#input), and counter (#i)
DECLARE #in VARBINARY(16), #out VARBINARY(16), #input VARCHAR(MAX), #i INT
-- Initialize #input string as the XML dump of the table
-- Use this as your comparison string if you choose to not use the MD5 checksum
SET #input = (SELECT * FROM #Test FOR XML RAW)
/*******************
OPTION 3
*******************/
SELECT #input
-- Initialise counter and output MD5.
SET #i = 1
SET #out = 0x00000000000000000000000000000000
WHILE #i <= LEN(#input)
BEGIN
-- calculate MD5 for this batch
SET #in = HASHBYTES('MD5', SUBSTRING(#input, #i, CASE WHEN LEN(#input) - #i > 8000 THEN 8000 ELSE LEN(#input) - #i END))
-- xor the results with the output
SET #out = CAST(CAST(SUBSTRING(#in, 1, 4) AS INT) ^ CAST(SUBSTRING(#out, 1, 4) AS INT) AS VARBINARY(4)) +
CAST(CAST(SUBSTRING(#in, 5, 4) AS INT) ^ CAST(SUBSTRING(#out, 5, 4) AS INT) AS VARBINARY(4)) +
CAST(CAST(SUBSTRING(#in, 9, 4) AS INT) ^ CAST(SUBSTRING(#out, 9, 4) AS INT) AS VARBINARY(4)) +
CAST(CAST(SUBSTRING(#in, 13, 4) AS INT) ^ CAST(SUBSTRING(#out, 13, 4) AS INT) AS VARBINARY(4))
SET #i = #i + 8000
END
/*******************
OPTION 2
*******************/
SELECT #out
I want to compare a number of values (up to ten) with a function that will return the smallest value of them.
My colleague wrote the function like:
set #smallest = null
if #smallest is null or #date0 < #smallest
begin
set #smallest = #date0
end
if #smallest is null or #date1 < #smallest
begin
set #smallest = #date1
end
... (repeating 10 times)
Beside of that the if statement could be written smarter (the null check can fall away after the first comparison) I was wondering if creating an in-memory indexed table and let the function return me the first value would be more efficient?
Is there any documentation that I could read for this?
creating an in-memory indexed table
There is no point having an index on 10 records. Create a derived table (will sit in memory) as shown below, then run MIN across the table:
select #smallest = MIN(Adate)
from (
select #date0 Adate union all
select #date1 union all
select #date2 union all
-- ....
select #date9) X