I have a table say employee(id, name, email, is_email_verified, created_date). I need to send an email everyday for the users which has been created and email is not verified. I need to create a service which can be deployed on two boxes. It will read the user table and get the user id and send the mail for verification. Since, there are two boxes and each will read the same number of users and hence will send the two mails.
How, can we make this service so as user ids will get distributed among the servers (if I have n servers).
There are few ways to do that.
The most straightforward & reliable:
Create a job task that read all user ID from the table and queue them to a task queue - 1 task per user ID.
Pull task from queue using as many servers as you want (1-N).
Not requiring queues (not fault tolerant):
Choose a hash function that would decide which user IDs should server process. For example 1st server processes odd IDs and 2nd even IDs, etc. Or something like:
def should_i_send(user_id, server_id):
return user_id % server_id == 0
Related
It is my understanding that modern web frameworks use a persistent database connection. I am wondering what impact this has on SQLite's last_insert_rowid().
I use last_insert_rowid() to get the unique ID for a previous insertion. Suppose I have a Users table currently with 22 records. I then do:
insert into Users (UserName,RealName) values ('jdoe','John Doe')
select last_insert_rowid()
The last_insert_rowid() in this case returns 23. I can use this ID value to explicitly access this newly inserted record.
Now, according to the SQLite documentation, last_insert_rowid() only applies to the current connection. So if other users are accessing the database with other connections, all is well. But for shared / persistent database connections, is there a problem?
Imagine this scenario:
(1) User A does:
insert into Users (UserName,RealName) values ('jdoe','John Doe')
(2) User B does:
insert into Tools (ToolName) values ('Widget')
(3) User A does:
last_insert_rowid()
In this case, two users are both doing insert operations. Although User A's steps 1 and 3 are supposed to be sequential, they are not. User B's insert actually weasels its way in between. So, in this scenario, does User A's step 3 last_insert_rowid() return the last row insert ID from step 1 or step 2? And if 2, how do you avoid this problem?
Situation:
I developed a Database that heavily depends on a lot of triggers to ensure data consistency (To fulfill the CFR 21 Part 11).
There are 2 tables to store users: Login & User (in favor of the Sql servers User management). Users contains static login data for External authentication and X1 are containing login data for our system (Fake User). A person have to login into our application using one of the Login data (Real User). When the system must communicate with an external system (Active Directory or FileSystem or Database) the corresponding User(Fake User) will be impersonated.
For Clarification will use X1(Real User) for Login and X2(Fake User) for User.
When the X2 is connected to the database the trigger does not know who the X1 is.
Problem:
The Trigger will log each action done by the X2 and write it the Log. As the server only knows that X2 is connected to the database it is not aware of the Real user X1.
The Customer whats to also see who the RealUser X1 was. As this information is lost when connected to the Database i have this Problem now.
Solution Attempts:
Create a StoredProcedure that takes the RealUser x1 as an Argument. This is not possible as old software relies on the current to Insert and Update the Table.
Add Additional Data the Insert/Update statement (is that possible i did not found anything related to that). The data will only used by the Trigger and should no be written to the Table.
Add the info about the current X1 user to the Connection or to a general Connection based store (does something like this exist in SqlServer?)
Use the ApplicationName property for "faking" the Username. But this makes me feel like wearing a monocle. There must be a better way
Thanks in Advance
You can use CONTEXT_INFO to associate extra data with the session that can be accessed in the trigger (like, say, the original user identity):
DECLARE #UID VARBINARY(128) = CONVERT(VARBINARY(128), N'User1')
SET CONTEXT_INFO #UID
-- Later, at the Hall of Justice
SELECT CONVERT(NVARCHAR(64), CONTEXT_INFO())
Of course, you could use a different way of identifying the user than a Unicode string, as long as it eventually fits in a VARBINARY(128).
For this to work you must consistently set the information before executing any command (the context info will be reset when the connection is closed). There are also security concerns -- SET CONTEXT_INFO is not a privileged statement and could be executed by anyone, so you must be sure you control all code in the database lest the audit information is falsified.
I need some guidance from anyone who has deployed a real-world, in-production application that uses the Sql Server Service Broker external activation mechanism (via the Service Broker External Activator from the Feature Pack).
Current mindset:
My specs are rather simple (or at least I think so), so I'm thinking of the following basic flow:
order-like entity gets inserted into a Table_Orders with state "confirmed"
SP_BeginOrder gets executed and does the following:
begins a TRANSACTION
starts a DIALOG from Service_HandleOrderState to Service_PreprocessOrder
stores the conversation handle (from now on PreprocessingHandle) in a specific column of the Orders table
sends a MESSAGE of type Message_PreprocessOrder containing the order id using PreprocessingHandle
ends the TRANSACTION
Note that I'm not ending the conversation, I don't want "fire-and-forget"
event notification on Queue_PreprocessOrder activates an instance of PreprocessOrder.exe (max concurrent of 1) which does the following:
begins a SqlTransaction
receives top 1 MESSAGE from Queue_PreprocessOrder
if message type is Message_PreprocessOrder (format XML):
sets the order state to "preprocessing" in Table_Orders using the order id in the message body
loads n collections of data of which computes an n-ary Carthesian product (via Linq, AFAIK this is not possible in T-SQL) to determine the order items collection
inserts the order items rows into a Table_OrderItems
sends a MESSAGE of type Message_PreprocessingDone, containing the same order id, using PreprocessingHandle
ends the conversation pertaining to PreprocessingHandle
commits the SqlTransaction
exits with Environment.Exit(0)
internal activation on Queue_HandleOrderState executes a SP (max concurrent of 1) that:
begins a TRANSACTION
receives top 1 MESSAGE from Queue_InitiatePreprocessOrder
if message type is Message_PreprocessingDone:
sets the order state to "processing" in Table_Orders using the order id in the message body
starts a DIALOG from Service_HandleOrderState to Service_ProcessOrderItem
stores the conversation handle (from now on ProcessOrderItemsHandle) in a specific column of Table_Orders
creates a cursor for rows in Table_OrderItems for current order id and for each row:
sends a MESSAGE of type Message_ProcessOrderItem, containing the order item id, using ProcessOrderItemsHandle
if message type is Message_ProcessingDone:
sets the order state to "processed" in Table_Orders using the order id in the message body
if message type is http://schemas.microsoft.com/SQL/ServiceBroker/EndDialog (END DIALOG):
ends the conversation pertaining to conversation handle of the message
ends the TRANSACTION
event notification on Queue_ProcessOrderItem activates an instance of ProcessOrderItem.exe (max concurrent of 1) which does the following:
begins a SqlTransaction
receives top 1 MESSAGE from Queue_ProcessOrderItem
if message type is Message_ProcessOrderItem (format XML):
sets the order item state to "processing" in Table_OrdersItems using the order item id in the message body, then:
loads a collection of order item parameters
makes a HttpRequest to a URL using the parameters
stores the HttpResponse as a PDF on filesystem
if any errors occurred in above substeps, sets the order item state to "error", otherwise "ok"
performs a lookup in the Table_OrdersItems to determine if all order items are processed (state is "ok" or "error")
if all order items are processed:
sends a MESSAGE of type Message_ProcessingDone, containing the order id, using ProcessOrderItemsHandle
ends the conversation pertaining to ProcessOrderItemsHandle
commits the SqlTransaction
exits with Environment.Exit(0)
Notes:
specs specify MSSQL compatibility 2005 through 2012, so:
no CONVERSATION GROUPS
no CONVERSATION PRIORITY
no POISON_MESSAGE_HANDLING ( STATUS = OFF )
I am striving to achieve overall flow integrity and continuity, not speed
given that tables and SPs reside in DB1 whilst Service Broker objects (messages, contracts, queues, services) reside in DB2, DB2 is SET TRUSTWORTHY
Questions:
Are there any major design flaws in the described architecture ?
Order completion state tracking doesn't seem right. Is there a better method ? Maybe using QUEUE RETENTION ?
My intuition tells me that in no case whatsoever should the activated external exe terminate with an exit code other than 0, so there should be try{..}catch(Exception e){..} finally{ Environment.Exit(0) } in Main. Is this assumption correct ?
How would you organize error handling in DB code ? Is an error log table enough?
How would you organize error handling in external exe C# code ? Same error logging
table ?
I've seen the SQL Server Service Broker Product Samples, but the Service Broker Interface seems overkill for my seemingly simpler case. Any alternatives for a simpler Service Broker object model ?
Any cross-version "portable" admin tool for Service Broker capable of at least draining poison messages ?
Have you any decent code samples for any of the above ?
Q: Are there any major design flaws in the described architecture ?
A: Couple of minor perks:
- waiting for an HTTP request to complete while holding open a transaction is bad. You can't achieve transactional consistency between a database and HTTP anyway, so don't risk to have a transaction stretch for minutes when the HTTP is slow. The typical pattern is to {begin tran/receive/begin conversation timer/commit} then issue the HTTP call w/o any DB xact. If the HTTP call succeeds then {begin xact/send response/end conversation/commit}. If the HTTP fails (or client crashes) then let the conversation time activate you again. You'll get a timer message (no body), you need to pick up the item id associated with the handle from your table(s).
Q: Order completion state tracking doesn't seem right. Is there a better method ? Maybe using QUEUE RETENTION ?
A: My one critique of your state tracking is the dependency on scanning the order items to determine that the current processed one is the last one (5.3.4). For example you could add the information that this is the 'last' item to be processed in the item state so you know, when processing it, that you need to report the completion. RETENTION is only useful in debugging or when you have logic that require to run 'logical rollback' and to compensating actions on conversation error.
Q: My intuition tells me that in no case whatsoever should the activated external exe terminate with an exit code other than 0, so there should be try{..}catch(Exception e){..} finally{ Environment.Exit(0) } in Main. Is this assumption correct ?
A: The most important thing is for the activated process to issue a RECEIVE statement on the queue. If it fails to do so the queue monitor may enter the notified state forever. Exit code is, if I remember correctly, irrelevant. As with any background process is important to catch and log exceptions, otherwise you'll never even know it has a problem when it start failing. In addition to disciplined try/catch blocks, Hookup Application.ThreadException for UI apps and AppDomain.UnhandledException for both UI and non-UI apps.
Q: How would you organize error handling in DB code ? Is an error log table enough?
A: I will follow up later on this. Error log table is sufficient imho.
Q: How would you organize error handling in external exe C# code ? Same error logging table ?
A: I created bugcollect.com exactly because I had to handle such problems with my own apps. The problem is more than logging, you also want some aggregation and analysis (at least detect duplicate reports) and suppress floods of errors from some deployment config mishap 'on the field'. Truth be told nowadays there are more options, eg. exceptron.com. And of course I think FogBugs also has logging capabilities.
Q: I've seen the SQL Server Service Broker Product Samples, but the Service Broker Interface seems overkill for my seemingly simpler case. Any alternatives for a simpler Service Broker object model ?
finally, an easy question: Yes, it is overkill. There is no simple model.
Q: Any cross-version "portable" admin tool for Service Broker capable of at least draining poison messages ?
A: The problem with poison messages is that the definition of poison message changes with your code: the poison message is whatever message breaks the current guards set in place to detect it.
Q: Have you any decent code samples for any of the above ?
A: No
One more point: try to avoid any reference from DB1 to DB2 (eg. 4.3.4 is activated in DB1 and reads the items table from DB2). This creates cross DB dependencies which break when a) one DB is offline (eg. for maintenance) or overloaded or b) you add database mirroring for HA/DR and one DB fails over. Try to make the code to work even if DB1 and DB2 are on different machines (and no linked servers). If necessary, add more info to the messages payload. And if you architect it that way that DB2 can be on a different machine and even multiple DB2 machines can exists to scale out the HTTP/PDF writing work.
And finally: this design will be very slow. I'm talking low tens messages per second slow, with so many dialogs/messages involved and everything with max_queue_readers 1. This may or may not be acceptable for you.
I'm working on a project in dead ASP (I know :( )
Anyway it is working with a kdb+ database which is major overkill but not my call. Therefore to do inserts etc we're having to write special functions so they can be handled.
Anyway we've hit a theoretical problem and I'm a bit unsure how it should be dealt with in this case.
So basically you register a company, when you submit validation will occur and the page will be processed, inserting new values to the appropriate tables. Now at this stage I want to pull ID's from the tables and use them in the session for further registration screens. The user will never add a specific ID of course so it needs to be pulled from the database.
But how can this be done? I'm particularly concerned with 2 user's simultaneously registering, how can I ensure the correct ID is passed back to the correct session?
Thank you for any help you can provide.
Instead of having the ID set at the point of insert, is it possible for you to "grab" an ID value before hand, and then use that value throughout the process?
So:
Start the registration.
System connects to the database, creates an ID (perhaps from an ID table) and Stores in ASP Session.
Company registers.
You validate and insert data into DB (including the ID session)
The things you put in the Session(...) collection is only visible to that session (i.e. the session is used only by the browser windows on one computer). The session is identified by a GUID value that is stored in a cookie on the client machine. It is "safe" to store your IDs there (other users won't be able to read them easily) .
either your id can include date and time - so it will be example - id31032012200312 - but if you still think that 2 people can register at the same type then I would use recordset locks liek the ones here - http://www.w3schools.com/ado/prop_rs_locktype.asp
To crea ids like above in asp you do - replace(date(),"/","") ' and then same with time with ":"
Thanks
My project needs to have a number of administrators, out of which only one will have super-admin privileges.
What is the best way to represent this in the database?
There are a few ways to do this.
Number 1: Have a column on your administrator (or user) table called IsSuperAdmin and have an insert/update trigger to ensure that only one has it set at any given time.
Number 2: Have a TimestampWhenMadeSuperAdmin column in your table. Then, in your query to figure out who it is, use something like:
select user_id from users
where TimestampWhenMadeSuperAdmin is not null
order by TimestampWhenMadeSuperAdmin desc
fetch first 1 row only;
Number 3/4: Put the SuperAdmin user ID into a separate table, using either the trigger or last-person-made-has-the-power approach from numbers 1 or 2.
Personally, I like number 2 since it gives you what you need without unnecessary triggers, and there's an audit trail as to who had the power at any given time (though not a complete audit trail since it will only store the most recent time that someone was made a SuperAdmin).
The trouble with number 1 is what to do if you just clear the current SuperAdmin. Either you have to give the power to someone else, or nobody has it. n other words, you can get yourself into a situation where there is no SuperAdmin. And number 3 and 4 just complicate things with an extra table.
Use a roles/groups approach. You have a table containing all the possible roles, and then you have an intersect table containing the key of the user and the key of the role they belong to (there can be multiple entries per user as each user could have several roles (or belong to several groups)).
Also, don't call them super admin - just admin is fine, call the rest power user or something similar.
Simple, yet effective: UserId = 1. Your application will always know it is the SuperUser.