I am using quartz.net for job scheduling. I have a table in my database JobQueue. Quartz.net reads from this JobQueue and executes the jobs. Everything works fine. Now there is a new requirement that i need to extend the job scheduling for more than one client. This means Quartz.net has to be configured to read from more than one database. How to handle this scenario? I have one service written using Quartz.net which should schedule jobs for multiple clients by reading from their respective databases. Any help will be greatly appreciated. Thanks.
How sure how it will scale up, but it is possible to create multiple Appdomains and have a different scheduler hosted inside each appdomain.
If you are reading the quartz settings from the app.config file, then you would probably need to change some settings such as the connectionstring at runtime.
Related
With the introduction of the new 'swarm mode' with Docker 1.12, we've been trying to migrate our application on containers and make use of the swarm mode's orchestration & clusters.
Our application requires some initial database scripts to be run for it to start.
We're not packaging the database inside our dockerized application so that it could follow a stateless microservice architecture and multiple containers would eventually talk to a single (at the moment) database instance.
While creating the service, we cannot use --replicas with the create service command as multiple instances would try and create tables on a single database and fail. Although our scripts would check if the database has been set-up and skip the creation but since all containers start simultaneously, it could not be used.
We couldn't find any wait-for kind of mechanism that we could leverage with dockers for this issue. It would have been good if we could only start the second container when the first one had created the database (and exposed the ports) but how can we configure inter-container communication for this?
Alternatively, can tools like flywaydb help in some way?
How should this be used in production?
From the Flyway FAQ:
Can multiple nodes migrate in parallel?
Yes! Flyway uses the locking technology of your database to coordinate multiple nodes. This ensures that even if even multiple instances of your application attempt to migrate the database at the same time, it still works. Cluster configurations are fully supported.
There is no easy way to coordinate this among containers. It basically requires a distributed lock solution. The first container that gets the lock could create db, while, other containers that not get the lock need to wait.
In AWS, you could leverage DynamoDB for it. DynamoDB supports conditional update. The container first tries to create the lock key in DynamoDB with "attribute_not_exists(yourKey)". The first creation will succeed and other creations will be rejected. The first container needs to create another key in DynamoDB to indicate the db is ready. Other containers simply waits till the ready key is created.
Or you could do it in your service deployment script. The script could create the service with 1 replica. Then keep checking if db is created. If yes, scale the service, such as docker service update yourservicce --replicas 5.
I have a database hosted on Azure. I have an MVC 4 website were users log in and interact with the database. I need something that I can use to traverse through a table in my database, check for certain conditions and then make the required changed to my database. What framework or coding language could I use to achieve this? My hope is that I could have this script run continually or at certain time intervals.
There are multiple ways of achieving this and they all depend on your need for scale/resources/etc of this script.
You can code the job in whatever language (including batch if you utilize sqlcmd or some such)
You can schedule the job to run anywhere you want, including a machine in your office, data center, etc.
You can utilize newly released Aditi's job scheduler (it's free, called Scheduler, and is available in Azure app store)
I am creating a system where users can setup mailings to go out at specific times. Before I being I wanted to get some advice. First, is there already a .Net component that will handle scheduling jobs (either running another application or calling a URL) that will do what I am suggesting (Open Source would be cool)? If there isn’t, is it better to schedule a job in SQL and run some sort of script, create a .Net service that will look at an xml file or db for schedules, or have an application create scheduled tasks? There could be a ton of tasks, so I am thinking creating scheduled tasks or SQL jobs might not be a good idea.
Here may be a typical scenario; a user wants to send a newsletter to their clients. The user creates the newsletter on a Saturday, but doesn’t want it to go out until Monday. The user wants that same e-mail to go out every Monday for a month.
Thanks for looking!
Check out Quartz.NET
Quartz.NET is a full-featured, open
source job scheduling system that can
be used from smallest apps to large
scale enterprise systems.
If you want to use the readily available services in Windows itself, check out this article A New Task Scheduler Task Library on CodeProject on how to create scheuled tasks in Windows from your C# application.
You probably have more flexibility and power if you use C# and scheduled tasks in Windows, rather than limiting yourself to what can be done in SQL Server. SQL Server Agent Jobs are great - for database specific stuff, mostly - maintenance plans and so forth.
You can build your own windows service that schedules and executes jobs. Be sure to make good abstractions. In a similar project, I have used an abstraction where scheduling items are abstracted as Jobs composed of tasks. For example, sending newsletter may be a job whereas sending newsletter to each subscriber can be considered as a task. Then you need to run the job and tasks in defined threading models preferably using Threadpool threads or Task Parallel Library. Be sure to use asynchronous API for IO whenever possible. Also separate your scheduling logic from the abstractions. so that the scheduling logic can execute arbitrary types of jobs and its inclusive tasks.
I'm in the process of testing an application and it's database and for this I want to restart my testing each time completely clean. This application loads a large amount of data from Twitter. Therefore, before I start, I delete all data from the database and kill any processes from my web account associated with this application. When I try to then load my application, I get the following error:
[Macromedia][SequeLink JDBC Driver][ODBC Socket][Microsoft][SQL Native Client]Communication link failure
I would assume this has something to do with me killing all the related processes in the DB. After some amount of time I am able to run queries again.
Does this have something to do with the connections setup information in Coldfusion Administrator?
Does it just take some time to reset the connection? Is there any way to get around this?
Is there a better way to start fresh and clean when testing the loading?
By default, ColdFusion pools connection threads. I would guess, based on your comment to Stephen Moretti, that you are killing a connection that CF expects to still be live. That said, I've never had problems killing long DB threads, so this is pure speculation.
I'm not sure what killing these threads gets you, as far as testing goes. Once the page has stopped processing, open DB connections should not push or pull additional data.
I suspect that the error is actually related to how you are "cleaning up", particularly when you say "kill all related processes". By this I'm guessing you go into task manager and actually kill the processes.
I'm also guessing that if you're using SQL Server, you're on windows.
Rather than killing processes, cleanly stop the services associated with your application. Go into the Services Control Panel :
Stop your IIS or Apache Service.
Stop your ColdFusion Server instance service.
In terms of your database:
- Create a script for creating your database schema, tables, views, users and permissions and any default data entries
- drop your schema
- restart the sql server services if you want to be sure you've created any cached data out.
- run the script to create a blank of your database.
You could at this point actually create a database back up and just restore this, but its always handy to have the scripts to run on servers if you don't want to restore a backup.
After this start your coldfusion and iis/apache services.
If I want to conduct some database operations on a scheduled basis, I could:
Use SQL Server Agent (if SQL Server) to periodically call the stored procedure and/or execute the T-SQL
Run some external process (scheduled by the operating system's task scheduler for example) which executes the database operation
etc.
Two questions:
What are some other means of accomplishing this
What decision criteria should one use to decide the best approach?
Thank you.
Another possibility is to have a queue of tasks somewhere, and when applications that otherwise use the database perform some operation, they also do some tasks out of the queue. Wikipedia does something like this with its job queue. The scheduling isn't as certain as with the other methods, but you can e.g. put off doing housekeeping work when your server happens to be heavily loaded.
Edit:
It's not necessarily better or worse than the other techniques. It's suitable for tasks that do not have to be performed by any specific deadline, but should be done "every now and then", or "soon, but not necessarily right now".
Advantages
You don't need to write a separate application or set up SQL Server Agent.
You can use any criteria you can program to decide whether to run a task or not: immediately, once a certain time has passed, or only if the server is not under heavy load.
If the scheduled tasks are ones like optimising indices, then you can do them less frequently when they are less necessary (e.g. when updates are rare), and more frequently when updates are common.
Disadvantages
You might need to modify multiple applications to cooperate correctly.
You need to ensure that the queue doesn't build up too much.
You can't reliably ensure that a task runs before a certain time.
You might have long periods where you get no requests (e.g. at night) where deferred/scheduled tasks could get done, but don't. You could combine it with one of the other ideas, having a special program that just does the jobs in the queue, but you could just not bother with the queue at all.
You can't really rely on external processes. All 'OS' based solutions I've seen failed to deliver in the real world: a database is way more than just the data, primarily because of the backup/restore strategy, the high availability strategy, the disaster recoverability strategy and all the other 'ities' you pay for in your SQL Server license. An OS scheduler based will be an external component completely unaware and unintegrated with any of them. Ie. you cannot back/restore your schedule with your data, it will not fail over with your database, you cannot ship it to a remote disaster recovery site through your SQL data shipping channel.
If you have Agent (ie. not Express edition) then use Agent. Has a long history of use and the know how around it is significant. The only problem with Agent is its dependence on msdb that makes it disconnect from the application database and thus does not play well with mirroring based availability and recoverability solutions.
For Express editions (ie. no Agent) the best option is to roll your own scheduler based on conversation timers (at least in SQL 2k5 and forward). You use conversations to chedule yourself messages at the desired moment and rely on activated procedures to run the tasks. They are transactional and integrated with your database, so you can rely on them being there after a restore and after a mirroring or clustering fail over. Unfortunately the know how around how to use them is fairly skim, I have several articles about the subject on my site rusanu.com. I've seen systems replicating a fair amount of Agent API on Express relying entirely on conversation timers.
I generally go with the operating systems scheduling method (task scheduler for Windows, cron for Unix).
I deal with multiple database platforms (SQL Server, Oracle, Informix) and want to keep the task scheduling as generic as possible.
Also, in our production environment we have to get a DBA involved for any troubleshooting / restarting of jobs that are running in the database. We have better access to the application servers with the scheduled tasks on them.
I think the best approach for the decision criteria is what the job is. If it's a completely internal SQL Server task or set of tasks that does not relate to the outside world, I would say a SQL Job is the best bet. If on the other hand, you are retrieving data and then doing something with it that is inherently outside SQL Server, very difficult to do in T-SQL or time consuming, perhaps the external service is the best bet.
I'd go with SQL Server Agent. It's well integrated with SQL Server; various SQL Server features use Agent (Log Shipping, for instance). You can create an Agent job to run one or more SSIS packages, for instance.
It's also integrated with operator notification, and can be scripted, or else executed through SMO.