My product has an ingestion service written using Java which runs Apache Camel routes. There are multiple ingestion service instances running on different VMs. Ingestion service uses SQL Server 2016 database server. When each route is executed, it creates a job in the database and then each step of job manages the job status till it reaches the completion status.
Requirement is to ensure that Camel routes are executed one after another and no routes run in parallel (i.e., at the same time). How can this be accomplished?
One option is to use a home-grown solution where each route looks at whether there is a job in running status and proceed further only if there is none. This would require polling the database which does not seem to be a good solution.
When each route is executed, it creates a job in the database and then each step of job manages the job status till it reaches the completion status.
I would recommend a central service that manages the ingestion services using rest-api, jmx or something similar. It would provide jobs to these services and track their status. If the manager service needs persistence of its own it can use whatever like log files, json, embed sqlite database, nosql database etc. This would remove the need for ingestion services to know about other ingestion services or their state.
You can look at many CD/CI tools like Jetbrains TeamCity or Jenkins for reference on how they handle jobs with multiple agents/instances.
Related
I have a VB .Net Windows Form application with MS SQL server for database part of it. I need to run a method which essentially depending on some date sensitive data in the database may or may not create a notification email to be sent to one or more recipients. This application may not be used everyday. So ideally I don't want that method to be bound to let say Form Load of the main Form. How can this be achieved?
You probably don't want that logic in your client application. There are three ways that come to mind:
put the logic in the SQL Server and create a job that is scheduled to run every day
create a small utility application and schedule that to once each day
create a windows service that runs all the time and handles these jobs for you
If all the data necessary to make the determination of whether or not to send the notification e-mails is available in the database, and you have access to create a job on the SQL server, I would recommend that route.
However if there are external components that you need in order to make the determinations or to send the e-mails then either approach 2 or 3 will be the way to go. Creating an application and scheduling it to run each day would be easier to implement but a service has the benefit of not requiring an interactive logon session (i.e. doesn't need a user to be actively logged in on the computer) which is preferable on a server.
I have a database hosted on Azure. I have an MVC 4 website were users log in and interact with the database. I need something that I can use to traverse through a table in my database, check for certain conditions and then make the required changed to my database. What framework or coding language could I use to achieve this? My hope is that I could have this script run continually or at certain time intervals.
There are multiple ways of achieving this and they all depend on your need for scale/resources/etc of this script.
You can code the job in whatever language (including batch if you utilize sqlcmd or some such)
You can schedule the job to run anywhere you want, including a machine in your office, data center, etc.
You can utilize newly released Aditi's job scheduler (it's free, called Scheduler, and is available in Azure app store)
We have a Java based web application to show the status update for the jobs started in last one hour. Jobs status information is maintained MS SQL server 2008. We need to capture the new change added to job status in DB and update the model objects cache in web application server. We are capturing the new updates by periodically making a query to DB and finding the diff in job status.
Is there a better way to capture new updates in DB? Are there any ORM tools available to cache and update this kind of real time data updates? Any suggestions please?
No, and this isn't the job of an ORM.
Unfortunately, Java does not have an equivalent of .NET's Reactive extension (Rx).
You could, however, apply these principles by creating an interface (service) in the web application that the job can talk to. But in most cases, the benefits does not worth the cost to implement it.
I am creating a system where users can setup mailings to go out at specific times. Before I being I wanted to get some advice. First, is there already a .Net component that will handle scheduling jobs (either running another application or calling a URL) that will do what I am suggesting (Open Source would be cool)? If there isn’t, is it better to schedule a job in SQL and run some sort of script, create a .Net service that will look at an xml file or db for schedules, or have an application create scheduled tasks? There could be a ton of tasks, so I am thinking creating scheduled tasks or SQL jobs might not be a good idea.
Here may be a typical scenario; a user wants to send a newsletter to their clients. The user creates the newsletter on a Saturday, but doesn’t want it to go out until Monday. The user wants that same e-mail to go out every Monday for a month.
Thanks for looking!
Check out Quartz.NET
Quartz.NET is a full-featured, open
source job scheduling system that can
be used from smallest apps to large
scale enterprise systems.
If you want to use the readily available services in Windows itself, check out this article A New Task Scheduler Task Library on CodeProject on how to create scheuled tasks in Windows from your C# application.
You probably have more flexibility and power if you use C# and scheduled tasks in Windows, rather than limiting yourself to what can be done in SQL Server. SQL Server Agent Jobs are great - for database specific stuff, mostly - maintenance plans and so forth.
You can build your own windows service that schedules and executes jobs. Be sure to make good abstractions. In a similar project, I have used an abstraction where scheduling items are abstracted as Jobs composed of tasks. For example, sending newsletter may be a job whereas sending newsletter to each subscriber can be considered as a task. Then you need to run the job and tasks in defined threading models preferably using Threadpool threads or Task Parallel Library. Be sure to use asynchronous API for IO whenever possible. Also separate your scheduling logic from the abstractions. so that the scheduling logic can execute arbitrary types of jobs and its inclusive tasks.
I'm in the process of testing an application and it's database and for this I want to restart my testing each time completely clean. This application loads a large amount of data from Twitter. Therefore, before I start, I delete all data from the database and kill any processes from my web account associated with this application. When I try to then load my application, I get the following error:
[Macromedia][SequeLink JDBC Driver][ODBC Socket][Microsoft][SQL Native Client]Communication link failure
I would assume this has something to do with me killing all the related processes in the DB. After some amount of time I am able to run queries again.
Does this have something to do with the connections setup information in Coldfusion Administrator?
Does it just take some time to reset the connection? Is there any way to get around this?
Is there a better way to start fresh and clean when testing the loading?
By default, ColdFusion pools connection threads. I would guess, based on your comment to Stephen Moretti, that you are killing a connection that CF expects to still be live. That said, I've never had problems killing long DB threads, so this is pure speculation.
I'm not sure what killing these threads gets you, as far as testing goes. Once the page has stopped processing, open DB connections should not push or pull additional data.
I suspect that the error is actually related to how you are "cleaning up", particularly when you say "kill all related processes". By this I'm guessing you go into task manager and actually kill the processes.
I'm also guessing that if you're using SQL Server, you're on windows.
Rather than killing processes, cleanly stop the services associated with your application. Go into the Services Control Panel :
Stop your IIS or Apache Service.
Stop your ColdFusion Server instance service.
In terms of your database:
- Create a script for creating your database schema, tables, views, users and permissions and any default data entries
- drop your schema
- restart the sql server services if you want to be sure you've created any cached data out.
- run the script to create a blank of your database.
You could at this point actually create a database back up and just restore this, but its always handy to have the scripts to run on servers if you don't want to restore a backup.
After this start your coldfusion and iis/apache services.