I am trying to create a scheduler for my organization which allows for admin to schedule 100+ people for a single time slot. Each scheduled person must have their own message and receive a reminder 24 hours before the event. The event is repeated every M-F at the exact same time. We also need to be able to check for duplicates, mark attendance and roll over no-shows to the next day. We used to use a Google sheet to do this, but this is no longer an option. I looked at using Google Calendar, which with scheduling would be perfect, but am limited to only 4 slots in an hour. I was curious if anyone had any ideas of what would work best for this solution. I have really wracked my brains out, and short of writing a custom app, which I do not have the time to do, I do not know where to go.
Thank you
Related
Disclaimer: I wouldn't be at all surprised to find out this is duplicate somewhere, but I have literally been searching for hours. All I can find it DBA information that I don't believe pertains to what I am trying to accomplish.
I am currently developing a small database for a friends startup. He has jobs that he needs to track that are performed on regular intervals (e.g. semi-annually, annually, quarterly, weekly, etc.).
I am trying to wrap my mind around how I can implement a solution in sql server where a customer has 0 to many tasks, and a specific task can have 0 many customers. Seems simple enough, but I also want to keep a history for each time the task was completed for each customer where it is assigned. This would be a record with a scheduled task and a completed date. Lastly, I need to be able to execute a query to retrieve upcoming scheduled tasks within a given period of time (e.g. all scheduled jobs coming up next month).
I know that there are is a ton of software that does stuff like this, so it can't be that uncommon, but I cannot find any information that is getting me anywhere. If there are any resources anyone could recommend it would greatly appreciated.
You should just create a table to hold all the list of tasks that need to be done, with columns where you can later record when they have been done.
You can easily then just query it to find what needs to be done.
Then create a stored procedure that does the work. Have the procedure do the right thing for different periods i.e. monthly, etc. Have it just work out what's needed, and do it.
Create a single SQL Server Agent job that runs that procedure each month. (Or however often it's needed).
As you do each step, no-one's going to mind if you ask for help with each part i.e. get comments on your table when you first design it, etc. (Or at least I wouldn't mind - some might but ignore them)
I am setting up push task queue on my Google App Engine App with a countdown parameter so it will execute at some point in the future.
However, my countdown parameter can be very large in seconds, for instance months or even a year in the future. Just want to make sure this will not cause any problems or overhead cost? Maybe there is a more efficient way to do this?
It probably would work, but it seems like a bad idea. What do you do if you change your task processing code? You can't modify a task in the queue. You'd somehow have to keep track of the tasks, delete the old ones and replace them with new ones that work with your updated code.
Instead, store information about the tasks in the data store. Run a cron job once a day or once a week, process the info in the data store, and launch the tasks as needed. You can still use a countdown if you need a precise execution date and time.
The current limit in Task Queues is 30 days, and we don't have plans to raise that substantially.
Writing scheduled operations to datastore and running a daily cron job to inject that day's tasks is a good strategy. That would allow you to update the semantics as your product evolves.
Are there any risks/cautions around creating tasks for a GAE Push Task Queue in say 1 month from now or even 1 year from now?
According to the documentation the maximum ETA for a task is 30 days.
The biggest risk for long-in-the-future tasks is that when this future finally arrives, you may no longer need this task. For example, a customer can close his account or you can release a new version of your software that is not compatible with the scheduled task. In fact, I cannot think of a use case where nothing can go wrong when a task is scheduled for 1 year into the future.
A better approach is to create entities that represent your events, and then have a cron job that checks once a day (or once a week) which entities are coming "due" in the next period and schedules tasks for them. This way you have only one day/week worth of scheduled tasks to deal with if you make changes in the code. It is also easy to delete these entities if a customer cancels an action or closes an account, for example.
I'm using GAE and task queue. In queue.yaml file, I keep default setting: 5/s. 1 month ago I thought it's enough but now there are about 40-50 tasks in one queue so my application runs too slow.
I want to know how many tasks per second is enough ? Can I change to 100/s ?
Thank you :)
Update:
My application gets data from some social networks, calculate and save to datastore. To over limit 30 seconds of GAE, I split this operation to tasks. I want to know the limit of GAE task queue before change and deploy to GAE :)
http://code.google.com/appengine/docs/python/taskqueue/overview.html#Quotas_and_Limits
or
http://code.google.com/appengine/docs/java/taskqueue/overview.html#Quotas_and_Limits
I'd highly recommend increasing the settings in steps to find your performance sweet spot. The number of tasks needed to run is obviously higher than 5/s but you don't know what's the proper number until you've tried running for a while, and heading straight to the top doesn't sound like a good idea.
I need to send out emails at a rate exceeding App Engine's free email quota (8 emails/minute). I'm planning to use a TaskQueue to queue the emails, but I wondered: is there already a library or Python module I could use to automate this? It seems like the kind of problem someone might have run into before.
If it's an option, why not just enable billing? It'll jump the max rate from 8 recipients/minute to 5,100 recipients/minute.
The first 2000 recipients is free each day, as long as you aren't going over the daily free quotas my understanding is that it will not cost you anything (and if you need to email more than 2000 people per day you're going to have to enable billing anyways).
The deferred library is designed for exactly this sort of thing. Simply use deferred.defer(message.send), and make sure the queue you're using has the appropriate execution rate.
its cheaper to just pay for it for a year than to engineer a workaround.
Easiest way in my opinion would be to use a queue, ex Amazon SQS, and pull 8 records per minute, in a cron job running every minute.
Considering it was pushed into the queue, then taken out, I am working out the math that it is an extremely cheap service.
See below, 0.000002 is the rate for 2 requests. (Add and View)
8 requests per minute, 60 minutes in an hour, and 24 hours in a day. Take into account 30 days in the average month, you are still under $1.
0.000002 * 8 * 60 * 24 * 30 = $0.6912
This might not be exactly what you were looking for, but it should be a pretty simple solution.
EDIT:
See here, a python SQS & S3 Lib (sqs is all that you should be looking for).
http://pypi.python.org/pypi/Python-Amazon/0.5
I'm not familiar with any canned solutions to this problem, but it should be something very easy to solve. Write the emails to a datastore table, with an auto_add_now date field to record the order in which they entered. Your cron job that runs every minute pulls the eight oldest records off, mails them and deletes them.
Certainly, if you can solve this is a reasonably generic manner, you can be the person who solves this problem for everyone with a nice open source module.