So I have a pyramid app which stores data in zodb (Substanced) and also creates a solr index for a speedy search of that data. Some of the solr indexing takes a while so I am wanting to make the solr indexing asynchronous. I am going to use rabbitmq and celery.
Do I benefit from using pyramid_celery? I don't want to use the ini file to store the celery config and there are no scheduled tasks so no celery beats. This is small scale and all of the processes/tasks will run on one machine.
Thanks
OK, so I am answering my own question. I asked this on the pylons google group and the response from the author of pyramid_celery was
Absolutely nothing. pyramid_celery is specifically for sharing your ini configuration / app configuration with your celery workers. If you don't have a need to share those things you have no need for pyramid_celery :)
I will also look at Mikko's option.
Related
I need to know, has someone integrated any DB to Semantria, and get output to any DB or excel or text file ?
I have tried to explore semantria via excel and API , but integration does not work perfectly.
It depends on what kind of integration you're looking for.
I have already done many integrations with different storages including indexing services and RDBMS solutions.
Unfortunately there are no ready-to-use components available on the market, so you will need to build integration by your own.
Semantria offers SDK (https://github.com/Semantria/semantria-sdk) for all modern languages, you will need to build a logic that will get analysis results and will save them to a certain storage.
Can you please explain what storage do you use and what Semantria output you're interested in?
Thanks George.
Well at the moment, we are just focusing on pulling the data from DB (take for instance mySQL, or Oracle), and output should again go back to same DB, i will take care of transformation needed in o/p.
Now where I am stuck, is the place where I can set up a link between DB and semantria, how will these SDK help, never worked on something like this.
A brief on this will surely be of great help
Is it possible to upload files to a web browser application (such as Dropbox) directly from a document management system (DMS) such as iManage (aka Worksite, Filesite, HP Autonomy)?
If not, what are the best ways of getting around this and enabling simple upload of files from the DMS to web applications?
If you are trying to share documents via a dropbox like solution, you might want t consider LinkSite. It is more manageable, which is quite a positive aspect since you will probably working in a legal environment.
More info can be found here: http://www.irisecm.com/hp-linksite-the-secure-and-enterprise-alternative-to-dropbox/
Best regards,
Daniel
By the way, LinkSite is rebranded and now called iManage Share.
If you are looking for a seamlessly integrated third party application you might want to consider Litéra Sync.
I have some text files with data to be indexed in my search-app on google app engine. I just don't understand when should I execute a function that puts all docs into index.
Currently I am trying to perform it in get-request on some admin page. But if I use all data then indexing process takes too much time to be run in get-request.
I will appreciate any kind of help.
store the data in the datastore (if it's more than 10K) then fire off a task to perform the indexing, and return a response to the user.
You haven't said if your using python, java or go.
If your using python look at https://developers.google.com/appengine/articles/deferred for info about the deferred lib, which is an easy way to start using tasks with python, alternately start reading up on tasks.
I have a couple of questions about the App Engine Map Reduce API. First of all there's a mapreduce package in the SDK, and there's a separate mapreduce bundle here:
https://developers.google.com/appengine/downloads
Which one should I be using? Should I be using the bundle, or is the documentation out of date and I should actually use the SDK version?
Second I'd like to be able to run mapreduce's on a non-default version to make sure that the requests from the mapreduce don't interfere with user requests.
What's the best way to do this? Can I start the pipeline with a task queue, and set the target version of that queue to be my non-default version?
We recommend using the open source version of Map Reduce for GAE at http://code.google.com/p/appengine-mapreduce/
The stale bundle link in the docs is a bug. That'll get cleaned up soon.
A few of our SDKs have bits of MapReduce (for historic reasons), but the open source version is the way to go for now.
As for using a separate version, this is kind of "it depends". If you're thinking of interference in terms of competition for the processor, that's not likely to be a noticeable issue. Depending on queue processing rates you've set up, more instances of your app will be spun up to handle mapping tasks as needed. I'd try some experiments first. Make sure you have a problem before you invest time and effort solving it.
mapreduce can be start on a not default version. And after it starts, it will continue run on that version automatically.
In my case I just deploy the code on a non default version and trigger the mapreduce with version_id.app_id.appspot.com/path_to_start_a_job.
cron job can also trigger the mapreduce on non default version without problem.
I'm working on an app engine project (java). I'm using the jdo interface. I haven't pushed the application yet (just running at localhost). Is there a way I can totally wipe my datastore after I publish? In eclipse, when working locally, I can just wipe the datastore by deleting the local file:
appengine-generated/local_db.bin
any facility like that once published?
I'm using jdo right now, but I might switch to objectify or slim3, and would want a convenient way to wipe my datastore should I switch over, or otherwise make heavy modifications to my classes.
Otherwise it seems like I have to setup methods to delete instances myself, right?
Thanks
you can delete it from admin console if there are not much enitty stored in your app. go to http://appengine.google.com and manually do it. easy for less than 2000-5000 entity.
This question addressed the same topic. There is no one command way to drop an entire datastore's worth of data. The only suggestion I have beyond those give in that previous question, would be to try out the new Mapper functionality, which would make it easy to map over an entire set of entities, deleting them as you went.