using elastic-search as database in django - django-models

I want to use elastic-search as database in my project which I build on django framework. As I learn, elastic search is used with together a database through haystack where elastic-search is used for only indexing and fast retrieval issues. Here it is explained how to integrate a database and elastic-search with haystack However, I would like to use elastic-search without a database, in other words I will index all data. I did not see anything like below that you can use elastic-search as database backend in django settings module.
# settings.py
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.mysql',
'OPTIONS': {
'read_default_file': '/path/to/my.cnf',
},
}
}
Is not it possible to use only elastic-search with model layer of django?
Here is the django-elasticsearch backend and in the examples section it does in the way I would like to do but I do not know whether it is storing first in a database and indexing on elastic-search or just store and indexing on elastic-search.

I think the short answer to your question is "no." I looked into this a while ago, and found that there was a project to create an elasticsearch database engine (https://github.com/aparo/django-elasticsearch), but with a last commit in 2011, I would not think that it is viable with post 1.0 versions of ES, and possibly 1.6.x versions of Django.

Related

How to have sample dataset on AWS ElasticSearch or other service provider for SearchKitManager()?

I am trying to use SearchKit and I want to know how to set up an ElasticSearch instance indexed with sample movie dataset.
Tried using:
AWS ElasticSearch Service. But don't have an actual dataset to upload through bulk API. The one that I have doesn't have indexing.
Sample data:
[
{
"movie_title":"Avatar ",
"director_name":"James Cameron",
"actor_1_name":"CCH Pounder",
"actor_2_name":"Joel David Moore",
"genres":"Action|Adventure|Fantasy|Sci-Fi",
"language":"English",
"country":"USA",
"content_rating":"PG-13",
"budget":"237000000",
"title_year":"2009",
"plot_keywords":"avatar|future|marine|native|paraplegic",
"movie_imdb_link":"http://www.imdb.com/title/tt0499549/?ref_=fn_tt_tt_1"
},
...
]
Tried using appbase.io ready made movie-dataset. But I am not sure how to connect it with SearchKitManager().
Would really appreciate any help.
you may want to try ReactiveSearch - it's an alternative to SearchKit from appbase.io (I work there) that's actively maintained. It supports direct use of appbase.io apps.
Re: SearchKitManager(), I am not sure if that may work as we don't support cluster level routes.

What are the differences between mobiledata and cloudant services?

hello I am analyzing these two services to use In my app, and I could not find some doc about the difference between these two services. I know that mobiledata extends the cloudant, but what are the pro and cons? In my case I need to have more control above the data, for example create views (I could not create In mobiledata service) etc
So, do you know what the main divergences between these two services? When to use mobile data or cloudant?
Thanks
There are three ways to use Cloudant on Bluemix from an iOS application:
Use Cloudant directly by adding it as a service to your application.
Use MobileFirst for iOS, the Data part specifically (currently beta).
Use the Mobile Data cross-platform SDK.
(1) and (2) allow you to use views within Cloudant by adding them
via either the Cloudant dashboard or by using the Cloudant HTTP API.
(3) is a service which uses Cloudant in the background, but provides
no direct access to your data using the Cloudant HTTP API; you're
limited to the services exposed by the SDK. Therefore you can't use
many Cloudant features like views or Cloudant Query. Think of
Cloudant more as an implementation detail here, rather than an
exposed component as it is for (1) and (2).
Therefore, (1) or (2) is probably more suitable for your needs as
you mention wanting to use views.
Backend of mobile data service of bluemix is cloudant.For details please refer below link on getting started:
https://www.ibm.com/developerworks/cloud/library/cl-rapiddev-app/
http://www.techrepublic.com/blog/the-enterprise-cloud/managing-your-databases-in-the-cloud-how-cloudant-does-it/
https://www.ibm.com/developerworks/cloud/library/cl-rapiddev-app/
P.S- Cloudant is no-sql(create view wont be supported) DBAAS
For details on no-sql,please follow below link:
http://www.zdnet.com/article/what-is-nosql-and-why-do-you-need-it/
Cloudant is the IBM bluemix mobile data backend. And yes you can build sorted secondary key:value indexes, called "views" using JavaScript MapReduce functions.
Here is an example :
map: function(doc){
if (doc.rep){ emit({"rep": doc.rep}, doc.amount); }
}
reduce: _sum
For more details you can refer these links :
http://examples.cloudant.com/sales/_design/sales/index.html
http://examples.cloudant.com/sales/_design/sales/index.html#basic
http://examples.cloudant.com/whatwouldbiebersay/_design/whatwouldbiebersay/index.html
For mobiledata cloudant is acting behind the scenes. Cloudant is an open source
non-relational, distributed database service of the same name that requires zero-configuration.
Cloudant is based on the Apache-backed CouchDB project and the open source BigCouch
project.
Please follow below link for more details:
https://cloudant.com/cloudant-ibm-bluemix-tutorials-and-demos/

Authentication with AngularJS - NodeJS

I cant figure out how to do a simple Authentication in my AngularJS app.
What would be the best and easiest way to do a normal server side authentication with my Setup:
Yeoman angular generator, running grunt server on :9000.
Does anyone have a good tutorial? or any tips?
Another question, what is the simplest way to store data with this setup? using MongoDB?
Thank you :)
AngularJS is a front-end JavaScript framework, you can use anything of your choice, loving and knowing at the back-end for your application. This question was something like you are asking "I am using HTML5, what should I use at my back-end?" Angluar can be used with many server-side languages, viz. Ruby, Node, PHP.
There is an awesome tutorial talking about Ruby on Rails + Angular by David Bryant Copeland.
If you want to use PHP, you could use any framework which comforts you, there are many
available. CodeIgniter is one of the popular PHP framework.
If you want to use Node for your application, Passport.js could
be something of interest. MEAN Stack is the new thing which is coming up, MEAN stands for MongoDB + Express.js + Angular.js + Node.js. There is a ready Yoeman generator for MEAN stack available.
Again depending upon the requirement you should choose between SQL or NoSQL database. Also depends upto certain extend on the choice of the server-side language.
If you need a scalable database which stores hierarchical data, NoSQL should be your choice. MongoDB is a popular NoSQL database; CouchDB, RethinkDB are other alternatives.
SQL database are used where application needs high transaction. Though we can use NoSQL database for transaction based application, but it is not stable in comparision to SQL databases. MySQL is the most commonly used SQL database.
Angularjs is really not involved in the authentication process.
The basic flow of authentication is quite simple:
You make a post request from your 'Angularjs app' (your client side application) to your server, passing as parameters a pair of username/password.
Then, on your nodejs application (server side) you query your database looking for the username provided and you try to match the password that was sent within the post request with the password you have stored in your database related to that user. If they matches, you set up a cookie on user's client which is related to a session stored in your database.
To make this simpler there are some libraries that help you.
You could use passport.js (http://passportjs.org/) together with express (http://expressjs.com/). Passport will help you setting up with ease the authentication process in a safe way and express will give you tools like the cookie parser, support for session and other tools you will need to use. Express will help you also setting an endpoint where you will post the request for logging in to.
Last thing, for storing data, you can use any database you want.
Nodejs has a relevant number of third party libraries that help interfacing with databases. If you want to use mongodb, this library (https://github.com/mongodb/node-mongodb-native) will make your life easy.
Your best bet would be to use the angular-fullstack generator as it comes with basic authentication -- both local and oAuth -- built in. From there, you can either just use the authentication that is setup for you or you can use it as a reference to help you figure out how its all working.
The easiest setup I am using is:
NodeJS / Express / Passport / Socket.io / MongoDB (can be anything else actually)
AngularJS
Express is handling all the security with Passport (there's a passport method being added to express req automatically that gives you an ability to check whether user is authenticated or not). It's called isAuthenticated and you can use it like below:
function loggedOnly(req, res, next) {
if (req.isAuthenticated()) {
next();
} else {
res.redirect('/');
}
}
Route / & /login are public, but when user is logged in, they redirect to /app.
Route /app is Angular app which is private and redirects to /login when user is not authenticated.
app.get('/', anonOnly);
Socket.io has passport.socketio middleware for automatically refusing unauthorised connections that may occur.
Then, I access user object by doing the following:
io.on('connection', function(socket) {
var user = socket.conn.request.user;
});
To sum up, login logic is handled by static Express views. Moreover, Express prints few constant's to Angular app containing e.g. userData. Quite useful for displaying e.g. userName at some point.
Although it may take some time to set that up, it's definitely worth doing if you want to understand the whole logic of your app. I've given you the names of open-source packages to use, all the details can be found in their readme & guides (lots of them already exists if you google for a while).

Connecting Backbone.js with CouchDB when hosting in Apache or IIS

I'm having trouble understanding how to connect Backbone.js with CouchDB cleanly if I am to serve the html and js files from Apache web server or IIS (not from node.js or couchApp).
Should I use a backbone.js CouchDB extension to abstract the requests? If so which ?
Should I connect Backbone.js connect directly to the CouchDB RESTful interface or should I use
an intermediate PHP layer (because I am familiar with PHP but and not Java or other server side setups) ? If so which ?
What is the best way to deal with the 'same origin policy' preventing me from interacting with the database ?
Basically I want a tidy way using backbone.js to GET json documents from a number of collections by id, an array of ids, or all ids.
Also from a second admin web interface I need to be able to update, create, and delete by id
I don't think you need any extension to work with CouchDB.
The main thing you need to do is set idAttribute to _id in all your model.
Also, I would advise you to put an intermediate layer between CouchDB and your client. Unless your use-case is very simple, you'll quickly want to transform your documents before sending them to your client. You can do that with PHP, Express and use CouchApp if you want to stick close to CouchDB.
If you want to bypass SOP cleanly, take a look at CORS, which is now supported by most browsers.
I'm not familiar with PHP, but here's how I do it with Express and felix-couchdb:
var couchdb = require('felix-couchdb'),
client = couchdb.createClient(config.port, config.host),
myDb = client.db('my_db');
// (... express boilerplate)
app.get('/resource/:resource_id', function (req, res) {
myDb.getDoc(req.params.resource_id, function (err, doc) {
if(err)
return res.send(err);
res.send(doc);
});
});
I have had very good luck with https://cloudant.com/blog/backbone-and-cloudant/

HTML5 database storage (SQL lite) - few questions

Hy there,
I can't find enough beginner resources on the web about HTML5 database storage usage examples (CRUD)
I'm opening(creating) my DB like this:
var db;
$(document).ready(function()
{
try
{
if (!window.openDatabase) {
alert('Not Supported -> Please try with a WebKit Browser');
} else {
var shortName = 'mydatab';
var version = '1.0';
var displayName = 'User Settings Database';
var maxSize = 3072*1024; // = 3MB in bytes 65536
db = openDatabase(shortName, version, displayName, maxSize);
}
}
catch(e)
{
if (e == 2) {
alert("Invalid database version.");
} else {
alert("Unknown error "+e+".");
}return;
}
});
QUESTION 1: How many databases can i create and use on one domain?
QUESTION 2. How to delete (drop) a database. -> i have not figured this out yet.
To create sql queries you use transaction:
function nullDataHandler(transaction, results) { }
function createTables(db)
{
db.transaction(function (transaction)
{
//first query causes the transaction to (intentionally) fail if the table exists.
transaction.executeSql('CREATE TABLE people(id INTEGER NOT NULL PRIMARY KEY AUTOINCREMENT, name TEXT NOT NULL DEFAULT "John Doe", shirt TEXT NOT NULL DEFAULT "Purple");', [], nullDataHandler, errorHandler);
});
}
QUESTION 3: How so is the above transaciton failed if a table exists? Is the nullDataHandler involved to do this? Where on the web is there documentation explaining the executeSql API? Arguments?
thx
The spec you're looking for is Web SQL Database. A quick reading suggests:
There is no limit, although once your databases increase beyond a certain size (5MB seems to be the default), the browser will prompt the user to allow for more space.
There is no way, in the current spec, to delete databases.
The executeSql() function takes an optional error callback argument.
HTML5 Doctor also has a good introduction.
Going forward, though, I'd recommend looking at Indexed DB. Web SQL has essentially been abandoned since there is no standard for SQL / SQLite. Even Microsoft has endorsed Indexed DB. See Consensus emerges for key Web app standard.
CREATE TABLE IF NOT EXISTS table_name
will create a table table_name only if if does not exist.
I found the following WebSQL tutorials helpful for basic CRUD operations, as they contained examples, and explained what the code was doing:
A Simple TODO list using HTML5 WebDatabases
HTML5 Web SQL Database
And the following links for SequelSphere (an HTML5 JavaScript SQL Relational Database Alternative to WebSQL that works in all browsers, storing data in LocalStorage and IndexedDB):
SequelSphere basic Usage instructions
API Documentation
Full Index of Guides and Helper Documentation
Using PersistenceJS there is a persistence.reset API which will wipe the database clean.
PersistenceJS Site
For developing / testing purposes, you can view content and delete webSQL, IndexedDB, cookies, etc by searching for your domain name at this URL in Chrome:
chrome://settings/cookies
There, you can delete all the storage for a domain or just certain local storage entities. Yes, the URL implies just 'cookies', but the interface at this URL includes all types of offline storage.
It would be great I think if the Chrome developer tools interface had the ability to right-click and delete a data storage entity in the Resources tab along with inspecting the content. But for now, all I know of is the settings/cookies URL.
It is supported on iOS safari,chrome and some latest version of opera....it's not yet adopted by IE and Firefox that's it......what more one can ask than local db on browser which has relational db system...so u can query it easily and handle complex data....which is very tougher in key vale based systems..
I remember reading it even supports upto one gb.i am not sure....
Note:
1)I'd like to mention one point there is a IDE called Dashcode which let's u build web apps that looks like iOS native.even there also web SQL is used.
2)actually web SQL is a implementation of SQLite on browsers.
3)SQLite is most prefered in both iOS and android as db for native code..
The drawbacks of SQLite:
The Lack of concurrency support but which is not a problem in browser as it's gonna be used by single user at a time..this is a case also in mobile.
Conclusions:
Web Sql is abandoned by w3 that's a sad thing so we've to explore other options.

Resources