How are you supposed to store users passwords in a Cloudant DB ? By users, I mean users of an application that uses Cloudant as a backend.
I've searched the docs, but I found nothing on that topic.
There's a _users database, in which you can create users and add a "password" field, but the password is a regular field that the DB admin (and possibly others) can read.
Is there a built-in way to hide it from view or encrypt it ?
EDIT
I've found a piece of the puzzle, the CouchDB security feature that encrypts user's passwords.
Since CouchDB 1.2.0, the password_sha and salt fields are
automatically created when a password field is present in the user
document. When the user document is written, CouchDB checks for the
existence of the password field and if it exists, it will generate a
salt, hash the value of the password field and hash the concatenation
of the password hash and the salt. It then writes the resulting
password into the password_sha field and the salt into the salt field.
The password field is removed.
This has the following implications: Clients no longer have to
calculate the password salt and hash manually. Yay.
Now what's missing is the link between that underlying DB feature and Cloudant (just setting the password field in the user document is not working).
EDIT 2
Found that other question which is similar to this one - it's a broader problem, but specifically for web apps. There's an accepted answer from #JasonSmith that addresses my question:
Can I use CouchDB security features
Answer's "yes you can"
Cloudant does not yet have the newer CouchDB feature where the server
will automatically hash the password for you
But the CouchDB doc states that this features is included in the 1.20 version from 2013! How is that a "newer" feature?
From the doc, I gather that Cloudant uses CouchDB 1.61.
To recap:
the feature exists,
it's a CouchDB security feature existing in the CouchDB version that Cloudant uses,
Cloudant can be configured to use CouchDB security features
So... the missing link is really really small...
As you've discovered, Cloudant does not automatically hash passwords server side, as introduced in Couch 1.2. Also, it only supports the simple password scheme: salted SHA1 (which you may find insufficient). That's how passwords are supposed to be saved (not plain text).
It also misses a bunch of other security features, such as special access rules to the _users database (described here).
Hashing passwords "automatically" can be accomplished by an update function (special access rules could be implemented through show/list functions). I have done this myself:
function (doc, req) {
var body = JSON.parse(req.body || '{}') || {};
if (doc == null) doc = {
_id: req.id,
type: 'user'
};
doc.name = body.name;
doc.roles = body.roles;
doc.salt = req.uuid;
doc.password_scheme = 'simple';
doc.password_sha = hex_sha1(body.password + doc.salt);
return [doc, { json: doc }];
}
Get hex_sha1 from here. Set the above as an update function in a design doc on the _users database. You can also use this as a validation function.
Then instead of PUTing a user into the database, you PUT the same JSON to the update function, and it generates the salted hash before committing it to the database.
If salted SHA1 is not enough for your purposes you can't rely on _users on Cloudant, as is.
Not knowing more about your design, I can't really give much advice.
But I should warn you that, thanks to poor _users support, it's e.g. nearly impossible to effectively implement a 2-tier architecture on Cloudant. I'd be glad to be contradicted by someone who knows better, but after banging my head against this for months (and nagging support), this is the conclusion I've come to.
Eventually, you'll need an application layer to do user management, either through _users or API keys. Once you have such a layer, that's where you can hash passwords, and/or skip the _users database and do user management some other way. Every sample posted by Cloudant eventually does this, as soon as things get complicated enough (and none of the samples scale to tens of thousands of users, btw).
Finally, to #Anti-weakpasswords, who says you must go with PBKDF2 and huge iteration counts.
This is sound advice regarding saving passwords in general, but:
this doesn't work with Cloudant, at all;
it doesn't really work very well with CouchDB either.
First, as stated, if salted SHA1 is all that Cloudant supports, period.
But even for CouchDB, it's bad advice. With basic HTTP auth, you're sending the password on every single request. Key stretching with huge iteration counts on every single request would put tremendous pressure on the server, so large iteration counts are not recommended (that's why the docs have a 10 in there). If you're going down that road, you need to make sure you always use _session and cookies, and avoid basic auth like the plague.
More likely, if you take security seriously, you need to get a layer between the client and the database that handles user management some other way, and have decoupled database users/roles with strong enough passwords not to need strong hashing at all.
Clusers just came out! It may be useful to you. Clusers is a user account creator meant for Cloudant and CouchDB. It uses the older "password_scheme: simple" which Cloudant and older CouchDB.
First, you really, really do need to read Thomas Pornin's canonical answer to How to Securely Hash Passwords.
Read it right now.
Now read that CouchDB link, and see one of the recommended ways to produce password_sha for 1.3 (and if you're not on at least 1.3, get there).
{
"_id": "org.couchdb.user:username",
"_rev": "1-227bbe6ddc1db6826fb6f8a250ef6264",
"password_scheme": "pbkdf2",
"iterations": 10,
"name": "username",
"roles": [
],
"type": "user",
"derived_key": "aa7dc3719f9c48f1ac72754b28b3f2b6974c2062",
"salt": "77bac623e30d91809eecbc974aecf807"
}
Make certain that password_scheme is pbkdf2!
See that "iterations": 10 in the same? You need to bump that up by a huge amount - I'd say try a number in the low hundreds of thousands and see how it runs; SHA-1 is very cheap these days.
As far as Cloudant goes, here's a Github repository with some code to have Cloudant use the CouchDB _users.
Related
I wanted to kow if it is save to store my hashed password inside a table, that resides next to the rest of my other data(tables) in the same database?
For example imagine the following (simplified) structure (with a lot of other data-tables):
authentication user
-------------- -----------
auth_id {PK} user_id {PK}
username realname
passwordhash auth_id {FK}
Is it safe to store a users authentication/password like this, or should it be stored differntly, maybe in an extra database or such?
I suggest reviewing this article by Sophos, as it really nicely describes the rationale and the secure design.
Article argues to have a separate password server, so that no one role can have access to the passwords.
Many enterprise systems compartmentize passwords away from other credentials.
It all depends.
In general, if an attacker can read one database, it's likely they can read another, so if your application would simply open a SQL connection to the other database, you'll get no significant benefit.
Also, if they can access your database, they can probably read the traffic to and from the database, including username/password combinations.
So, I would suggest storing the authentication data in a separate database alone isn't a huge improvement in security; you'd be better off spending your time and energy on other security matters such as key management (you are salting your hashes, right?).
However...if you can set up physically separate infrastructure, and expose the authentication features as a service to the front end, using e.g. OAuth, you would create a significant barrier to the attacker. They would have to compromise this alternative database, and as the integration would be at the front end, they can't rely on sniffing traffic to your database.
I am creating a mobile app that uses Google App Engine (python) for the backend. Users sign in with Twitter on the app, and the auth token and secret are passed to the backend (over https) so that the server can authenticate with Twitter and also periodically sync friends and followers in a background task. Because they are used by the background thread, I want to store the information in the datastore so they can be retrieved and used later.
Right now, during development and testing, I just put these in the datastore in plain text. But I'd like to add a little more security by storing it encrypted and decrypting it when its needed. Thank you for any help!
For general account passwords, I use
security.generate_password_hash(raw_password, length=12)
based on how webapp2_extras stores the passwords. But this approach wouldn't allow me to retrieve the data. Is there anything similar that allows for encryption and decryption?
Normally for password storage you would use a unidirectional (One way) encryption technique so that no one can work out what the password is and then take the user supplied values and compare them to the stored values. This way you're never really storing the actual password and it's less likely to be stolen.
What you're looking for is a bidirectional encryption technique where by you provide the value and a key to create an encrypted value and can apply the key to the encrypted value to get the original.
You haven't stated which language you're using so I cannot provide a good example, however I suggest looking at techniques such as AES. Please keep in mind that if you choose an encryption technique with a short key it will be much easier to brute force. Any encryption that is bidirectional is at risk of easier brute force and once the key has been determined ALL passwords are at risk of being decrypted. Most languages have some form of support for AES and similar encryption techniques.
There are many techniques available, some much newer and more secure so do some research and see what you deem 'secure enough'.
Read more here: http://en.wikipedia.org/wiki/Advanced_Encryption_Standard
I have a web app that stores objects in a database and then sends emails based on changes to those objects. For debugging and tracking, I am thinking of including the Document Id in the email metadata. Is there a security risk here? I could encrypt it (AES-256).
In general, I realize that security through obscurity isn't good practice, but I am wondering if I should still be careful with Document Ids.
For clarity, I am using CouchDB, but I think this can apply to databases in general.
By default, CouchDB uses UUIDs with a UTC time prefix. The worst you can leak there is the time the document was created, and you will be able to correlate about 1k worth of IDs likely having been produced on the same machine.
You can change this in the CouchDB configuration to use purely 128bit random UUIDs by setting the algorithm setting within the uuids section to random. For more information see the CouchDB Docs. Nothing should be possible to be gained from them.
Edit: If you choose your own document IDs, of course, you leak whatever you put in there :)
Compare Convenience and Security:
Convenience:
how useful is it for you having the document id in the mail?
can you quickly get useful information / the document having the ID ?
does encrypting/hashing it mean it's harder to get the actual database document? (answer here is yes unless you have a nice lookup form/something which takes the hash directly, avoid manual steps )
Security:
having a document ID what could I possibly do that's bad?
let's say you have a web application to look at documents..you have the same ID in a URL, it can't be considered 'secret'
if I have the ID can I access the 'document' or some other information I shouldn't be able to access. Hint: you should always properly check rights, if that's done then you have no problem.
as long as an ID isn't considered 'secret', meaning there aren't any security checks purely on ID, you should have no problems.
do you care if someone finds out the time a document was created? ( from Jan Lehnardt's answer )
I have migrated my project site from magento to django, it is a site with millions of users. I have successfully imported all the data from the site but the main problem I am having, is to migrate customer database especially with password, because obviously I can't give new passwords to each customer on django site because it will be so time consuming. And for the customers to reset the password, will be very irritating and many customers will be lost from the site.
Please give me some ideas how can I retrieve customer password in magento from database?
You can not restore passwords. Magento uses hashes with salt for storing passwords (md5 and sha1 according to the edition).
Only things you can do - reset passwords or implement in your site the same function for checking passwords as it's done in magento.
At a glance it looks like django stores it's passwords using md5${salt}${hashed_password} and they prepend the salt. This being the case you should be able to take the Magento passwords and reformat the string so that it passes django password checks.
$hashArr = explode(':', $hash);
$newPasswordString = 'md5$'.$hashArr[1].'$'.$hashArr[0];
The viability / accuracy of this answer will depend on whether you are using Community / Professional or Enterprise. The code is based on the Community code as I can share this more freely, depending on the implementation the same may or may not be possible with Enterprise, you'd need to check Mage_Core_Model_Encryption:validateHash.
If you keep old and weak hashes from legacy systems, you are endangering your users.
If some attacker (or some evil admin) gets access to the hashes and user data, he can easily crack them (see e.g. http://hashcat.net/oclhashcat-plus/ ) and use them globally against any service your users are using (of course users are advised to not reuse passwords, have strong passwords, etc., but many will not know or ignore this).
Weak hashes are about anything except bcrypt, pbkdf2, shaXXX_crypt. Even something like randomly salted sha256 has to be considered as weak.
So, a sane procedure could be:
Do NOT just support weak hashes directly
Import the weak hashes, but hash them again using strong algo: strong(weak(cleartext))
At login time, upgrade the double-hashed stuff to just strong(cleartext)
Run the system in this mode for a while, your frequent users will have strong(cleartext) hashes at some time.
To get rid of the inner weak() hash, invalidate all such hashes (this will only affect your infrequent users), send users a password-reset link and some contact info for cases of trouble. In the E-Mail you can point out that you care very much about security and that the reset is necessary to improve it. If the infrequent user is still interested in your service, he will just reset his password. If he is not interested any more, he will ignore the email. In any case, you won't have a double-hashed pw in your password storage any more, but just an invalid one.
Disable support for the inner weak() hash, for double-hashing in the site configuration.
Deprecate support for the inner hash / double hashing in the software.
Later, remove support for the inner hash / double hashing in the software, to simplify it.
Note: if the legacy system had security breaches, taking the old hashes is no option as you have to consider the passwords to be exposed.
This could be a guide in you quest:
How to decrypt magento enterprise edition password?
The answer by Cags works for me. I'm moving from Magento community edition 1.7 to Django-Oscar.
I only have a few hundred users to move, but resetting their passwords wasn't an option.
For convenience...
The python version of his code:
hash = xxxxxxxxxxxxxxxxx:xx # The hash from Magento DB table customer_entity_varchar
splithash = hash.split(':')
djangopass = 'md5$' + splithash[1] + '$' + splithash[0]
I am building an application with health information inside. This application will be consumer-facing with is new for me. I would like a method to put privacy concerns completely at ease. As I review methods for securing sensitive data in publicly accessible databases I have frequently come across the notion of database translucency. There is the original book on the subject and an excellent tutorial on the subject from Oriellynet.
My concern is that I have seen very little information regarding this idea on what I would consider very-modern programming sites (like this one). There does not seem to be an article about the idea on wikipedia. No questions on the subject here, and no very recent tutorials or articles on the subject. To be uber-brief, the idea is that certain data is clear to some users of the system, while other users a cryptographically prevented from accessing that data, even if they have administrator access.
I have done substantial work on a prototype database that provides translucent data access. I have run across a considerable problem: To be truly translucent, there can be no mechanism for password recovery. If an administrator can reset a users password, then they can briefly gain access to a users data. To be truly translucent, the user must never loose the password.
Those of us who use strong encryption to protect private data in our daily lives (technorati to be sure) are used to this problem when using these kinds of strong encryption systems. If the word "blowfish" is part of your daily lexicon that is one thing, but a website that is consumer focused? I am concerned that users will not be willing to wrap their mind around the "truly encrypted just for you" notion implicit with true database translucency. I am afraid of the support call that begins with "I lost my password" and ends with me saying "There is nothing that I can do for you".
My question: Should I implement this method in my application? Are there other open source applications that have gone down this route that I can compare database designs with (esp using php/MySQL)? I anyone else pursuing these kind of truly secure, but really inconvenient feature sets? Is there another database security model that is more popular and modern that I have missed? Was database translucency a fad or a legitimate database design method that I should embrace? While I always appreciate discussion I would prefer objective answers that I can leverage in my design.
So, I've been looking at something similar to this recently, and hit upon the same issue. The solution I'm considering implementing is as follows:
Upon registration, create a unique, secure (long) key for the user and use this to encrypt their data.
Encrypt this key with the user's password using e.g. AES and store it in the database.
At this point, you're still in the situation where if the user forgets their password, they've had it.
Create a public/private key pair representing your organisation, and store the public key on the server.
Split the private portion of the key into several components and give each to people (e.g. directors of your company) who have a significant stake (preferably financial) in the continued success of your company. Do this such that any two, or any three people can get together and restore the full private key when required. Encrypt each person's key with their own password.
When a user registers, as well as encrypting their key with their password, encrypt it with the organisational public key and store it somewhere.
Create a password reset form which records a request to reset the password of a user, along with some proof that the user is who they say they are (e.g. challenge/response).
Record these reset requests (optionally encrypted using the public key again) in the database.
Once per hour/day/week/month, get the requisite key-holders together, and use their combined keys to process the accrued reset requests, decrypting the keys of users who successfully proved they are who they say they are.
There are lots of challenges and considerations in this. I've got a few thoughts on most of these, but would be interested in others opinions too:
How to split the key safely between multiple people so that no one person can decrypt the stored keys.
How to minimise the number of keys that would be exposed if the 'master keys' genuinely fell into the wrong hands.
How to make sure that if (heaven forbid) your key-holders lost their keys, then (a) there's no risk of exposure of the data, and (b) there's no risk that suddenly the ability to reset passwords is lost forever.
How to successfully validate that someone really is who they say they are without making this a glaring hole in your whole security approach.
Anything you implement in this area WILL reduce the security of the translucent database approach, without a doubt, but this may be a worthwhile compromise depending on the nature of your data.
Should I implement this method in my application?
Well like other things in life, there is a trade off :) It's probably more secure but harder to built.
Are there other open source applications that have gone down this route that I can compare database designs with (esp using php/MySQL)?
Don't know, I guess the tools are there to do it yourself :)
Is anyone else pursuing these kind of truly secure, but really inconvenient feature sets?
Yes, but it seems like it's still in an immature state, like your problem you describe concerning lost passwords.
Is there another database security model that is more popular and modern that I have missed?
Basically there are two kinds of database connections. One option gives users a real database account, the other is to use single sign-on to the database. Prior to the web coming along, there were proponents of both models in the client/server world, but amongst web developers the single sign-on method is leading.
Was database translucency a fad or a legitimate database design method that I should embrace?
Don't think so, the UNIX password database, for instance, is a great example of a basic translucent database ;)
here something to read link text
Re: translucent databases. You could, I suppose, use fingerprints. What about burn victims, or people who end up losing their fingerprints? Oops. Is it worth that small percentage of users?
Familiarize yourself with HIPAA, especially when it comes to technology.
Remember that no system is truly secure, except Skynet*, and look what happened with that! Humans are in charge. When you work in a medical company, you sign an NDA indicating that you won't release any of the information you learn as part of your duties because it is confidential.
There will be someone to reset people's passwords. That's the way it is, because not everyone is technologically competent, and that's the way it stays for now.
You only have to implement security as well as HIPAA says.
in truth, there is another truly secure system: it is unplugged from both the network and the electricity, and it is turned off.
Slightly different solution, you might want to check out cryptdb:
http://css.csail.mit.edu/cryptdb/