I have migrated my project site from magento to django, it is a site with millions of users. I have successfully imported all the data from the site but the main problem I am having, is to migrate customer database especially with password, because obviously I can't give new passwords to each customer on django site because it will be so time consuming. And for the customers to reset the password, will be very irritating and many customers will be lost from the site.
Please give me some ideas how can I retrieve customer password in magento from database?
You can not restore passwords. Magento uses hashes with salt for storing passwords (md5 and sha1 according to the edition).
Only things you can do - reset passwords or implement in your site the same function for checking passwords as it's done in magento.
At a glance it looks like django stores it's passwords using md5${salt}${hashed_password} and they prepend the salt. This being the case you should be able to take the Magento passwords and reformat the string so that it passes django password checks.
$hashArr = explode(':', $hash);
$newPasswordString = 'md5$'.$hashArr[1].'$'.$hashArr[0];
The viability / accuracy of this answer will depend on whether you are using Community / Professional or Enterprise. The code is based on the Community code as I can share this more freely, depending on the implementation the same may or may not be possible with Enterprise, you'd need to check Mage_Core_Model_Encryption:validateHash.
If you keep old and weak hashes from legacy systems, you are endangering your users.
If some attacker (or some evil admin) gets access to the hashes and user data, he can easily crack them (see e.g. http://hashcat.net/oclhashcat-plus/ ) and use them globally against any service your users are using (of course users are advised to not reuse passwords, have strong passwords, etc., but many will not know or ignore this).
Weak hashes are about anything except bcrypt, pbkdf2, shaXXX_crypt. Even something like randomly salted sha256 has to be considered as weak.
So, a sane procedure could be:
Do NOT just support weak hashes directly
Import the weak hashes, but hash them again using strong algo: strong(weak(cleartext))
At login time, upgrade the double-hashed stuff to just strong(cleartext)
Run the system in this mode for a while, your frequent users will have strong(cleartext) hashes at some time.
To get rid of the inner weak() hash, invalidate all such hashes (this will only affect your infrequent users), send users a password-reset link and some contact info for cases of trouble. In the E-Mail you can point out that you care very much about security and that the reset is necessary to improve it. If the infrequent user is still interested in your service, he will just reset his password. If he is not interested any more, he will ignore the email. In any case, you won't have a double-hashed pw in your password storage any more, but just an invalid one.
Disable support for the inner weak() hash, for double-hashing in the site configuration.
Deprecate support for the inner hash / double hashing in the software.
Later, remove support for the inner hash / double hashing in the software, to simplify it.
Note: if the legacy system had security breaches, taking the old hashes is no option as you have to consider the passwords to be exposed.
This could be a guide in you quest:
How to decrypt magento enterprise edition password?
The answer by Cags works for me. I'm moving from Magento community edition 1.7 to Django-Oscar.
I only have a few hundred users to move, but resetting their passwords wasn't an option.
For convenience...
The python version of his code:
hash = xxxxxxxxxxxxxxxxx:xx # The hash from Magento DB table customer_entity_varchar
splithash = hash.split(':')
djangopass = 'md5$' + splithash[1] + '$' + splithash[0]
Related
I wanted to kow if it is save to store my hashed password inside a table, that resides next to the rest of my other data(tables) in the same database?
For example imagine the following (simplified) structure (with a lot of other data-tables):
authentication user
-------------- -----------
auth_id {PK} user_id {PK}
username realname
passwordhash auth_id {FK}
Is it safe to store a users authentication/password like this, or should it be stored differntly, maybe in an extra database or such?
I suggest reviewing this article by Sophos, as it really nicely describes the rationale and the secure design.
Article argues to have a separate password server, so that no one role can have access to the passwords.
Many enterprise systems compartmentize passwords away from other credentials.
It all depends.
In general, if an attacker can read one database, it's likely they can read another, so if your application would simply open a SQL connection to the other database, you'll get no significant benefit.
Also, if they can access your database, they can probably read the traffic to and from the database, including username/password combinations.
So, I would suggest storing the authentication data in a separate database alone isn't a huge improvement in security; you'd be better off spending your time and energy on other security matters such as key management (you are salting your hashes, right?).
However...if you can set up physically separate infrastructure, and expose the authentication features as a service to the front end, using e.g. OAuth, you would create a significant barrier to the attacker. They would have to compromise this alternative database, and as the integration would be at the front end, they can't rely on sniffing traffic to your database.
How are you supposed to store users passwords in a Cloudant DB ? By users, I mean users of an application that uses Cloudant as a backend.
I've searched the docs, but I found nothing on that topic.
There's a _users database, in which you can create users and add a "password" field, but the password is a regular field that the DB admin (and possibly others) can read.
Is there a built-in way to hide it from view or encrypt it ?
EDIT
I've found a piece of the puzzle, the CouchDB security feature that encrypts user's passwords.
Since CouchDB 1.2.0, the password_sha and salt fields are
automatically created when a password field is present in the user
document. When the user document is written, CouchDB checks for the
existence of the password field and if it exists, it will generate a
salt, hash the value of the password field and hash the concatenation
of the password hash and the salt. It then writes the resulting
password into the password_sha field and the salt into the salt field.
The password field is removed.
This has the following implications: Clients no longer have to
calculate the password salt and hash manually. Yay.
Now what's missing is the link between that underlying DB feature and Cloudant (just setting the password field in the user document is not working).
EDIT 2
Found that other question which is similar to this one - it's a broader problem, but specifically for web apps. There's an accepted answer from #JasonSmith that addresses my question:
Can I use CouchDB security features
Answer's "yes you can"
Cloudant does not yet have the newer CouchDB feature where the server
will automatically hash the password for you
But the CouchDB doc states that this features is included in the 1.20 version from 2013! How is that a "newer" feature?
From the doc, I gather that Cloudant uses CouchDB 1.61.
To recap:
the feature exists,
it's a CouchDB security feature existing in the CouchDB version that Cloudant uses,
Cloudant can be configured to use CouchDB security features
So... the missing link is really really small...
As you've discovered, Cloudant does not automatically hash passwords server side, as introduced in Couch 1.2. Also, it only supports the simple password scheme: salted SHA1 (which you may find insufficient). That's how passwords are supposed to be saved (not plain text).
It also misses a bunch of other security features, such as special access rules to the _users database (described here).
Hashing passwords "automatically" can be accomplished by an update function (special access rules could be implemented through show/list functions). I have done this myself:
function (doc, req) {
var body = JSON.parse(req.body || '{}') || {};
if (doc == null) doc = {
_id: req.id,
type: 'user'
};
doc.name = body.name;
doc.roles = body.roles;
doc.salt = req.uuid;
doc.password_scheme = 'simple';
doc.password_sha = hex_sha1(body.password + doc.salt);
return [doc, { json: doc }];
}
Get hex_sha1 from here. Set the above as an update function in a design doc on the _users database. You can also use this as a validation function.
Then instead of PUTing a user into the database, you PUT the same JSON to the update function, and it generates the salted hash before committing it to the database.
If salted SHA1 is not enough for your purposes you can't rely on _users on Cloudant, as is.
Not knowing more about your design, I can't really give much advice.
But I should warn you that, thanks to poor _users support, it's e.g. nearly impossible to effectively implement a 2-tier architecture on Cloudant. I'd be glad to be contradicted by someone who knows better, but after banging my head against this for months (and nagging support), this is the conclusion I've come to.
Eventually, you'll need an application layer to do user management, either through _users or API keys. Once you have such a layer, that's where you can hash passwords, and/or skip the _users database and do user management some other way. Every sample posted by Cloudant eventually does this, as soon as things get complicated enough (and none of the samples scale to tens of thousands of users, btw).
Finally, to #Anti-weakpasswords, who says you must go with PBKDF2 and huge iteration counts.
This is sound advice regarding saving passwords in general, but:
this doesn't work with Cloudant, at all;
it doesn't really work very well with CouchDB either.
First, as stated, if salted SHA1 is all that Cloudant supports, period.
But even for CouchDB, it's bad advice. With basic HTTP auth, you're sending the password on every single request. Key stretching with huge iteration counts on every single request would put tremendous pressure on the server, so large iteration counts are not recommended (that's why the docs have a 10 in there). If you're going down that road, you need to make sure you always use _session and cookies, and avoid basic auth like the plague.
More likely, if you take security seriously, you need to get a layer between the client and the database that handles user management some other way, and have decoupled database users/roles with strong enough passwords not to need strong hashing at all.
Clusers just came out! It may be useful to you. Clusers is a user account creator meant for Cloudant and CouchDB. It uses the older "password_scheme: simple" which Cloudant and older CouchDB.
First, you really, really do need to read Thomas Pornin's canonical answer to How to Securely Hash Passwords.
Read it right now.
Now read that CouchDB link, and see one of the recommended ways to produce password_sha for 1.3 (and if you're not on at least 1.3, get there).
{
"_id": "org.couchdb.user:username",
"_rev": "1-227bbe6ddc1db6826fb6f8a250ef6264",
"password_scheme": "pbkdf2",
"iterations": 10,
"name": "username",
"roles": [
],
"type": "user",
"derived_key": "aa7dc3719f9c48f1ac72754b28b3f2b6974c2062",
"salt": "77bac623e30d91809eecbc974aecf807"
}
Make certain that password_scheme is pbkdf2!
See that "iterations": 10 in the same? You need to bump that up by a huge amount - I'd say try a number in the low hundreds of thousands and see how it runs; SHA-1 is very cheap these days.
As far as Cloudant goes, here's a Github repository with some code to have Cloudant use the CouchDB _users.
The first & accepted answer on this question about passwords management suggests to encrypt the user identifiers in DB.
The good point is that if anyone gets a password, he has to know how to decrypt the user login to get the full login/password pair.
Some disadvantages I see, for example:
you have to decrypt user logins every time you want to display them
if you want to do a 'begins with' search on user login to find users, you cannot simply use LIKE '...%'
ORDER BY on login field may be quite difficult too...
What would you recommend (encrypt user identifiers or not)?
As usual, the answer is "it depends".
In general, I'd say that if an attacker has access to your database, your security situation is so badly compromised that encrypting the passwords will likely do you no favours. This is different to using a one-way hash - it's likely that an attacker who has access to your database also has access to your decryption key, whereas one-way hashes, by definition, are one way.
As you already say, it's likely that you will need regular access to the userIDs (esp. if you use email addresses as user IDs); in that case, again, an attacker who can read your database likely can intercept the unencrypted data.
So, if you work for a bank, the government, or any other place where data security has to be at the very top of the list, this additional protection may just be worth it, especially if you have a strong key management system.
For other uses, I'd consider the additional security too small to merit the additional pain.
Encryption is considered to be a lesser form of secret storage than message digest functions. In fact, storing an encrypted password is a clear violation of CWE-257.
But why not hash the username? When the login the application will have the plain text. Depending on your application, you might not need to display a list of users. this would be an added layer of security, as both hashes have to be broken before the attacker can login.
That being said, if you have a plain text list of every username it will be trivial to perform a dictionary attack against any recovered hash. Further more user names are not created to be difficult to guess, often times users choose goofy names of birds or silly games like chess so that they are easy to remember.
I am building an application with health information inside. This application will be consumer-facing with is new for me. I would like a method to put privacy concerns completely at ease. As I review methods for securing sensitive data in publicly accessible databases I have frequently come across the notion of database translucency. There is the original book on the subject and an excellent tutorial on the subject from Oriellynet.
My concern is that I have seen very little information regarding this idea on what I would consider very-modern programming sites (like this one). There does not seem to be an article about the idea on wikipedia. No questions on the subject here, and no very recent tutorials or articles on the subject. To be uber-brief, the idea is that certain data is clear to some users of the system, while other users a cryptographically prevented from accessing that data, even if they have administrator access.
I have done substantial work on a prototype database that provides translucent data access. I have run across a considerable problem: To be truly translucent, there can be no mechanism for password recovery. If an administrator can reset a users password, then they can briefly gain access to a users data. To be truly translucent, the user must never loose the password.
Those of us who use strong encryption to protect private data in our daily lives (technorati to be sure) are used to this problem when using these kinds of strong encryption systems. If the word "blowfish" is part of your daily lexicon that is one thing, but a website that is consumer focused? I am concerned that users will not be willing to wrap their mind around the "truly encrypted just for you" notion implicit with true database translucency. I am afraid of the support call that begins with "I lost my password" and ends with me saying "There is nothing that I can do for you".
My question: Should I implement this method in my application? Are there other open source applications that have gone down this route that I can compare database designs with (esp using php/MySQL)? I anyone else pursuing these kind of truly secure, but really inconvenient feature sets? Is there another database security model that is more popular and modern that I have missed? Was database translucency a fad or a legitimate database design method that I should embrace? While I always appreciate discussion I would prefer objective answers that I can leverage in my design.
So, I've been looking at something similar to this recently, and hit upon the same issue. The solution I'm considering implementing is as follows:
Upon registration, create a unique, secure (long) key for the user and use this to encrypt their data.
Encrypt this key with the user's password using e.g. AES and store it in the database.
At this point, you're still in the situation where if the user forgets their password, they've had it.
Create a public/private key pair representing your organisation, and store the public key on the server.
Split the private portion of the key into several components and give each to people (e.g. directors of your company) who have a significant stake (preferably financial) in the continued success of your company. Do this such that any two, or any three people can get together and restore the full private key when required. Encrypt each person's key with their own password.
When a user registers, as well as encrypting their key with their password, encrypt it with the organisational public key and store it somewhere.
Create a password reset form which records a request to reset the password of a user, along with some proof that the user is who they say they are (e.g. challenge/response).
Record these reset requests (optionally encrypted using the public key again) in the database.
Once per hour/day/week/month, get the requisite key-holders together, and use their combined keys to process the accrued reset requests, decrypting the keys of users who successfully proved they are who they say they are.
There are lots of challenges and considerations in this. I've got a few thoughts on most of these, but would be interested in others opinions too:
How to split the key safely between multiple people so that no one person can decrypt the stored keys.
How to minimise the number of keys that would be exposed if the 'master keys' genuinely fell into the wrong hands.
How to make sure that if (heaven forbid) your key-holders lost their keys, then (a) there's no risk of exposure of the data, and (b) there's no risk that suddenly the ability to reset passwords is lost forever.
How to successfully validate that someone really is who they say they are without making this a glaring hole in your whole security approach.
Anything you implement in this area WILL reduce the security of the translucent database approach, without a doubt, but this may be a worthwhile compromise depending on the nature of your data.
Should I implement this method in my application?
Well like other things in life, there is a trade off :) It's probably more secure but harder to built.
Are there other open source applications that have gone down this route that I can compare database designs with (esp using php/MySQL)?
Don't know, I guess the tools are there to do it yourself :)
Is anyone else pursuing these kind of truly secure, but really inconvenient feature sets?
Yes, but it seems like it's still in an immature state, like your problem you describe concerning lost passwords.
Is there another database security model that is more popular and modern that I have missed?
Basically there are two kinds of database connections. One option gives users a real database account, the other is to use single sign-on to the database. Prior to the web coming along, there were proponents of both models in the client/server world, but amongst web developers the single sign-on method is leading.
Was database translucency a fad or a legitimate database design method that I should embrace?
Don't think so, the UNIX password database, for instance, is a great example of a basic translucent database ;)
here something to read link text
Re: translucent databases. You could, I suppose, use fingerprints. What about burn victims, or people who end up losing their fingerprints? Oops. Is it worth that small percentage of users?
Familiarize yourself with HIPAA, especially when it comes to technology.
Remember that no system is truly secure, except Skynet*, and look what happened with that! Humans are in charge. When you work in a medical company, you sign an NDA indicating that you won't release any of the information you learn as part of your duties because it is confidential.
There will be someone to reset people's passwords. That's the way it is, because not everyone is technologically competent, and that's the way it stays for now.
You only have to implement security as well as HIPAA says.
in truth, there is another truly secure system: it is unplugged from both the network and the electricity, and it is turned off.
Slightly different solution, you might want to check out cryptdb:
http://css.csail.mit.edu/cryptdb/
I'm currently working on basic user authentication for an app I'm putting together, and I don't have much experience with security.
This said, I understand the practice (and necessity) of salting/storing a (salted) password hash in a database as a blob, as opposed to the password (encrypted or no). I've already implemented this.
Is there anything to be gained by salting/hashing a user name and storing the hash in the database, as opposed to the username in plain-text (or encrypted)? It strikes me this would make it quite a bit harder to determine which users may access the system using the database for authentication.
Since it's vital to make it difficult for someone to crack the password for a user account, wouldn't it also make sense to increase the difficulty for determining which users are viable?
Edit: it's possible some of the language I'm using isn't 100% correct: feel free to correct :-)
Edit2: I changed one of my first points to indicate salting hashes -- thanks everyone for pointing out that I missed this :-)
Edit3: Removed wording indicating I am encrypting/decrypting a password. I'm using salted hashes and am storing that in the DB -- thanks Scotty for pointing this out.
It depends on the context
It's important to assess the sensitivity of the material you're serving. To dig deeper, I'll provide a few use cases.
Scenario 1: A social networking application
All of your user's interactions happen in the public. Their email address will be used as their username. There username is not considered private because their name appears in all their posts. The username may be searched by other users and/or email invites are enabled.
Verdict - Hashing = Bad
Scenario 2: An E-Commerce site
The user may or may not participate in public interactions (ex. commenting, reviews). Using an email address as the username is probably a bad idea because, through the use of password recovery, a compromised email account means a compromised user account on your site.
There is a lot of gray area here that is typically exploited for 'convenience.' If your site uses email as the user name, stores shipping history, and credit card numbers; a compromised email could mean a lot of identity theft troubles for your user.
In this case, using a policy where the username is not the user's email address is probably a good idea. Hashing the email adds no value.
Note: I'm looking at you Amazon.com.
Verdict: Common Practice != good practice
Scenario 3: A porn site
Make the username a pseudonym and the login name the user's email address. They may feel inclined to talk about the content and don't necessarily want their name to show up on Google's results for a smut site.
Assume the worst here. If somehow your database is hacked, the exposing of your user's identities could cause irreparable harm. Don't think this could happen to you? Take a look at this article.
Not only are their user's accounts hacked and passwords exposed but, there's a good chance a lot of those users used the same password on their email accounts. Now their info is posted anonymously on PasteBin for the whole world to see. Even worse, most of them probably don't even know this has happened yet.
By simply hashing both the username and password, they would have saved themselves and their users a whole lot of trouble.
Verdict: Definitely hash the email address whether or not it's used as the username.
Scenario 4: A bank
It should go without saying that no expense should be spared when it comes to banking/financial sites.
Security can be increased by:
Using a username other than the email address
Forcing a unique username by requiring numbers and letters
Hashing passwords
Requiring 2-point authentication (in case the user's email password is compromised)
Hashing email addresses
etc...
No expense should be spared to protect your users because, to not do so, means you're gambling with their livelihood.
Conclusion:
There is no hard and fast rule for security that applies to all sites. In some cases, the username is made public so hashing it adds no value. In others, not hashing it could cause irreparable harm. If you do end up developing a site where a username/email hash could be made useful here's a good approach.
Hash the username
Generate a unique salt for the user
Hash the password using the salt
Store the password with the salt in the database
By not hashing the username with a salt you avoid the chicken/egg problem. Unless you use a static salt for all of the usernames.
Keep in mind that a static salt for all the user accounts may be found out by reading the code. Once a static salt is found out, it'll essentially be useless when a rainbow table attack is employed. If you salt the passwords, generate a dynamic salt and store it along with the rest of the user's credentials in the database.
If you want hard/fast rules for simplicity here are a few good assumptions to remember:
Assume your database may be compromised at some point
Assume your source code will be compromised at some point
Assume your user's email will be compromised at some point
Assume your user's are dumb and use the same password for your site as they use for their email
Assume that hackers are smart/resourceful and financially driven.
If you choose to store sensitive/private data, then going the extra step may save you a PR/legal nightmare in the future.
Update:An interesting article about seed hashing just showed up on Coding Horror.
Short answer: most likely no.
Long answer: Your situation seems to lack the key "my usernames are sensitive because of ..." which raises the question: "Why? What is the specific, demonstrable problem that protecting usernames would solve?"
Without that problem, what you are describing is a common pitfall in security-related development (and really development as a whole): coming up with some idea to secure or obfuscate some part of the system, and then searching for a reason to use it. As with anything in software development, you should avoid doing anything other than exactly what is needed until a clear problem presents itself that can only be solved by using a specific tool.
Extra hint (for free!): salt your password hashes. Plain-old hashes are far less secure.
If you salted & hashed the username, you'd leave yourself with a bit of a chicken & egg problem.
If you salted & hashed the username, how would you find it in the database? You'd need to look up the user's record to find the salt you used to hash the username...
Probably not. Think of it this way - the username is the application's way of figuring out which account a user is trying to login as. The password is the application's way of figuring out whether the user is actually allowed to login as that account. In practice, this means you're going to look up a row in your accounts table using the username as an index. By encrypting the username, you're simply making it harder to find the right row.
However, if you're using the same encryption scheme to encrypt the username and password, they are pretty much equally secure - if you can break one, you can break the other. Thus, encrypting both makes it harder to lookup the user, but doesn't add any additional security.
Note: In your question you talk about decrypting your password field. You probably want to make this impossible (literally). Most people encrypt their passwords using a one-way hash function of some sort (MD5 and SHA256 are popular), along with a salt. The "one-way" part simply means that once you run something through the function, you can't use what you get out to get what you started with. However, if you start with the same input, you'll always get the same output. The salt is a secret that only your application knows (sort of like an encryption key), which is added to whatever you are encrypting, before it is run through the one-way hash. This makes it impossible to do things like match two encrypted passwords from two different sites (assuming they're using different salts).
It's not very fair to your users to store their password in plain text since everybody that has access to your database can see their passwords. You should use a salted hash.
Salt_(cryptography)
You can never properly evaluate the security of a system by looking at a single part of it in isolation. Whereabouts are you storing the key to decrypt the passwords?
Do the people that have access to the database also have access to the location you are storing the encyption key? If so you've only gained a minor improvement in security by encrypting the passwords and probably nothing much more to gain by encrypting the usernames.
If the decryption key and program using it are more secure than the database - which is pretty unusual, normally the database is in the most secure spot possible - then there would possibly be additional benefit to also encrypting the username as you'd be depriving attackers of useful information in brute force attacks.