GAE taskqueue access application storage - google-app-engine

My GAE application is written in Python with webapp2. My application targets at analyzing user's online social network. Users could login and authorize my application, hence the access token will be stored for further crawling the data. Then i use the taskqueue to launch a backend task, as the crawling process is time consuming. However, when i access the datastore to fetch the access token, i can get it. I wonders whether there is a way to access the data of the frontend, rather than the temporary storage for the taskqueue.
the handler to the process http request from the user
class Callback(webapp2.RequestHandler):
def get(self):
global client
global r
code = self.request.get('code')
try:
client = APIClient(app_key=APP_KEY, app_secret=APP_SECRET,redirect_uri=CALLBACK_URL)
r = client.request_access_token(code)
access_token = r.access_token
record = model.getAccessTokenByUid(r.uid)
if record is None or r.access_token != record.accessToken:
# logging.debug("access token stored")
**model.insertAccessToken(long(r.uid), access_token, r.expires_in, "uncrawled", datetime.datetime.now())** #data stored here
session = self.request.environ['beaker.session']
session['uid'] = long(r.uid)
self.redirect(CLUSTER_PAGE % ("true"))
except Exception, e:
logging.error("callback:%s" % (str(e)));
self.redirect(CLUSTER_PAGE % ("false"))
the handle to process task submitted to taskqueue
class CrawlWorker(webapp2.RequestHandler):
def post(self): # should run at most 1/s
uid = self.request.get('uid')
logging.debug("start crawling uid:%s in the backend" % (str(uid)))
global client
global client1
global r
tokenTuple = model.getAccessTokenByUid(uid)
if tokenTuple is None: **#here i always get a None**
logging.error("CounterWorker:oops, authorization token is missed.")
return

The question is not clear (is it can or cant?) But if you want to access frontend data from the taskqueue, pass it as parameters to the task queue.

Related

How to get a session's key with webapp2 and datastore?

I am trying to implement a 'remember me' feature on the login page.
This is the logic I have in mind: Store sessions in the datastore,
pass the session id/key to the client so that next time the user visits
the site, i get information from the datastore with the key the client has
I was about to do something like this:
class Session(ndb.Model):
username = ndb.StringProperty()
email = ndb.StringProperty()
if request.get('rememberme'):
session = Session()
session.email = 'john#doe.com'
session.username = 'jon snow'
key = session.put()
# send `key` back to client and store in a cookie, so
# when client visits the site again, get the session
# values from the datastore
self.response.write(key.id())
But I'm using this snippet from the docs to handle my sessions:
class BaseHandler(webapp2.RequestHandler):
def dispatch(self):
self.session_store = sessions.get_store(request=self.request)
try:
webapp2.RequestHandler.dispatch(self)
finally:
self.session_store.save_sessions(self.response)
#webapp2.cached_property
def session(self):
return self.session_store.get_session(name='foo', backend='datastore')
This too ,inserts data into the datastore under the 'Session' Kind.
So it feels like I'd be doing redudant work if I were to manually store sessions in the datastore as well.
When using webapp2's session with datastore as the backend, how can I know that session's id/key?

how does get_current_user work

I'm really confused how Google App Engine's User's get_current_user() works. I've looked around the internet at a bunch of different guides and tutorials about login and authentication, and many of them mention similar methods.
If there are a million users logged in to my application at the same time, how can that method possibly work? Does each user get their own instance of the server? How does the server know which client it is talking to?
It doesn't make sense to me at all.
When logging in (by clicking on the URL generated by create_login_url()) a cookie containing user identifying information is prepared and pushed on the client side, then used in subsequent requests until the user logs out or the cookie expires. Calling get_current_user() simply checks the cookie existance/information and responds accordingly.
On the development server the cookie is named dev_appserver_login. I can no longer check the cookie name on GAE as I switched away from the Users API.
The actual handling of the cookie seems to happen somewhere on the Users service backend, for example, by looking at the google/appengine/api/users.py file in the python SDK:
def create_login_url(dest_url=None, _auth_domain=None,
federated_identity=None):
...
req = user_service_pb.CreateLoginURLRequest()
resp = user_service_pb.CreateLoginURLResponse()
try:
apiproxy_stub_map.MakeSyncCall('user', 'CreateLoginURL', req, resp)
...
The end point (at least for the development server) seems to somehow land somewhere in google/appengine/tools/appengine_rpc.py, for example:
#staticmethod
def _CreateDevAppServerCookieData(email, admin):
"""Creates cookie payload data.
Args:
email: The user's email address.
admin: True if the user is an admin; False otherwise.
Returns:
String containing the cookie payload.
"""
if email:
user_id_digest = hashlib.md5(email.lower()).digest()
user_id = "1" + "".join(["%02d" % ord(x) for x in user_id_digest])[:20]
else:
user_id = ""
return "%s:%s:%s" % (email, bool(admin), user_id)
def _DevAppServerAuthenticate(self):
"""Authenticates the user on the dev_appserver."""
credentials = self.auth_function()
value = self._CreateDevAppServerCookieData(credentials[0], True)
self.extra_headers["Cookie"] = ('dev_appserver_login="%s"; Path=/;' % value)

Using webapp2 session as namespace name. How?

I have an application which is school based. Each tenant is a different school and to access the application all users for each school have the same password.
Alongside this each school user has to have a google email if they want access to the application. So the application first checks they are a google user, checks wether they are a school user and finally checks that their google email is in the school user list before they are allowed access to any page.
The school user part is using session data from webapp2 sessions to ensure each request they have appropriate access
class Handler(webapp2.RequestHandler):
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
try:
# Dispatch the request.
webapp2.RequestHandler.dispatch(self)
finally:
# Save all sessions.
self.session_store.save_sessions(self.response)
#webapp2.cached_property
def session(self):
# Returns a session using the default cookie key.
return self.session_store.get_session()
When a user logins I check the password then create a session which checks their password / user combination every request.
def check_u(self):
try:
uid = self.session.get('user')
parent = self.session.get('school-id')
udl = m.models.User.by_id(int(uid),parent)
if uid and udl:
return udl
else:
return False
except (TypeError,AttributeError):
return False
A parent datastore entity for each different school is used called MetaSchool which I have been currently using to ensure that there is no data leak across schools. Each datastore entry uses this parent session key as a way of setting the datastore entry with MetaSchool as parent then using this session key again to read back this data.
This method works but is onerous. I would like to use namespace as a way of separating the data but would like to use the Metaschool id as the name.
def namespace_manager_default_namespace_for_request():
### Here I need to get ------ parent = self.session.get('school-id')
### use this session to gain the MetaSchool key id
### Set this as the namespace name
Basically trying to emulate from the docs the below scenario
from google.appengine.api import users
def namespace_manager_default_namespace_for_request():
# assumes the user is logged in.
return users.get_current_user().user_id()
I am having difficulty getting the session data from Handler object???
Any thoughts
This is what I came up with.
from google.appengine.api import namespace_manager
from webapp2_extras import sessions
def namespace_manager_default_namespace_for_request():
session = sessions.get_store()
s = session.get_session()
name = s.get('key')
if name:
return name
else:
return namespace_manager.set_namespace('string')

Facebook login in Google Cloud Endpoints

Can someone explain steps to implement login process with other OAuth2 providers
This link Google Cloud Endpoints with another oAuth2 provider gives little info about writing custom authentication, but I guess for beginner like me that's not enough, please give detailed steps.
Especially, interested in Facebook.
You need to implement Facebook's client side APIs according to their documentation and the environment you are deploying your client app to (Browser vs iOS vs Android). This includes registering your app with them. Your registered app will direct the user to go through an authentication flow and at the end of it your client app will have access to a short-lived access token. Facebook has multiple types of access tokens, but the one it sounds like you're interested in is called a User Access Token since it identifies an authorized user.
Pass the access token to your Cloud Endpoints API via a field or header. Inside of your API code receive the access token and implement Facebook's API that checks the validity of the access token. The first answer on this SO question makes it look rather easy, but you probably want to reference their documentation again. If that check passes then you would run your API code, otherwise throw an exception.
You will typically also want to implement a caching mechanism to prevent calling the Facebook server side validation API for each Cloud Endpoints request.
Finally, I mentioned that your client app has a short lived token. If you have a client app that is browser-based then you will probably want to upgrade that to a long lived token. Facebook has a flow for that as well which involves your API code requesting a long lived token with the short lived one. You would then need to transfer that long lived token back to the client app to use for future Cloud Endpoints API calls.
If your client app is iOS or Android based then your tokens are managed by Facebook code and you simply request access tokens from the respective APIs when you need them.
So I actually tried to implement that custom authentication flow. It seems working fine although there might be further consideration on security side.
First, user go to my application and authenticate with facebook, the application got his user_id and access_token. Then the application call auth API to the server with these info.
class AuthAPI(remote.Service):
#classmethod
def validate_facebook_user(cls, user_id, user_token):
try:
graph = facebook.GraphAPI(user_token)
profile = graph.get_object("me", fields='email, first_name, last_name, username')
except facebook.GraphAPIError, e:
return (None, None, str(e))
if (profile is not None):
# Check if match user_id
if (profile.get('id', '') == user_id):
# Check if user exists in our own datastore
(user, token) = User.get_by_facebook_id(user_id, 'auth', user_token)
# Create new user if not
if user is None:
#print 'Create new user'
username = profile.get('username', '')
password = security.generate_random_string(length=20)
unique_properties = ['email_address']
if (username != ''):
(is_created, user) = User.create_user(
username,
unique_properties,
email_address = profile.get('email', ''),
name = profile.get('first_name', ''),
last_name = profile.get('last_name', ''),
password_raw = password,
facebook_id = user_id,
facebook_token = user_token,
verified=False,
)
if is_created==False:
return (None, None, 'Cannot create user')
token_str = User.create_auth_token(user.get_id())
#print (user, token_str)
# Return if user exists
if token is not None:
return (user, token.token, 'Successfully logged in')
else:
return (None, None, 'Invalid token')
return (None, None, 'Invalid facebook id and token')
# Return a user_id and token if authenticated successfully
LOGIN_REQ = endpoints.ResourceContainer(MessageCommon,
type=messages.StringField(2, required=True),
user_id=messages.StringField(3, required=False),
token=messages.StringField(4, required=False))
#endpoints.method(LOGIN_REQ, MessageCommon,
path='login', http_method='POST', name='login')
def login(self, request):
type = request.type
result = MessageCommon()
# TODO: Change to enum type if we have multiple auth ways
if (type == "facebook"):
# Facebook user validation
user_id = request.user_id
access_token = request.token
(user_obj, auth_token, msg) = self.validate_facebook_user(user_id, access_token)
# If we can get user data
if (user_obj is not None and auth_token is not None):
print (user_obj, auth_token)
result.success = True
result.message = msg
result.data = json.dumps({
'user_id': user_obj.get_id(),
'user_token': auth_token
})
# If we cannot
else:
result.success = False
result.message = msg
return result
In addition to this, you might want to implement the normal user authentication flow following instruction here: http://blog.abahgat.com/2013/01/07/user-authentication-with-webapp2-on-google-app-engine/ .
This is because the user_id and user_token that I obtain was provided by webapp2_extras.appengine.auth.
Implementation of User.get_by_facebook_id:
class User(webapp2_extras.appengine.auth.models.User):
#classmethod
def get_by_facebook_id(cls, fb_id, subj='auth', fb_token=""):
u = cls.query(cls.facebook_id==fb_id).get()
if u is not None:
user_id = u.key.id()
# TODO: something better here, now just append the facebook_token to a prefix
token_str = "fbtk" + str(fb_token)
# get this token if it exists
token_key = cls.token_model.get(user_id, subj, token_str)
print token_key, fb_token
if token_key is None:
# return a token that created from access_token string
if (fb_token == ""):
return (None, None)
else:
token = cls.token_model.create(user_id, subj, token_str)
else:
token = token_key
return (u, token)
return (None, None)
Server verify if the user is authenticated with facebook once more time. If it passes, user is considered logged in. In this case, server pass back a user_token (generated based on facebook_token) and user_id from our datastore.
Any further API calls should use this user_id and user_token
def get_request_class(messageCls):
return endpoints.ResourceContainer(messageCls,
user_id=messages.IntegerField(2, required=False),
user_token=messages.StringField(3, required=False))
def authenticated_required(endpoint_method):
"""
Decorator that check if API calls are authenticated
"""
def check_login(self, request, *args, **kwargs):
try:
user_id = request.user_id
user_token = request.user_token
if (user_id is not None and user_token is not None):
# Validate user
(user, timestamp) = User.get_by_auth_token(user_id, user_token)
if user is not None:
return endpoint_method(self, request, user, *args, **kwargs )
raise endpoints.UnauthorizedException('Invalid user_id or access_token')
except:
raise endpoints.UnauthorizedException('Invalid access token')
#endpoints.api(name='blah', version='v1', allowed_client_ids = env.CLIENT_IDS, auth=AUTH_CONFIG)
class BlahApi(remote.Service):
# Add user_id/user_token to the request
Blah_Req = get_request_class(message_types.VoidMessage)
#endpoints.method(Blah_Req, BlahMessage, path='list', name='list')
#authenticated_required
def blah_list(self, request, user):
newMessage = BlahMessage(Blah.query().get())
return newMessage
Note:
I am using this library to handle facebook authentication checking on server: https://github.com/pythonforfacebook/facebook-sdk
I implemented this use case by adding a webapp2 handler to exchange the Facebook access token for one generated by my own application, using the SimpleAuth mixin for verification:
class AuthHandler(webapp2.RequestHandler, SimpleAuthHandler):
"""Authenticates a user to the application via a third-party provider.
The return value of this request is an OAuth token response.
Only a subset of the PROVIDERS specified in SimpleAuthHandler are currently supported.
Tested providers: Facebook
"""
def _on_signin(self, data, auth_info, provider):
# Create the auth ID format used by the User model
auth_id = '%s:%s' % (provider, data['id'])
user_model = auth.get_auth().store.user_model
user = user_model.get_by_auth_id(auth_id)
if not user:
ok, user = user_model.create_user(auth_id)
if not ok:
logging.error('Unable to create user for auth_id %s' % auth_id)
self.abort(500, 'Unable to create user')
return user
def post(self):
# Consider adding a check for a valid endpoints client ID here as well.
access_token = self.request.get('x_access_token')
provider = self.request.get('x_provider')
if provider not in self.PROVIDERS or access_token is None:
self.abort(401, 'Unknown provider or access token')
auth_info = {'access_token': access_token}
fetch_user_info = getattr(self, '_get_%s_user_info' % provider)
user_info = fetch_user_info(auth_info)
if 'id' in user_info:
user = self._on_signin(user_info, auth_info, provider)
token = user.create_bearer_token(user.get_id())
self.response.content_type = 'application/json'
self.response.body = json.dumps({
'access_token': token.token,
'token_type': 'Bearer',
'expires_in': token.bearer_token_timedelta.total_seconds(),
'refresh_token': token.refresh_token
})
else:
self.abort(401, 'Access token is invalid')
The exchanged access token can be passed on each endpoints request in the Authorization header, or as part of the RPC message if you prefer. Here's an example of reading it from the header:
def get_current_user():
token = os.getenv('HTTP_AUTHORIZATION')
if token:
try:
token = token.split(' ')[1]
except IndexError:
pass
user, _ = User.get_by_bearer_token(token)
return user
I posted the complete example on Github: https://github.com/loudnate/appengine-endpoints-auth-example
So no body has thrown a light on the android client side stuff. Since, you do not require Google login in this case hence the code for getting api handle will look like:
private Api getEndpointsApiHandle() {
Api.Builder api = new Api.Builder(HTTP_TRANSPORT, JSON_FACTORY, null);
api.setRootUrl(yourRootUrl);
return api.build();
}
If you notice; You will require to pass null as the Credential. This code works like a charm
I too have written my own solution for this problem. You can check out the code here: https://github.com/rggibson/Authtopus
Authtopus is a python library for custom authentication with Google Cloud Endpoints. It supports basic username and password registrations + logins, as well as logins via Facebook and Google (and could probably be extended to support other social providers without too much hassle). I know this doesn't directly answer the original question, but it seems related enough that I thought I'd share.

Detecting first time login of user into application (Google Appengine)

My app requires users to login using their google account.
I have this set in my App.yamp file:
url: /user/.*
script: user.py
login: required
Now when any user tries to access files under /user/secret.py he will need to authenticate via google, which will redirect the user back to /user/secret.py after successful authentication. Now the problem I am facing is when the user is redirected back to the app, I cannot be sure if this is the first time the user has logged in or is it a regular user to my site who has come back again from just the user object which google passes using users.get_current_user() .
I thus need to maintain state in the datastore to check if the user already exists or not everytime. If he does not exist i need to create a new entry with other application specific settings.
My question is: Is there some easier way to handle this? without having to query the datastore to figure if this is a first time user or a regular one?
No, Google doesn't keep track of if a user has logged in to your app before. Since you presumably need to store some sort of state against the user, the simplest way is to try and retrieve the user's record from the datastore. If they don't have one, you can send them to the registration screen to gather this information. You can use memcache to cache a user's information and avoid extra datastore round-trips.
I tend to use my own user and session manangement
For my web handlers I will attach a decorator called session and one called authorize. The session decorator will attach a session to every request, and the authorize decorator will make sure that the user is authorised
(A word of caution, the authorize decorator is specific to how I develop my applications - the username being the first parameter in most requests)
So for example a web handler may look like:
class UserProfile(webapp.RequestHandler):
#session
#authorize
def get(self, user):
# Do some funky stuff
# The session is attached to the self object.
someObjectAttachedToSession = self.SessionObj.SomeStuff
self.response.out.write("hello %s" % user)
In the above code, the session decorator attaches some session stuff that I need based on the cookies that are present on the request. The authorize header will make sure that the user can only access the page if the session is the correct one.
The decorators code are below:
import functools
from model import Session
import logging
def authorize(redirectTo = "/"):
def factory(method):
'Ensures that when an auth cookie is presented to the request that is is valid'
#functools.wraps(method)
def wrapper(self, *args, **kwargs):
#Get the session parameters
auth_id = self.request.cookies.get('auth_id', '')
session_id = self.request.cookies.get('session_id', '')
#Check the db for the session
session = Session.GetSession(session_id, auth_id)
if session is None:
self.redirect(redirectTo)
return
else:
if session.settings is None:
self.redirect(redirectTo)
return
username = session.settings.key().name()
if len(args) > 0:
if username != args[0]:
# The user is allowed to view this page.
self.redirect(redirectTo)
return
result = method(self, *args, **kwargs)
return result
return wrapper
return factory
def session(method):
'Ensures that the sessions object (if it exists) is attached to the request.'
#functools.wraps(method)
def wrapper(self, *args, **kwargs):
#Get the session parameters
auth_id = self.request.cookies.get('auth_id', '')
session_id = self.request.cookies.get('session_id', '')
#Check the db for the session
session = Session.GetSession(session_id, auth_id)
if session is None:
session = Session()
session.session_id = Session.MakeId()
session.auth_token = Session.MakeId()
session.put()
# Attach the session to the method
self.SessionObj = session
#Call the handler.
result = method(self, *args, **kwargs)
self.response.headers.add_header('Set-Cookie', 'auth_id=%s; path=/; HttpOnly' % str(session.auth_token))
self.response.headers.add_header('Set-Cookie', 'session_id=%s; path=/; HttpOnly' % str(session.session_id))
return result
return wrapper
def redirect(method, redirect = "/user/"):
'When a known user is logged in redirect them to their home page'
#functools.wraps(method)
def wrapper(self, *args, **kwargs):
try:
if self.SessionObj is not None:
if self.SessionObj.settings is not None:
# Check that the session is correct
username = self.SessionObj.settings.key().name()
self.redirect(redirect + username)
return
except:
pass
return method(self, *args, **kwargs)
return wrapper
Can you not just set a Cookie the first time the user logs in and check for this? If they're a new user it won't be there and but if they're an old user it will be. It's not 100% accurate since some users might clear their cookies but it might do depending on what it is you want to achieve.
If you're using Django in your application managing Cookies is pretty straightforward.
I agree that managing your own authenticated users is the best way to approach this problem. Depending on your application scope obviously but at the very least an AuthUser(Model) class that contains the UserProperty for the users that have logged in with your account.
...
class AuthUser(db.Model):
user = UserProperty(required=True)
...
Then when a user logs in just
...
user = users.get_current_user()
user_exists = AuthUser.gql('where user = :1', user) # or easy check db.GqlQuery("select __key__ from AuthUser where user = :1", user)
if user_exists:
# do user has been before stuff
else:
# do first time user stuff
...
Alternately a super easy way to do this is have a Model for your site that has a ListProperty(users.User) and then you can easily check the list to see if the user has been into your app before.
...
class SiteStuff(db.Model):
auth_users = ListProperty(users.User)
...
and when they log in: check if they are in the list; if not, you add them to the list, put() it and do whatever you need to do for first time users. If you find them in there then do the other stuff.

Resources