Getting Google App stats with python - google-app-engine

I am trying to scrape some web content to get stastics from google appstats that my google app generated. Please note this is different then google analytics. I am using python 2.7.5.
The issue I am facing is the initial google authentication in my request.
I have the apis I need to call from google app stats but I keep getting a DENY in response back when I am using my own google appengine account credentials. This results in a redirect to accounts.google.com page.
I have tried several different approaches without a successfull login to accounts.google.com.
Anyone have any ideas on this?
More helpfull would be if you can point me to some good reference material
Thanks

This code sample will allow you to get content of /secure page which protected by google login. Do not forget to set email, password and application ID. Then you can fetch other protected pages using this opener.
import urllib
import urllib2
import cookielib
import logging
EMAIL = ''
PASSWORD = ''
APPID = 'YOURAPPID'
# Setup to be able to get the needed cookies that GAE returns
cookiejar = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
urllib2.install_opener(opener)
# This is the setup to construct the login URL for authentication.
authreq_data = urllib.urlencode({'Email': EMAIL,
'Passwd': PASSWORD,
'service': 'ah',
'source': '',
'accountType': 'HOSTED_OR_GOOGLE'})
# Get an AuthToken from Google Accounts
auth_req = urllib2.Request('https://www.google.com/accounts/ClientLogin',
data=authreq_data)
try:
auth_resp = opener.open(auth_req)
logging.info('Successful authorization as %s' % EMAIL)
except urllib2.HTTPError:
logging.warning('Authorization as %s failed. '
'Please, check your email and password' % EMAIL)
auth_resp_body = auth_resp.read()
auth_resp_dict = dict(x.split('=')
for x in auth_resp_body.split('\n') if x)
authtoken = auth_resp_dict['Auth']
authreq_data = urllib.urlencode({'continue': 'http://%s.appspot.com/secure' % APPID,
'auth': authtoken})
login_uri = ('http://%s.appspot.com/_ah/login?%s' % (APPID, authreq_data))
# Do the actual login and getting the cookies.
print opener.open(urllib2.Request(login_uri)).read()

Related

React, Django: How do I manage a session after a user changes their password?

I am developing an administration panel in React with Django for user management, amongst other actions.
Django takes care of ORM and users' login.When the user first logs in, the React app obtains the session id and the csrf token. The app asks for the csrf token from the document cookies and that's the token used for later REST queries.
Issue/Challenge:
One of the app component allows a user to change the password. However, after a successful password change doing so, the session id and the csrf token become expired and the REST queries I do from that time on are unauthorized.
If I reload the page, after then, the session is expired. I tried to get the token again but it doesn't work.
Question:
How can I update the session id after changing the password?
Should I just send the user outside the app for them to log in again?
I don't know if that is the common way to proceded.
Below is the function that I use to get the token:
getCookie(name) {
console.log(name);
let cookieValue = null;
if (document.cookie && document.cookie !== '') {
const cookies = document.cookie.split(';');
for (let i = 0; i < cookies.length; i += 1) {
const cookie = jQuery.trim(cookies[i]);
// Does this cookie string begin with the name we want?
if (cookie.substring(0, name.length + 1) === (`${name}=`)) {
cookieValue = decodeURIComponent(cookie.substring(name.length + 1));
break;
}
}
}
console.log(cookieValue);
this.setState({
token: cookieValue,
});
//getCookie('csrftoken')
I don't usually work with cookies, apart from getting the csrf token, so I would appreciate any help.
Update
I found the problem might be due to Django: https://docs.djangoproject.com/en/3.1/topics/auth/default/#session-invalidation-on-password-change
As I am changing the password with Django Rest Framework -- following this tutorial: https://medium.com/django-rest/django-rest-framework-change-password-and-update-profile-1db0c144c0a3 -- I used the function update_session_auth_hash in the serializer as follows:
## serializer
def update(self, instance, validated_data):
instance.set_password(validated_data['password'])
instance.save()
request = self.context['request']
user = request.user
update_session_auth_hash(request, user)
return instance
I doesn't work either...
I finally managed to make it work by using instance instead of user:
## serializer
def update(self, instance, validated_data):
instance.set_password(validated_data['password'])
instance.save()
request = self.context['request']
update_session_auth_hash(request, instance)
return instance

how does get_current_user work

I'm really confused how Google App Engine's User's get_current_user() works. I've looked around the internet at a bunch of different guides and tutorials about login and authentication, and many of them mention similar methods.
If there are a million users logged in to my application at the same time, how can that method possibly work? Does each user get their own instance of the server? How does the server know which client it is talking to?
It doesn't make sense to me at all.
When logging in (by clicking on the URL generated by create_login_url()) a cookie containing user identifying information is prepared and pushed on the client side, then used in subsequent requests until the user logs out or the cookie expires. Calling get_current_user() simply checks the cookie existance/information and responds accordingly.
On the development server the cookie is named dev_appserver_login. I can no longer check the cookie name on GAE as I switched away from the Users API.
The actual handling of the cookie seems to happen somewhere on the Users service backend, for example, by looking at the google/appengine/api/users.py file in the python SDK:
def create_login_url(dest_url=None, _auth_domain=None,
federated_identity=None):
...
req = user_service_pb.CreateLoginURLRequest()
resp = user_service_pb.CreateLoginURLResponse()
try:
apiproxy_stub_map.MakeSyncCall('user', 'CreateLoginURL', req, resp)
...
The end point (at least for the development server) seems to somehow land somewhere in google/appengine/tools/appengine_rpc.py, for example:
#staticmethod
def _CreateDevAppServerCookieData(email, admin):
"""Creates cookie payload data.
Args:
email: The user's email address.
admin: True if the user is an admin; False otherwise.
Returns:
String containing the cookie payload.
"""
if email:
user_id_digest = hashlib.md5(email.lower()).digest()
user_id = "1" + "".join(["%02d" % ord(x) for x in user_id_digest])[:20]
else:
user_id = ""
return "%s:%s:%s" % (email, bool(admin), user_id)
def _DevAppServerAuthenticate(self):
"""Authenticates the user on the dev_appserver."""
credentials = self.auth_function()
value = self._CreateDevAppServerCookieData(credentials[0], True)
self.extra_headers["Cookie"] = ('dev_appserver_login="%s"; Path=/;' % value)

GAE Python Webapp2 auth token doesn't register user nor delete old tokens in production

I'm using webapp2 for user authentication of Google App Engine, based on this article: http://blog.abahgat.com/2013/01/07/user-authentication-with-webapp2-on-google-app-engine/
Locally through the SDK, everything works exactly as expected. However when I deploy to production, the entities under UserToken only have {} as value for user. Because of this I can't properly use the authentication tokens.
What could influence this? The website is only accessible through HTTPS, but I can't imagine that that is causing it. Any input would be very much appreciated!
EDIT:
Adding requested code.
User model
import webapp2_extras.appengine.auth.models
from google.appengine.ext import ndb
from webapp2_extras import security
class User(webapp2_extras.appengine.auth.models.User):
def set_password(self, raw_password):
"""Sets the password for the current user
:param raw_password:
The raw password which will be hashed and stored
"""
self.password = security.generate_password_hash(raw_password, length=12)
#classmethod
def get_by_auth_token(cls, user_id, token, subject='auth'):
"""Returns a user object based on a user ID and token.
:param user_id:
The user_id of the requesting user.
:param token:
The token string to be verified.
:returns:
A tuple ``(User, timestamp)``, with a user object and
the token timestamp, or ``(None, None)`` if both were not found.
"""
token_key = cls.token_model.get_key(user_id, subject, token)
user_key = ndb.Key(cls, user_id)
# Use get_multi() to save a RPC call.
valid_token, user = ndb.get_multi([token_key, user_key])
if valid_token and user:
timestamp = int(time.mktime(valid_token.created.timetuple()))
return user, timestamp
return None, None
When creating the user, I use following code:
from webapp2_extras import sessions, auth
import webapp2_extras.appengine.auth.models
unique_properties = ['email_address']
user_data = self.user_model.create_user(email, unique_properties,
email_address=email, first_name=first_name, password_raw=password, verified=False)
The code that is then used to create a token for 'user' is:
user_id = user.get_id()
token = self.user_model.create_signup_token(user_id)
The problem I have exists in the signup process as well as e.g. the forgot password process. As far as I can see, this is where something goes wrong. Not sure what though.
I've checked to make sure that an id is actually passed, and it is. Funny enough, the keyname of the token does include the user id, however on production the user field is left empty. Working locally it does include the user field.

Facebook login in Google Cloud Endpoints

Can someone explain steps to implement login process with other OAuth2 providers
This link Google Cloud Endpoints with another oAuth2 provider gives little info about writing custom authentication, but I guess for beginner like me that's not enough, please give detailed steps.
Especially, interested in Facebook.
You need to implement Facebook's client side APIs according to their documentation and the environment you are deploying your client app to (Browser vs iOS vs Android). This includes registering your app with them. Your registered app will direct the user to go through an authentication flow and at the end of it your client app will have access to a short-lived access token. Facebook has multiple types of access tokens, but the one it sounds like you're interested in is called a User Access Token since it identifies an authorized user.
Pass the access token to your Cloud Endpoints API via a field or header. Inside of your API code receive the access token and implement Facebook's API that checks the validity of the access token. The first answer on this SO question makes it look rather easy, but you probably want to reference their documentation again. If that check passes then you would run your API code, otherwise throw an exception.
You will typically also want to implement a caching mechanism to prevent calling the Facebook server side validation API for each Cloud Endpoints request.
Finally, I mentioned that your client app has a short lived token. If you have a client app that is browser-based then you will probably want to upgrade that to a long lived token. Facebook has a flow for that as well which involves your API code requesting a long lived token with the short lived one. You would then need to transfer that long lived token back to the client app to use for future Cloud Endpoints API calls.
If your client app is iOS or Android based then your tokens are managed by Facebook code and you simply request access tokens from the respective APIs when you need them.
So I actually tried to implement that custom authentication flow. It seems working fine although there might be further consideration on security side.
First, user go to my application and authenticate with facebook, the application got his user_id and access_token. Then the application call auth API to the server with these info.
class AuthAPI(remote.Service):
#classmethod
def validate_facebook_user(cls, user_id, user_token):
try:
graph = facebook.GraphAPI(user_token)
profile = graph.get_object("me", fields='email, first_name, last_name, username')
except facebook.GraphAPIError, e:
return (None, None, str(e))
if (profile is not None):
# Check if match user_id
if (profile.get('id', '') == user_id):
# Check if user exists in our own datastore
(user, token) = User.get_by_facebook_id(user_id, 'auth', user_token)
# Create new user if not
if user is None:
#print 'Create new user'
username = profile.get('username', '')
password = security.generate_random_string(length=20)
unique_properties = ['email_address']
if (username != ''):
(is_created, user) = User.create_user(
username,
unique_properties,
email_address = profile.get('email', ''),
name = profile.get('first_name', ''),
last_name = profile.get('last_name', ''),
password_raw = password,
facebook_id = user_id,
facebook_token = user_token,
verified=False,
)
if is_created==False:
return (None, None, 'Cannot create user')
token_str = User.create_auth_token(user.get_id())
#print (user, token_str)
# Return if user exists
if token is not None:
return (user, token.token, 'Successfully logged in')
else:
return (None, None, 'Invalid token')
return (None, None, 'Invalid facebook id and token')
# Return a user_id and token if authenticated successfully
LOGIN_REQ = endpoints.ResourceContainer(MessageCommon,
type=messages.StringField(2, required=True),
user_id=messages.StringField(3, required=False),
token=messages.StringField(4, required=False))
#endpoints.method(LOGIN_REQ, MessageCommon,
path='login', http_method='POST', name='login')
def login(self, request):
type = request.type
result = MessageCommon()
# TODO: Change to enum type if we have multiple auth ways
if (type == "facebook"):
# Facebook user validation
user_id = request.user_id
access_token = request.token
(user_obj, auth_token, msg) = self.validate_facebook_user(user_id, access_token)
# If we can get user data
if (user_obj is not None and auth_token is not None):
print (user_obj, auth_token)
result.success = True
result.message = msg
result.data = json.dumps({
'user_id': user_obj.get_id(),
'user_token': auth_token
})
# If we cannot
else:
result.success = False
result.message = msg
return result
In addition to this, you might want to implement the normal user authentication flow following instruction here: http://blog.abahgat.com/2013/01/07/user-authentication-with-webapp2-on-google-app-engine/ .
This is because the user_id and user_token that I obtain was provided by webapp2_extras.appengine.auth.
Implementation of User.get_by_facebook_id:
class User(webapp2_extras.appengine.auth.models.User):
#classmethod
def get_by_facebook_id(cls, fb_id, subj='auth', fb_token=""):
u = cls.query(cls.facebook_id==fb_id).get()
if u is not None:
user_id = u.key.id()
# TODO: something better here, now just append the facebook_token to a prefix
token_str = "fbtk" + str(fb_token)
# get this token if it exists
token_key = cls.token_model.get(user_id, subj, token_str)
print token_key, fb_token
if token_key is None:
# return a token that created from access_token string
if (fb_token == ""):
return (None, None)
else:
token = cls.token_model.create(user_id, subj, token_str)
else:
token = token_key
return (u, token)
return (None, None)
Server verify if the user is authenticated with facebook once more time. If it passes, user is considered logged in. In this case, server pass back a user_token (generated based on facebook_token) and user_id from our datastore.
Any further API calls should use this user_id and user_token
def get_request_class(messageCls):
return endpoints.ResourceContainer(messageCls,
user_id=messages.IntegerField(2, required=False),
user_token=messages.StringField(3, required=False))
def authenticated_required(endpoint_method):
"""
Decorator that check if API calls are authenticated
"""
def check_login(self, request, *args, **kwargs):
try:
user_id = request.user_id
user_token = request.user_token
if (user_id is not None and user_token is not None):
# Validate user
(user, timestamp) = User.get_by_auth_token(user_id, user_token)
if user is not None:
return endpoint_method(self, request, user, *args, **kwargs )
raise endpoints.UnauthorizedException('Invalid user_id or access_token')
except:
raise endpoints.UnauthorizedException('Invalid access token')
#endpoints.api(name='blah', version='v1', allowed_client_ids = env.CLIENT_IDS, auth=AUTH_CONFIG)
class BlahApi(remote.Service):
# Add user_id/user_token to the request
Blah_Req = get_request_class(message_types.VoidMessage)
#endpoints.method(Blah_Req, BlahMessage, path='list', name='list')
#authenticated_required
def blah_list(self, request, user):
newMessage = BlahMessage(Blah.query().get())
return newMessage
Note:
I am using this library to handle facebook authentication checking on server: https://github.com/pythonforfacebook/facebook-sdk
I implemented this use case by adding a webapp2 handler to exchange the Facebook access token for one generated by my own application, using the SimpleAuth mixin for verification:
class AuthHandler(webapp2.RequestHandler, SimpleAuthHandler):
"""Authenticates a user to the application via a third-party provider.
The return value of this request is an OAuth token response.
Only a subset of the PROVIDERS specified in SimpleAuthHandler are currently supported.
Tested providers: Facebook
"""
def _on_signin(self, data, auth_info, provider):
# Create the auth ID format used by the User model
auth_id = '%s:%s' % (provider, data['id'])
user_model = auth.get_auth().store.user_model
user = user_model.get_by_auth_id(auth_id)
if not user:
ok, user = user_model.create_user(auth_id)
if not ok:
logging.error('Unable to create user for auth_id %s' % auth_id)
self.abort(500, 'Unable to create user')
return user
def post(self):
# Consider adding a check for a valid endpoints client ID here as well.
access_token = self.request.get('x_access_token')
provider = self.request.get('x_provider')
if provider not in self.PROVIDERS or access_token is None:
self.abort(401, 'Unknown provider or access token')
auth_info = {'access_token': access_token}
fetch_user_info = getattr(self, '_get_%s_user_info' % provider)
user_info = fetch_user_info(auth_info)
if 'id' in user_info:
user = self._on_signin(user_info, auth_info, provider)
token = user.create_bearer_token(user.get_id())
self.response.content_type = 'application/json'
self.response.body = json.dumps({
'access_token': token.token,
'token_type': 'Bearer',
'expires_in': token.bearer_token_timedelta.total_seconds(),
'refresh_token': token.refresh_token
})
else:
self.abort(401, 'Access token is invalid')
The exchanged access token can be passed on each endpoints request in the Authorization header, or as part of the RPC message if you prefer. Here's an example of reading it from the header:
def get_current_user():
token = os.getenv('HTTP_AUTHORIZATION')
if token:
try:
token = token.split(' ')[1]
except IndexError:
pass
user, _ = User.get_by_bearer_token(token)
return user
I posted the complete example on Github: https://github.com/loudnate/appengine-endpoints-auth-example
So no body has thrown a light on the android client side stuff. Since, you do not require Google login in this case hence the code for getting api handle will look like:
private Api getEndpointsApiHandle() {
Api.Builder api = new Api.Builder(HTTP_TRANSPORT, JSON_FACTORY, null);
api.setRootUrl(yourRootUrl);
return api.build();
}
If you notice; You will require to pass null as the Credential. This code works like a charm
I too have written my own solution for this problem. You can check out the code here: https://github.com/rggibson/Authtopus
Authtopus is a python library for custom authentication with Google Cloud Endpoints. It supports basic username and password registrations + logins, as well as logins via Facebook and Google (and could probably be extended to support other social providers without too much hassle). I know this doesn't directly answer the original question, but it seems related enough that I thought I'd share.

How to get Google profile info including custom fields from an Apps Domain user?

Using the user.profile and user.email scope and the /oauth2/v2/userinfo feed doesn't seem to return any custom fields (in my case Department) or phone numbers. These fields show up in the Domain Shared Contacts directory.
Is there perhaps an Apps Domain specific feed URL something like /oauth2/{DOMAIN}/v2/userinfo ?
Does the API/Service not support any custom fields yet?
Is there a way to fudge this into working?
Read access to your own Apps Domain Shared Contacts profile that's connected to your account shouldn't be so difficult.
I'd prefer a non-admin solution because my domain uses Common Access Cards w/ SAML authentication so I can't just store admin credentials (user : password) in an App Engine app and access the /m8/ feed. If there's a flow to access Domain Shared Contacts (with custom fields) with a beforehand authorized consumer key and secret I'd be interested in the instructions for getting that to work.
EDIT Jay Lee nailed it "https://www.google.com/m8/feeds/gal/{domain}/full"
Here's the proof of concept script using Google Apps Script (I'll add the final OAuth2 version when I finish it)
function getGal(email, passwd, domain) {
var res = UrlFetchApp.fetch("https://www.google.com/accounts/ClientLogin", {
contentType: "application/x-www-form-urlencoded",
method: "post",
payload: { "Email": email, "Passwd": passwd, "accountType": "HOSTED", "service":"cp" }
});
var auth = res.getContentText().match(/Auth=(.*)/i)[1];
Logger.log("Auth: " + auth);
res = UrlFetchApp.fetch("https://www.google.com/m8/feeds/gal/" + domain + "/full", {
method: "get",
headers: { "Authorization": "GoogleLogin auth=" + auth, "GData-Version": "1.0" }
});
Logger.log(res.getHeaders());
Logger.log(res.getContentText());
}
EDIT 2 OAuth version that returns JSON and only the info for the user accessing the script.
function googleOAuthM8() {
var oAuthConfig = UrlFetchApp.addOAuthService("m8");
oAuthConfig.setRequestTokenUrl('https://www.google.com/accounts/OAuthGetRequestToken?scope=https://www.google.com/m8/feeds/');
oAuthConfig.setAuthorizationUrl('https://www.google.com/accounts/OAuthAuthorizeToken');
oAuthConfig.setAccessTokenUrl('https://www.google.com/accounts/OAuthGetAccessToken');
oAuthConfig.setConsumerKey('anonymous');
oAuthConfig.setConsumerSecret('anonymous');
return {oAuthServiceName:"m8", oAuthUseToken:'always'};
}
function getGal(domain) {
res = UrlFetchApp.fetch("https://www.google.com/m8/feeds/gal/" + domain + "/full?alt=json&q=" + Session.getActiveUser().getEmail(), googleOAuthM8());
Logger.log(res.getHeaders());
Logger.log(res.getContentText());
}
Any non-admin user can access the GAL programmatically, see:
https://github.com/google/gfw-deployments/blob/master/apps/shell/gal/gal_feed.sh
I don't believe this API call is documented or supported officially but it works even with OAuth authentication rather than the example's ClientLogin (tested on the OAuth 2.0 playground with a non-admin user and the standard https://www.google.com/m8/feeds/ Contacts scope).
Note that the Global Address List is a compilation of user profiles, groups and shared contacts. You'll need to parse it out to find the user(s) you wish to get department information for.
I would utilize the Google Apps Profiles API to do this. It'll give you a bunch of meta information, including profile data and even profile photos:
https://developers.google.com/google-apps/profiles/
Even if you're using PIV/CAC/SAML, you will be able to auth using Two-Legged-OAuth.
https://developers.google.com/accounts/docs/OAuth#GoogleAppsOAuth
Two-legged-oauth is the path of least resistance, but you should also take a look at OAuth2, especially the JWT-signed service accounts portion -- however, it can be a little tricky to get working with the older GData xml apis.
As far as fields available go, you'll have to work with the ones on this page. There are extended properties where you add in arbitrary data, but they don't show up in the Contacts browser with Google Mail itself:
https://developers.google.com/gdata/docs/2.0/elements#gdProfileKind
On a sidenote, if you're in an LDAP environment (and since you mentioned CAC, I think you probably are), you should take a look at Google Apps Directory Sync, which can synchronize that profile data with your local AD/LDAP.
source: I deployed Google Apps to large organizations (3000+), public and private.
I have used the following approach with TwoLeggedOAuthHmacToken:
Consumer key and secret can be found in google apps admin dashboard
CONSUMER_KEY = 'domain.com'
CONSUMER_SECRET = 'secret_key'
class ContactClient():
def __init__(self, username):
# Contacts Data API Example ====================================================
self.requestor_id = username + '#' + CONSUMER_KEY
self.two_legged_oauth_token = gdata.gauth.TwoLeggedOAuthHmacToken(
CONSUMER_KEY, CONSUMER_SECRET, self.requestor_id)
self.contacts_client = gdata.contacts.client.ContactsClient(source=SOURCE_APP_NAME)
self.contacts_client.auth_token = self.two_legged_oauth_token
def newuser(self, username):
self.contacts_client.auth_token.requestor_id = username + '#' + CONSUMER_KEY
def getContacts(self, username=None):
if username:
self.newuser(username)
return self.contacts_client.GetContacts()
class MainPage(webapp2.RequestHandler):
def get(self):
contacts = ContactClient(username='username')
feed = contacts.getContacts()
output = ""
if feed:
for entry in feed.entry:
if entry.title and entry.title.text:
output += entry.title.text + "<br/>"
for email in entry.email:
if email.primary and email.primary == 'true':
output += ' %s<br/>' % (email.address)
self.response.headers['Content-Type'] = 'text/html'
self.response.write('''<h1>Contact Access via GData Client</h1>''' + output)

Resources