I am trying to consume IoTHub messages with EventHubConsumerClient in python.
(I also tried the csharp EventProcessorClient, it seems to have the same problem)
I am running the EventHubConsumerClient in a VM.
The setup:
IoT Hub accessible from the internet.
User assigned identity, added to VM.
User assigned identity has role "IoT Hub Data Reader" with scope set to the IoT Hub
BlobCheckpointStore connected to a blob storage, authenticated with the managed identity (I checked, it works)
Region West Europe
Everything works fine if I use the event hub connection string to connect to the IoT Hub.
But when I use managed identity credential, I get the following error:
evtconsumer | INFO:uamqp.c_uamqp:Token put complete with result: 3, status: 401, description: b'InvalidIssuer: Token issuer is invalid. TrackingId:0315ff67-60c5-4bb2-ba6d-160f45eb91eb, SystemTracker:NoSystemTracker, Timestamp:2021-09-15T10:30:05', connection: b'a44766f1-5d50-4d61-958e-52f7529315a4'
evtconsumer | INFO:uamqp.authentication.cbs_auth:Authentication status: 401, description: b'InvalidIssuer: Token issuer is invalid. TrackingId:0315ff67-60c5-4bb2-ba6d-160f45eb91eb, SystemTracker:NoSystemTracker, Timestamp:2021-09-15T10:30:05'
evtconsumer | INFO:uamqp.authentication.cbs_auth:Authentication Put-Token failed. Retrying.
My code:
import logging
import os
import sys
from azure.identity import ManagedIdentityCredential
from azure.eventhub import EventHubConsumerClient
from azure.eventhub.extensions.checkpointstoreblob import BlobCheckpointStore
logging.basicConfig(stream = sys.stdout, level = logging.INFO)
_logger = logging.getLogger()
azure_client_id = os.getenv("AZURE_CLIENT_ID")
evthubnamespace = os.getenv("IOTHUB_EVTHUB_FULLY_QUALIFIED_NAMESPACE")
evthubname = os.getenv("IOTHUB_EVTHUB_NAME")
evthubconnectionstring = os.getenv("IOTHUB_EVTHUB_CONNECTION_STRING")
blob_account_url = os.getenv("BLOB_ACCOUNT_URL")
blob_container_name = os.getenv("BLOB_CONTAINER_NAME")
# for toggling between authentication methods:
use_connection_string = os.getenv("USE_CONNECTION_STRING") == "true"
credential = ManagedIdentityCredential(client_id=azure_client_id)
def on_event(partition_context, event):
# Print the event data.
_logger.info("Received the event: \"{}\" from the partition with ID: \"{}\"".format(event.body_as_str(encoding='UTF-8'), partition_context.partition_id))
# Update the checkpoint so that the program doesn't read the events
# that it has already read when you run it next time.
partition_context.update_checkpoint(event)
def main():
# Create an Azure blob checkpoint store to store the checkpoints.
checkpoint_store = BlobCheckpointStore(
credential=credential,
blob_account_url=blob_account_url,
container_name=blob_container_name)
if use_connection_string:
# this works fine
_logger.info("Using connection string")
client = EventHubConsumerClient.from_connection_string(
evthubconnectionstring,
consumer_group="$Default",
checkpoint_store=checkpoint_store)
else:
# This causes errors
_logger.info(f"Using managed identity. fully_qualified_namespace: {evthubnamespace} eventhub_name: {evthubname}")
client = EventHubConsumerClient(
fully_qualified_namespace=evthubnamespace,
eventhub_name=evthubname,
consumer_group="$Default",
checkpoint_store=checkpoint_store,
credential=credential)
with client:
# Call the receive method. Read from the beginning of the partition (starting_position: "-1")
client.receive(on_event=on_event)
if __name__ == '__main__':
main()
I am all out of ideas with this one. It seems the AMQP event hub interface of the IoT hub does not accept the tokens generated from the managed identity credential?
Related
I'm using gmail API to insert an email to the inbox. When the From domain is outside the organization, I expect the external label to be added but it's not.
I'm using python and everything else is working, I am using a service account and able to impersonate on behalf the user's email and I'm using labelIds of ['INBOX', 'UNREAD'] and I need the newly external label as well but couldn't figure a way to add it through the API.
This feature is turned ON for the workspace.
Update - adding my code:
from googleapiclient import discovery, errors
from google.oauth2 import service_account
from email.mime.text import MIMEText
import base64
SERVICE_ACCOUNT_FILE = 'insert-messages-91e77b62878f.json'
SCOPES = ['https://www.googleapis.com/auth/gmail.insert']
def validationService():
# Set the credentials
credentials = service_account.Credentials.\
from_service_account_file(SERVICE_ACCOUNT_FILE, scopes= SCOPES)
# Delegate the credentials to the user you want to impersonate
delegated_credentials = credentials.with_subject('<some_user>')
service = discovery.build('gmail', 'v1', credentials=delegated_credentials)
return service
def SendMessage(service, message):
message = service.users().messages().insert(userId='me', body=message).execute() # me will use <some_user> from above
return message
def CreateMessage(sender, to, subject, message_text):
message = MIMEText(message_text)
message['To'] = to
message['From'] = sender
message['Subject'] = subject
return {'raw': base64.urlsafe_b64encode(message.as_string().encode()).decode(), 'labelIds': ['INBOX', 'UNREAD']}
def main():
try:
service = validationService()
email = CreateMessage('some#external.com', "<some_user>", "Test", "This is a test")
email_sent = SendMessage(service, email)
print('Message Id:', email_sent['id'])
except errors.HttpError as err:
print('\n---------------You have the following error-------------')
print(err)
print('---------------You have the following error-------------\n')
if __name__ == '__main__':
main()
I'm trying to update the user password for a user in Microsoft Active Directory with LDAP, using JNDI library over SASL (DIGEST-MD5). And there are a couple of issues that makes the operation fail.
First issue
During the authentication phase, I receive the error
Exception: #javax.naming.AuthenticationException: [LDAP: error code 49 - 80090303: LdapErr: DSID-0C09054F, comment: The digest-uri does not match any LDAP SPN's registered for this server., data 0, v2580
Steps taken
I added the required SPN to the DC in AD, and the issue was resolved only for one time, then it came back. When I checked the DC, I found out that the SPN that I have just added has been removed. And this keeps happening every time I add the SPN to the DC!
Second issue
During the time that the authentication proceeds successfully, the server refuses to update the user's password. I'm trying to update the "unicodePwd" attribute using a "DirContext.REPLACE_ATTRIBUTE" operation (I'm using a domain controller administrator account for the authentication, and trying to update a normal user account).
This is the error I receive "Error:
#javax.naming.OperationNotSupportedException: [LDAP: error code 53 - 0000001F: SvcErr: DSID-031A12D2, problem 5003 (WILL_NOT_PERFORM), data 0 ]; remaining name '<the DN of the user that I was trying to update>'
Another note, when I check the attribute "unicodePwd", it's always unset!! So, the question here "How does the AD authenticate the user? Which attribute holds the user's password?!!
Third issue
I can use a couple of LDAP clients, and I can update/reset the user's password. I only need to specify the authentication protocol as (SASL) and the operation goes seamlessly =, without having to make any Changes to the AD/SC!
This is the code sample I'm using
// Session variables
String adminUsername = "<administrator sAMAccountName value>";
String adminPwd = "<admin password>";
String userDN = "<DN for the user being updated>";
String newPwd = "<The new password for the user being updated>";
String ipAddress = "<AD ip address>";
// LDAP configuration
String securityProtocol = "sasl";
String providerURL = "ldap://" + ipAddress;
Hashtable<Object, Object> env = new Hashtable<>();
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
env.put("javax.security.sasl.strength", "high");
env.put("javax.security.sasl.policy.noplaintext", "true");
env.put(Context.PROVIDER_URL, providerURL);
env.put(Context.SECURITY_AUTHENTICATION, "DIGEST-MD5");
env.put(Context.SECURITY_PRINCIPAL, adminUsername);
env.put(Context.SECURITY_CREDENTIALS, adminPwd);
env.put(Context.SECURITY_PROTOCOL, securityProtocol);
env.put(Context.REFERRAL, "follow");
// Prepare the modifications list
String newQuotedPassword = "\"" + newPwd + "\"";
byte[] newUnicodePassword = newQuotedPassword.getBytes("UTF-16LE");
ModificationItem[] mods = new ModificationItem[1];
mods[0] = new ModificationItem(DirContext.REPLACE_ATTRIBUTE,
new BasicAttribute("unicodePwd", newUnicodePassword));
// Initiate the LDAP connection
LdapContext ctx = new InitialLdapContext(env, null);
// Modify the password
ctx.modifyAttributes(userDN, mods);
// Close LDAP connection
ctx.close();
Your help is much appreciated.
So, after a good deal of R&D. There is no way to update the password in the AD without using SSL. Java and MS AD are very strict on that.
I'm really confused how Google App Engine's User's get_current_user() works. I've looked around the internet at a bunch of different guides and tutorials about login and authentication, and many of them mention similar methods.
If there are a million users logged in to my application at the same time, how can that method possibly work? Does each user get their own instance of the server? How does the server know which client it is talking to?
It doesn't make sense to me at all.
When logging in (by clicking on the URL generated by create_login_url()) a cookie containing user identifying information is prepared and pushed on the client side, then used in subsequent requests until the user logs out or the cookie expires. Calling get_current_user() simply checks the cookie existance/information and responds accordingly.
On the development server the cookie is named dev_appserver_login. I can no longer check the cookie name on GAE as I switched away from the Users API.
The actual handling of the cookie seems to happen somewhere on the Users service backend, for example, by looking at the google/appengine/api/users.py file in the python SDK:
def create_login_url(dest_url=None, _auth_domain=None,
federated_identity=None):
...
req = user_service_pb.CreateLoginURLRequest()
resp = user_service_pb.CreateLoginURLResponse()
try:
apiproxy_stub_map.MakeSyncCall('user', 'CreateLoginURL', req, resp)
...
The end point (at least for the development server) seems to somehow land somewhere in google/appengine/tools/appengine_rpc.py, for example:
#staticmethod
def _CreateDevAppServerCookieData(email, admin):
"""Creates cookie payload data.
Args:
email: The user's email address.
admin: True if the user is an admin; False otherwise.
Returns:
String containing the cookie payload.
"""
if email:
user_id_digest = hashlib.md5(email.lower()).digest()
user_id = "1" + "".join(["%02d" % ord(x) for x in user_id_digest])[:20]
else:
user_id = ""
return "%s:%s:%s" % (email, bool(admin), user_id)
def _DevAppServerAuthenticate(self):
"""Authenticates the user on the dev_appserver."""
credentials = self.auth_function()
value = self._CreateDevAppServerCookieData(credentials[0], True)
self.extra_headers["Cookie"] = ('dev_appserver_login="%s"; Path=/;' % value)
I have an application which is school based. Each tenant is a different school and to access the application all users for each school have the same password.
Alongside this each school user has to have a google email if they want access to the application. So the application first checks they are a google user, checks wether they are a school user and finally checks that their google email is in the school user list before they are allowed access to any page.
The school user part is using session data from webapp2 sessions to ensure each request they have appropriate access
class Handler(webapp2.RequestHandler):
def dispatch(self):
# Get a session store for this request.
self.session_store = sessions.get_store(request=self.request)
try:
# Dispatch the request.
webapp2.RequestHandler.dispatch(self)
finally:
# Save all sessions.
self.session_store.save_sessions(self.response)
#webapp2.cached_property
def session(self):
# Returns a session using the default cookie key.
return self.session_store.get_session()
When a user logins I check the password then create a session which checks their password / user combination every request.
def check_u(self):
try:
uid = self.session.get('user')
parent = self.session.get('school-id')
udl = m.models.User.by_id(int(uid),parent)
if uid and udl:
return udl
else:
return False
except (TypeError,AttributeError):
return False
A parent datastore entity for each different school is used called MetaSchool which I have been currently using to ensure that there is no data leak across schools. Each datastore entry uses this parent session key as a way of setting the datastore entry with MetaSchool as parent then using this session key again to read back this data.
This method works but is onerous. I would like to use namespace as a way of separating the data but would like to use the Metaschool id as the name.
def namespace_manager_default_namespace_for_request():
### Here I need to get ------ parent = self.session.get('school-id')
### use this session to gain the MetaSchool key id
### Set this as the namespace name
Basically trying to emulate from the docs the below scenario
from google.appengine.api import users
def namespace_manager_default_namespace_for_request():
# assumes the user is logged in.
return users.get_current_user().user_id()
I am having difficulty getting the session data from Handler object???
Any thoughts
This is what I came up with.
from google.appengine.api import namespace_manager
from webapp2_extras import sessions
def namespace_manager_default_namespace_for_request():
session = sessions.get_store()
s = session.get_session()
name = s.get('key')
if name:
return name
else:
return namespace_manager.set_namespace('string')
My GAE application is written in Python with webapp2. My application targets at analyzing user's online social network. Users could login and authorize my application, hence the access token will be stored for further crawling the data. Then i use the taskqueue to launch a backend task, as the crawling process is time consuming. However, when i access the datastore to fetch the access token, i can get it. I wonders whether there is a way to access the data of the frontend, rather than the temporary storage for the taskqueue.
the handler to the process http request from the user
class Callback(webapp2.RequestHandler):
def get(self):
global client
global r
code = self.request.get('code')
try:
client = APIClient(app_key=APP_KEY, app_secret=APP_SECRET,redirect_uri=CALLBACK_URL)
r = client.request_access_token(code)
access_token = r.access_token
record = model.getAccessTokenByUid(r.uid)
if record is None or r.access_token != record.accessToken:
# logging.debug("access token stored")
**model.insertAccessToken(long(r.uid), access_token, r.expires_in, "uncrawled", datetime.datetime.now())** #data stored here
session = self.request.environ['beaker.session']
session['uid'] = long(r.uid)
self.redirect(CLUSTER_PAGE % ("true"))
except Exception, e:
logging.error("callback:%s" % (str(e)));
self.redirect(CLUSTER_PAGE % ("false"))
the handle to process task submitted to taskqueue
class CrawlWorker(webapp2.RequestHandler):
def post(self): # should run at most 1/s
uid = self.request.get('uid')
logging.debug("start crawling uid:%s in the backend" % (str(uid)))
global client
global client1
global r
tokenTuple = model.getAccessTokenByUid(uid)
if tokenTuple is None: **#here i always get a None**
logging.error("CounterWorker:oops, authorization token is missed.")
return
The question is not clear (is it can or cant?) But if you want to access frontend data from the taskqueue, pass it as parameters to the task queue.