Connect to Cloud SQL using Sql2o in Google Cloud Endpoint - google-app-engine

I'm trying to connect to cloud sql inside google cloud endpoint and using lightweight jdbc wrapper sql2o as data access method.
#Api(name = "questionapi", version = "v1", description = "question api")
public class QuestionService {
private static Sql2o sql2o = new Sql2o(
"jdbc:mysql://xxx.xxx.xxx.xxx:3306/xxxxx", "root",
"xxxxxxx");
#ApiMethod(name = "get", httpMethod = HttpMethod.GET)
public List<Question> get() {
String q = "select * from questions";
try (Connection conn = sql2o.open()) {
return conn.createQuery(q).executeAndFetch(Question.class);
}
}
After the app is running, I can visit localhost:8888/_ah/api/explorer to try the api. However, there is an error says:
org.sql2o.Sql2oException: Could not acquire a connection from DataSource - No suitable driver found for jdbc:mysql://xxx.xxx.xxx.xxx:3306/xxxxx
How can I solve this issue?
EDIT:
After change to maven project and I got this new error message:
503 Service Unavailable
- Show headers -
{
"error": {
"message": "java.lang.NoClassDefFoundError: Could not initialize class com.mysql.jdbc.ConnectionImpl",
"code": 503,
"errors": [
{
"domain": "global",
"reason": "backendError",
"message": "java.lang.NoClassDefFoundError: Could not initialize class com.mysql.jdbc.ConnectionImpl"
}
]
}
}
EDIT
It's a new day, I still stuck here.
What I did is I use maven to download the endpoints-skeleton-archetype project, it's a new, empty Cloud Endpoints backend API project ready for use, with required files and directories.
I immediately deploy it to app engine, and try to return a meaningful value. It worked, a simple 'hellp world' string will be returned.
Next, I tried to connect to cloud sql using jdbc. In order to do that, I followed the tutorial here
to add <use-google-connector-j>true</use-google-connector-j> into appengine-web.xml
and I try different combination of connection string
Class.forName("com.mysql.jdbc.GoogleDriver");
String url = "jdbc:google:mysql://xxxxxxxxxxxx:xxxxx?user=root";
conn = DriverManager.getConnection(url);
ResultSet rs = conn.createStatement().executeQuery("SELECT 1 + 1");
After all these, I still get this error message.
503 Service Unavailable
- Show headers -
{
"error": {
"errors": [
{
"domain": "global",
"reason": "backendError",
"message": "java.lang.NullPointerException"
}
],
"code": 503,
"message": "java.lang.NullPointerException"
}
}

This error occurs when the jdbc driver is not found in the classpath. How are you managing dependencies? Do you use maven? The error should be fixed if you add the mysql jdbc driver to your list of dependencies.
I have another comment to your code, which has nothing to do with your question. But here it comes anyway.
The codeline below has a connection leak as you never closes the connection. This will eventually deplete the connection pool and your application will hang.
return sql2o.open().createQuery(q).executeAndFetch(Question.class);
This is the correct way of doing it when using sql2o:
try (Connection con = sql2o.open()) {
return con.createQuery(q).executeAndFetch(Question.class);
}

Related

Configuring Auth0 authentication on Google App Engine Standard and the Cloud Endpoints Frameworks with Python

I am using Cloud Endpoints Frameworks with Python in a Google Cloud App Engine Standard environment to provide an API.
As far as I can tell, I should be able to use python decorators from the Endpoints Frameworks in combination with the endpointscfg.py command-line tool to automatically set up token-based authentication with Auth0; the endpointscfg.py command-line automatically creates the openapi.json file that is used to configure the Google Endpoints proxy.
Here's an example of my decorator for an API that echos stuff back:
# # [START echo_api]
#endpoints.api(
name='echo',
version=_VERSION,
api_key_required=True,
audiences={'auth0': ['https://echo.<my-project>.appspot.com/_ah/api/echo/v1/echo']},
issuers={'auth0': endpoints.Issuer(
'https://<my-project>.auth0.com',
'https://<my-project>.auth0.com/.well-known/jwks.json')}
)
class EchoApi(remote.Service):
...
When I run the endpointscfg.py command-line tool, I get something in my openapi.json file that looks about right:
"paths": {
"/echo/v1/echo": {
"post": {
"operationId": "EchoApi_echo",
"parameters": [
{
"in": "body",
"name": "body",
"schema": {
"$ref": "#/definitions/MainEchoRequest"
}
}
],
"responses": {
"200": {
"description": "A successful response",
"schema": {
"$ref": "#/definitions/MainEchoResponse"
}
}
},
"security": [
{
"api_key": [],
"auth0_jwt": []
}
]
}
}
"securityDefinitions": {
"api_key": {
"in": "query",
"name": "key",
"type": "apiKey"
},
"auth0_jwt": {
"authorizationUrl": "https://<my-project>.auth0.com/authorize",
"flow": "implicit",
"type": "oauth2",
"x-google-issuer": "https://<my-project>.auth0.com",
"x-google-jwks_uri": "https://<my-project>.auth0.com/.well-known/jwks.json",
"x-google-audiences": "https://echo.<my-project>.appspot.com/_ah/api/echo/v1/echo"
}
}
So, the problem is that this set-up appears to do nothing and does not check incoming tokens to prevent access if no token is present or if the token is invalid.
I have been able to set-up manual processing of the bearer token within the API echo function using the python-jose library (sorry if it's not well done, but I'm just testing and comments are welcome):
authorization_header = self.request_state.headers.get('authorization')
if authorization_header is not None:
if authorization_header.startswith('Bearer '):
access_token = authorization_header[7:]
logging.info(access_token)
else:
logging.error("Authorization header did not start with 'Bearer '!")
raise endpoints.UnauthorizedException(
"Authentication failed (improperly formatted authorization header).")
else:
logging.error("Authorization header did not start with 'Bearer '!")
raise endpoints.UnauthorizedException("Authentication failed (bearer token not found).")
r = urlfetch.fetch(_JWKS_URL)
jwks_content = json.loads(r.content)
keys = jwks_content['keys']
public_key = jwk.construct(keys[0])
logging.info(public_key)
message, encoded_signature = str(access_token).rsplit('.', 1)
# decode the signature
decoded_signature = base64url_decode(encoded_signature.encode('utf-8'))
# verify the signature
if not public_key.verify(message.encode("utf8"), decoded_signature):
logging.warning('Signature verification failed')
raise endpoints.UnauthorizedException("Authentication failed (invalid signature).")
else:
logging.info('Signature successfully verified')
claims = jwt.get_unverified_claims(access_token)
# additionally we can verify the token expiration
if time.time() > claims['exp']:
logging.warning('Token is expired')
raise endpoints.UnauthorizedException("Authentication failed (token expired).")
# and the Audience (use claims['client_id'] if verifying an access token)
if claims['aud'] != _APP_CLIENT_ID:
logging.warning('Token was not issued for this audience')
raise endpoints.UnauthorizedException("Authentication failed (incorrect audience).")
# now we can use the claims
logging.info(claims)
This code works, but I assumed that the whole point of setting up the decorator and configuring the openapi.json file was to off-load these checks to the proxy so that only valid tokens hit my code.
What am I doing wrong?
UPDATE:
It may be that I need to check endpoints.get_current_user() in my code to control access. However, I have just noticed the following in my logs:
Cannot decode and verify the auth token. The backend will not be able to retrieve user info (/base/data/home/apps/e~<my-project>/echo:alpha23.414400469228485401/lib/endpoints_management/control/wsgi.py:643)
Traceback (most recent call last):
File "/base/data/home/apps/e~<my-project>/echo:alpha23.414400469228485401/lib/endpoints_management/control/wsgi.py", line 640, in __call__
service_name)
File "/base/data/home/apps/e~<my-project>/echo:alpha23.414400469228485401/lib/endpoints_management/auth/tokens.py", line 75, in authenticate
error)
UnauthenticatedException: (u'Cannot decode the auth token', UnauthenticatedException(u'Cannot find the `jwks_uri` for issuer https://<my-project>.auth0.com/: either the issuer is unknown or the OpenID discovery failed',))
However, I think everything is configured ok. Any idea why 'jwks_uri' cannot be found despite the fact that path in the openapi.json file is correct?
I'm the current maintainer of these Frameworks. You do need to check endpoints.get_current_user() to control access, yes. I'm working on a feature to make this much simpler.
As for that UnauthenticatedException, you can ignore it. That's coming from the 'management framework', which attempts to check auth tokens even though it's not involved in the Frameworks' oauth security (only the api key security).

Create datasource in azure search returns 403 Forbidden

I have a search service running on azure in a free tier. On this service I already have a datasource, and indexer and an index defined.
I'd like to add another datasource (and index + indexer). When I do this (using postman) I get 403 Forbidden without any other error message.
This is the POST I made to this url - https://my-search-service-name.search.windows.net/datasources?api-version=2016-09-01:
"Content-Type": "application/json",
"api-key": "API-KEY-HERE"
{
"name": "datasource-prod",
"description": "Data source for search",
"type": "azuresql",
"credentials": { "connectionString" : "Server=tcp:xxxx.database.windows.net,1433;Initial Catalog=xxxxx_prod;Persist Security Info=False;User ID=xxxxxx;Password=xxxxxx!;Trusted_Connection=False;Encrypt=True;Connection Timeout=30;" },
"container": {"name": "DataAggregatedView"},
"dataChangeDetectionPolicy": {
"#odata.type" : "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
"highWaterMarkColumnName" : "ChangeIndicator"
},
"dataDeletionDetectionPolicy": {
"#odata.type" : "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
"softDeleteColumnName" : "isDeleted",
"softDeleteMarkerValue" : "0"
}
}
Using the same request, with different name and database name worked perfectly and generated the existing (first) datasource. This error (403) - not even got the error message - happens only when I try to define a second datasource.
As I can understand from documentation, free search tier allows 3 datasources. Anyone had this issue? Any help/direction is appreciate!
Thank you.
Make sure you're using the admin API key. It looks like you may be using a query key.

How to configure IdentityServerAuthenticationOptions.Authority to use wildcards

I successfully setup IdentityServer4 with ASP.NET Core.
As a default config I had this:
IdentityServerAuthenticationOptions options = new IdentityServerAuthenticationOptions()
{
Authority = "http://localhost:5000",
ScopeName = "scope",
ScopeSecret = "ScopeSecret",
AutomaticAuthenticate = true,
AutomaticChallenge = true,
RequireHttpsMetadata = false,
};
Now, using this guide I configured to be read from configuration files and so they can be any numbers in production.
For example if I setup API to be running at http://*:5000 then the client can connect to it via the service IP address like http://192.168.1.100:5000.
Once the client obtains the Bearer token and tries to use it, an Internal Server Error occures with this exception:
Unable to obtain configuration from:
'http://*:5000/.well-known/openid-configuration'.
---> System.IO.IOException: IDX10804: Unable to retrieve document from: 'http://*:5000/.well-known/openid-configuration'.
---> System.UriFormatException: Invalid URI: The hostname could not be parsed.
What is the correct way to configure IdS4 to have dynamic authority?
Update
It seems the problem is with Issuer, any idea on this?
Microsoft.IdentityModel.Tokens.SecurityTokenInvalidIssuerException:
IDX10205: Issuer validation failed. Issuer: 'http://192.168.1.100:5000'. Did not match: validationParameters.ValidIssuer: 'http://localhost:5000' or validationParameters.ValidIssuers: 'null'.
at Microsoft.IdentityModel.Tokens.Validators.ValidateIssuer(String issuer, SecurityToken securityToken, TokenValidationParameters validationParameters)
By a big surprise, all I needed, was to set a value (almost any value) for IssuerUri:
public IServiceProvider ConfigureServices(IServiceCollection services)
{
////...
var identiyBuilder = services.AddIdentityServer(options =>
{
options.RequireSsl = false;
options.IssuerUri = "MyCompany";
});
////...
}
Now, by the above config, I can use the service by any IP address.
I didn't find I could just put in MyCompany
But in my log files I had the following:
Bearer was not authenticated. Failure message: IDX10205: Issuer validation failed. Issuer: 'https://crm.example.com'. Did not match: validationParameters.ValidIssuer: 'MyCompany' or validationParameters.ValidIssuers: 'null'.
I don't quite know what 'issuer' means but I was able to just take 'https://crm.example.com' and get things working with this :
options.IssuerUri = "https://crm.example.com";

Interface Between Google Sign-in and MailKit

I am writing an app in WPF (Windows 10 desktop) that should
include a component where the user can download message headers
and messages from G-Mail.
I am trying to use MailKit to interface with G-Mail via a secure
connection (without having to turn on "allow less-secure apps"
for G-Mail) and download messages with POP3. I am a bit confused
as to the proper procedure.
FYI: I know next to nothing about OAuth and TLS, so KISS please.
I have created and downloaded a JSON file for OAuth 2.0 from Google.
I have visited the FAQ for MailKit, and the following section
seems relevant, but I'm not sure as to what I should plug in
to the interface.
(Please see the code below.)
For "password", would that be the password for the account?
I'm not sure as to what to give for
"your-developer-id#developer.gserviceaccount.com".
.........................................................
https://github.com/jstedfast/MailKit/blob/master/FAQ.md#ProtocolLog
.........................................................
From the Q & A:
How can I log in to a GMail account using OAuth 2.0?
The first thing you need to do is follow Google's instructions for
obtaining OAuth 2.0 credentials for your application.
Once you've done that, the easiest way to obtain an access token is to
use Google's Google.Apis.Auth library:
var certificate = new X509Certificate2 (#"C:\path\to\certificate.p12", "password",
X509KeyStorageFlags.Exportable);
var credential = new ServiceAccountCredential (new ServiceAccountCredential
.Initializer ("your-developer-id#developer.gserviceaccount.com") {
// Note: other scopes can be found here: [links]
Scopes = new[] { "https mail google com " },
User = "username#gmail.com"
}.FromCertificate (certificate));
bool result = await credential.RequestAccessTokenAsync (CancellationToken.None);
// Note: result will be true if the access token was received successfully
// Now that you have an access token (credential.Token.AccessToken), you can
// use it with MailKit as if it were the password:
using (var client = new ImapClient ()) {
client.Connect ("imap.gmail.com", 993, true);
// use the access token as the password string
client.Authenticate ("username#gmail.com", credential.Token.AccessToken);
}
My next question: Would the user be able to access their own account(s)
with my app without having to follow the same procedure?
IOW: Will the credentials that I've downloaded work for any account?
... or allow access only to the account from which the credentials
were created?
If the credentials are only good for my own account, then I'll have to
do something else.
Would Google Sign-In be a better approach?
I've downloaded the example code for .NET from Google:
https://github.com/googlesamples/oauth-apps-for-windows
I've built and ran ran "OAuthConsoleApp", as well as "OAuthDesktopApp".
It would seem that I am getting a secure connection from those,
as I have gotten the following output:
.........................................................
redirect URI: http 127.0.0.1:64003
Listening..
Authorization code: qwerty ...
Exchanging code for tokens...
Send the request ...
GetRequestStream ...
await stream.WriteAsync ...
Get the response ...
responseText ...
{
"access_token": "qwerty ...",
"token_type": "Bearer",
"expires_in": 3600,
"refresh_token": "qwerty ...",
"id_token": "qwerty ..."
}
Making API Call to Userinfo...
+++ userinfoResponseText : {
"sub": "117108120545711995673",
"name": "My Name",
"given_name": "My",
"family_name": "Name",
"picture": "qwerty ...",
"locale": "en"
}
.....................................................
I see that I have an "access_token" in the response and I thought
that I could plug that in to the "client.Authenticate" method for
MailKit as the password (as mentioned in the docs for MailKit):
string access_token = tokenEndpointDecoded["access_token"];
client.Connect ("pop.gmail.com", 995, SecureSocketOptions.SslOnConnect);
client.Authenticate ("username#gmail.com", access_token);
It threw an exception:
.....................................................
"POP3 server did not respond with a +OK response to the AUTH command."
at MailKit.Net.Pop3.Pop3Client.Authenticate(Encoding encoding,
ICredentials credentials, CancellationToken cancellationToken)
at MailKit.MailService.Authenticate(String userName, String
password, CancellationToken cancellationToken)
at
NS_MailKit_01.Pop3.cls_mailKit_Pop3_01.connect_and_authenticate(Object
p3_client, String p_access_token)
in :\Software_Develpoment_Sys_03_K\MIME_EMail\TEST_02\Mail_Kit_01\MailKit_01.cs:line
465
at
LIB1_01_G_Mail_Auth.cls_G_mail_authorization.str_token_NTRF.invoke_access_token(String
p_access_token)
in K:\Software_Develpoment_Sys_03_K\MIME_EMail\TEST_02\OAuth\oauth-apps-for-windows\OAuthConsoleApp\LIB1_01_G_Mail_Auth\G_Mail_Auth_01.cs:
line 95
at
LIB1_01_G_Mail_Auth.cls_G_mail_authorization.d__13.MoveNext()
in K:\Software_Develpoment_Sys_03_K\MIME_EMail\TEST_02\OAuth\oauth-apps-for-windows\OAuthConsoleApp\LIB1_01_G_Mail_Auth\G_Mail_Auth_01.cs:line
343
.....................................................
Does anyone know how I could get a "credential" object from
the Google interface that I could use with MailKit?
Any help would be appreciated.
Thanks!
For "password", would that be the password for the account?
No. It would be the password for your PKCS12 file containing your X.509 Certificate and your private key.
I'm not sure as to what to give for "your-developer-id#developer.gserviceaccount.com".
You need to register yourself and your application with Google's Developer program which will give you a developer id to use. You need to follow their directions.

Leasing app engine task in compute engine

I'm trying to lease an app engine task from a pull queue in a compute engine instance but it keeps giving this error:
{
"error": {
"errors": [
{
"domain": "global",
"reason": "forbidden",
"message": "you are not allowed to make this api call"
}
],
"code": 403,
"message": "you are not allowed to make this api call"
}
}
This is the code I'm using:
import httplib2, json, urllib
from oauth2client.client import AccessTokenCredentials
from apiclient.discovery import build
def FetchToken():
METADATA_SERVER = ('http://metadata/computeMetadata/v1/instance/service-accounts')
SERVICE_ACCOUNT = 'default'
http = httplib2.Http()
token_uri = '%s/%s/token' % (METADATA_SERVER, SERVICE_ACCOUNT)
resp, content = http.request(token_uri, method='GET',
body=None,
headers={'Metadata-Flavor': 'Google'})
print token_uri
print content
if resp.status == 200:
d = json.loads(content)
access_token = d['access_token'] # Save the access token
credentials = AccessTokenCredentials(d['access_token'],
'my-user-agent/1.0')
autho = credentials.authorize(http)
print autho
return autho
else:
print resp.status
task_api = build('taskqueue', 'v1beta2')
lease_req = task_api.tasks().lease(project='project-name',
taskqueue='pull-queue',
leaseSecs=30,
numTasks=1)
result = lease_req.execute(http=FetchToken()) ####ERRORS HERE
item = result.items[0]
print item['payload']
It seems like an authentication issue but it gives me the exact same error if I do the same lease request using a bullshit made-up project name so I can't be sure.
I also launched the instance with taskqueue enabled.
Any help would be greatly appreciated
In case anyone else is stuck on a problem like this I'll explain how it's working now.
Firstly I'm using a different (shorter) method of authentication:
from oauth2client import gce
credentials = gce.AppAssertionCredentials('')
http = httplib2.Http()
http=credentials.authorize(http)
credentials.refresh(http)
service = build('taskqueue', 'v1beta2', http=http)
Secondly, the reason my lease request was being denied is that in queue.yaml my service account email was set as a writer email. In the documentation it's mentioned that an email ending with #gmail.com will not have the rights of a user email when set as a writer email. It's not mentioned that that extends to emails ending with #developer.gserviceaccount.com.

Resources