Using lambda function trying to publsh a message to AWS iot core. Where lambda is connected to VPC. Using VPC endpoint I need to publish message to IOT core.
I configured VPC endpoint and created private hosted zone for it
When I tried to publish message. Getting ERROR TIMED OUT
import json
import boto3
client = boto3.client('iot-data', region_name='xxxx')
def lambda_handler(event, context):
response = client.publish(
topic='esp32/sub',
qos=1,
payload=json.dumps({"foo":"bar"})
)
print(response)
return {
'statusCode': 200,
'body': json.dumps('Published to topic')
}
If your Lambda function needs to reach endpoints on the internet then ensure a default route from the Lambda function's private subnets to a NAT instance or NAT Gateway in a public subnet. And configure an IGW, if needed, without which internet access is not possible.
Related
I am using GCP Cloud SQL instances for getting data. This SQL instance when accessed with its public IP address, connection is happening and data is visible. But due to security constraint I will have to access it only via private IP address.
I made code changes as said in Google documentations for connection via private IP address:
(https://cloud.google.com/sql/docs/postgres/connect-app-engine-standard#private-ip_1)
(https://cloud.google.com/vpc/docs/configure-serverless-vpc-access#java-8)
In ConnectionPoolContextListener.java file: iptypes=Private added
config.addDataSourceProperty("ipTypes", "PRIVATE");
In appengine-web.xml file: serverless vpc connector element added
projects/PROJECTNAME/locations/europe-west1/connectors/CONNECTORNAME
all-traffic
In gitlab-ci.yml file: line to deploy the connector service added
deploy_env-name:
script:-gcloud app deploy src/main/webapp/WEB-INF/appengine-web.xml
These changes are not working and the API calls made are failing giving CORS errors(cross origin resource sharing error preflight missing allow origin header)(refer to screenshot)
UI CORS Error
App engines logs are as follows:
com.zaxxer.hikari.pool.HikariPool throwPoolInitializationException: HikariPool-1 - Exception during pool initialization. (HikariPool.java:587)
org.postgresql.util.PSQLException: The connection attempt failed.
...
Caused by: java.io.IOException: Connection refused
Everything is working when public IP address is used no CORS error also. But with private IP address connection is failing, not sure what is wrong here.
DB Connection code:
private DataSource createConnectionPool() {
HikariConfig config = new HikariConfig();
config.setJdbcUrl(String.format("jdbc:postgresql://google/%s",
DB_NAME));
config.setUsername(DB_USER);
config.setPassword(DB_PASS);
config.addDataSourceProperty("socketFactory",
"com.google.cloud.sql.postgres.SocketFactory");
config.addDataSourceProperty("cloudSqlInstance",
CLOUD_SQL_CONNECTION_NAME);
config.addDataSourceProperty("ipTypes", "PRIVATE");
config.setMaximumPoolSize(10);
config.setMinimumIdle(5);
config.setConnectionTimeout(10000); // 10 seconds
config.setIdleTimeout(600000); // 10 minutes
config.setMaxLifetime(1800000); // 30 minutes
DataSource pool = new HikariDataSource(config);
return pool;
}
There was something wrong in VPC connector that I had created before. I created a new VPC connector with same network and region as those of the cloudsql instance and assigned an IP address range and now this DB connection is happening and data is getting loaded and CORS error has gone.
So to connect to a SQL instance via Private IP address from App engine I had to make only the following changes:
Create a Serverless VPC connector.
add vpc-connector element in appengine.yml file
include property "iptypes" in ConnectionPoolContextListener.java file
Here's the problem. The connection string has an error in it:
// Not correct
config.setJdbcUrl(String.format("jdbc:postgresql://google/%s", DB_NAME));
// Should be
config.setJdbcUrl(String.format("jdbc:postgresql:///%s", DB_NAME));
See the documentation for details: https://github.com/GoogleCloudPlatform/cloud-sql-jdbc-socket-factory/blob/main/docs/jdbc-postgres.md#creating-the-jdbc-url.
I've been trying to deploy my Blazor PWA for 2 days without any success so far, if someone has an idea of what I’m doing wrong I would be really grateful.
hello
I could resolve most of my issues by myself but I'm now stuck on a CORS problem using AAD.
Here's my project setup:
Blazor webassembly client hosted on Static Website Storage (works
great), Net 5
AzureFunctions connected to an Azure Sql Server database (works great
with anonymous authentication in Blazor)
Azure Active Directory I want to use to authenticate the users.
(protecting both the blazor app and the functions)
So I’ve created an App registration, added my static web site address as SPA uri and uncheck both implicit.
In my blazor client, program.cs, I’ve added the following code to connect to AAD:
builder.Services.AddMsalAuthentication(options =>
{
builder.Configuration.Bind("AzureAd", options.ProviderOptions.Authentication); //contains clientId, Authority
options.ProviderOptions.DefaultAccessTokenScopes.Add("https://graph.microsoft.com/User.Read");
options.ProviderOptions.LoginMode = "redirect";
});
That works great too, I can login, authorize view works as expected.
The problem is when I try to authenticate Azure functions with «Login with Azure Active Directory»,
I' tried with both express and advanced configurations (using clientId, tenantId) but when I
Access to fetch at 'https://login.windows.net/tenantid/oauth2/authorize ... (redirected from 'https://myfunctionstorage.azurewebsites.net/api/client/list') from origin 'https://*****9.web.core.windows.net' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.
I have of course enabled CORS for my Blazor Client Address on Azure Functions configuration but the problem seems to be about the login windows uri...
Also, if I enable the token id implicit flow in the App registration and access the login url in the browser it works perfectly fine.
All the examples I could find so far are about msal 1.0 and App registration using implicit flow instead of SPA so it doesn't help...
Thank you for your time and your help.
UPDATE:
I’ve done more researches since yesterday and I think it could by related to my HTTPClient, currently I use the basic one (with just a base adress).
But I’ve seen on some example that when using the Client using AAD it needs more parameters, for example:
builder.Services.AddHttpClient("companiesAPI", cl => { cl.BaseAddress = new Uri("https://localhost:5001/api/"); }) .AddHttpMessageHandler(sp => { var handler = sp.GetService<AuthorizationMessageHandler>() .ConfigureHandler( authorizedUrls: new[] { "https://localhost:5001" }, scopes: new[] { "companyApi" } ); return handler; });
Is that AuthorizationMessageHandler needed ?
Also I see some references to the need of using the «use_impersonation» scope.
Are those changes (on HttpClient and the use_impersonation scope) also required when using msal2/SPA app registration ?
Thank you
If want to call the Azure function projected by Azure AD in Blazor webassembly client, please refer to the following steps
If you want to call Azure function projected by Azure AD in angular application, please refer to the following code
Create Azure AD application to protect function
Register Azure AD application. After doing that, please copy Application (client) ID and the Directory (tenant) ID
Configure Redirect URI. Select Web and type <app-url>/.auth/login/aad/callback.
Enable Implicit grant flow
Define API scope and copy it
Create client secret.
Enable Azure Active Directory in your App Service app
Configure CORS policy in Azure Function
Create Client AD application to access function
Register application
Enable Implicit grant flow
configure API permissions. Let your client application have permissions to access function
Code
Create Custom AuthorizationMessageHandler class
using Microsoft.AspNetCore.Components;
using Microsoft.AspNetCore.Components.WebAssembly.Authentication;
public class CustomAuthorizationMessageHandler : AuthorizationMessageHandler
{
public CustomAuthorizationMessageHandler(IAccessTokenProvider provider,
NavigationManager navigationManager)
: base(provider, navigationManager)
{
ConfigureHandler(
authorizedUrls: new[] { "<your function app url>" },
scopes: new[] { "<the function app API scope your define>" });
}
}
Add the following code in Program.cs.
public static async Task Main(string[] args)
{
var builder = WebAssemblyHostBuilder.CreateDefault(args);
builder.RootComponents.Add<App>("app");
// register CustomAuthorizationMessageHandler
builder.Services.AddScoped<CustomAuthorizationMessageHandler>();
// configure httpclient
// call the following code please add packageMicrosoft.Extensions.Http 3.1.0
builder.Services.AddHttpClient("ServerAPI", client =>
client.BaseAddress = new Uri("<your function app url>"))
.AddHttpMessageHandler<CustomAuthorizationMessageHandler>();
// register the httpclient
builder.Services.AddScoped(sp => sp.GetRequiredService<IHttpClientFactory>()
.CreateClient("ServerAPI"));
// configure Azure AD auth
builder.Services.AddMsalAuthentication(options =>
{
builder.Configuration.Bind("AzureAd", options.ProviderOptions.Authentication);
options.ProviderOptions.DefaultAccessTokenScopes.Add("<the function app API scope your define>");
});
await builder.Build().RunAsync();
}
Call the API
#page "/fetchdata"
#using Microsoft.AspNetCore.Components.WebAssembly.Authentication
#inject HttpClient Http
<h1>Call Azure Function</h1>
<p>This component demonstrates fetching data from the server.</p>
<p>Result: #forecasts</p>
<button class="btn btn-primary" #onclick="CallFun">Call Function</button>
#code {
private string forecasts;
protected async Task CallFun()
{
forecasts = await Http.GetStringAsync("api/http");
}
}
I have an App Engine project.
I also have a Google Cloud Function.
And I want to call that Google Cloud Function from the App Engine project. I just can't seem to get that to work.
Yes, if I make the function full public (i.e. set the Cloud Function to 'allow all traffic' and create a rule for 'allUsers' to allow calling the function) it works. But if I limit either of the two settings, it stops working immediately and I get 403's.
The App and Function are in the same project, so I would at least assume that setting the Function to 'allow internal traffic only' should work just fine, provided that I have a rule for 'allUsers' to allow calling the function.
How does that work? How does one generally call a (non-public) Google Cloud Function from Google App Engine?
You need an auth header for the ping to the function url. It should look like:
headers = {
....
'Authorization': 'Bearer some-long-hash-token'
}
Here is how to get the token:
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://[your zone]-[your app name].cloudfunctions.net/[your function name]',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
'Allow internal traffic only' does not work as expected. My App Engine app is in the same project as the Functions, and it does not work. I had to turn on 'Allow all traffic', and use the header method.
Example:
def get_access_token():
import requests
token_response = requests.get(
'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/identity?audience=' +
'https://us-central1-my_app.cloudfunctions.net/my_function',
headers={'Metadata-Flavor': 'Google'})
return token_response.content.decode("utf-8")
def test():
url_string = f"https://us-central1-my_app.cloudfunctions.net/my_function?message=it%20worked"
access_token = get_access_token()
print(
requests.get(url_string, headers={'Authorization': f"Bearer {access_token}"}
)
As mentioned in the docs, Allow internal traffic only mentions the following:
Only requests from VPC networks in the same project or VPC Service Controls perimeter are allowed. All other requests are rejected.
Please note that since App Engine Standard is a serverless product, it is not part of the VPC and then the requests made from this product are not considered "Internal" calls, actually the calls are made from the Public IPs of the instances and for this reason you get an HTTP 403 error message.
Also using a VPC Serverless Connector won't work since this more a bridge to reach resources in the VPC (like VMs or Memorystore instances) but not a Cloud Function because this is also a Serverless product and it does not have an IP in the VPC.
I think here are three options:
Using App Engine Flex:
Since App Engine Flex uses VM instances, these instances will be part of the VPC and you'll reach the Function even when setting the "Allow internal traffic only" option.
Use a VM as a proxy:
You can create a VPC Serverless Connector and assign it to the app in App Engine. Then you can create a VM and reach the function using the VM as a proxy. This is not the best option because of the costs but at the end is an option.
The last option considers that the function can use the Allow All Traffic option:
You can set some security on the Cloud Function to only allow a particular Service Account and you can use this sample code to authenticate.
EDITED:
A good sample of the code for this option was shared by #gaefan in the other answer.
#GAEfan is correct.
As an addition: I used the official Google Auth library to give me the necessary headers.
const {GoogleAuth} = require('google-auth-library');
// Instead of specifying the type of client you'd like to use (JWT, OAuth2, etc)
// this library will automatically choose the right client based on the environment.
const googleCloudFunctionURL = 'https://europe-west1-project.cloudfunctions.net/function';
(async function() {
const auth = new GoogleAuth();
let googleCloudFunctionClient = await auth.getIdTokenClient(googleCloudFunctionURL);
console.log(await googleCloudFunctionClient.getRequestHeaders(googleCloudFunctionURL));
})();
We're having trouble publishing messages to a Google Cloud PubSub topic on Google AppEngine. Using the Application Default credentials works perfect locally. But once it's deployed on Google AppEngine it gives the following error:
<HttpError 403 when requesting https://pubsub.googleapis.com/v1/projects/our-project-id/topics/our-topic:publish?alt=json returned "The request cannot be identified with a project. Please pass a valid API key with the request.">
I would assume that it's will use the service account of app engine to access the PubSub API. Here is the code we used to create the credentials.
credentials = GoogleCredentials.get_application_default()
if credentials.create_scoped_required():
credentials = credentials.create_scoped(['https://www.googleapis.com/auth/pubsub'])
http = httplib2.Http()
credentials.authorize(http)
pubsub_service = build('pubsub', 'v1', http=http)
The error is thrown when publishing the actual message to PubSub.
pubsub_service.projects().topics().publish(topic="projects/out-project-id/topics/out-topic", body = { 'messages' : [ { 'data': base64.b64encode(request.get_data()) }]}).execute()
Not that the same flow works doing API call's to "BigQuery", so it's not a general Google API problem. It seems to be specific to PubSub...
It's a rare case of the service account without project id embedded in it. We fixed your service account and you should be good to go now. Sorry for the trouble.
import endpoints
from protorpc import messages #for encapsulating data for sending
from protorpc import message_types
WEB_CLIENT_ID = '644515464936-552tgvthbmnhopg6qe71e99rrfbjatj7.apps.googleusercontent.com'
class HelloMessage(messages.Message):
msg = messages.StringField(1)
#endpoints.api(name='myapiversion1',
version='v1',
allowed_client_ids=[WEB_CLIENT_ID],
audiences=[WEB_CLIENT_ID],
scopes=[endpoints.EMAIL_SCOPE],
auth_level=AUTH_LEVEL_REQUIRED,
hostname='maulikversion1.appspot.com')
api url : https://maulikversion1.appspot.com/_ah/api/maulikversion1/v1/say_hello
I tried accesing it from www.hurl.it a web tool for testing rest api, and it was able to access the api.
Q1
what error it should give instead of allowing to access my custom api?
unfortunately it is allowing any web client to access api.
Q2.
AUTH_LEVEL_REQUIRED is not defined - internal server error
It is now restricting other web clients to access the api
I guess the client id generation took time to reflect at production-environment