I have a container with SQL (following this example https://github.com/croblesm/mssql-polybase)
I have a container with MongoDB running as normal.
I need to create a external table to consume mongoDB data and create a report.
I did follow this instructions https://learn.microsoft.com/en-us/sql/relational-databases/polybase/polybase-configure-mongodb?view=sql-server-ver16
but I'm getting an error on SQL
Msg 7320, Level 16, State 110, Line 2
Cannot execute the query "Remote Query" against OLE DB provider "MSOLEDBSQL" for linked server "(null)". External access operation failed because the location URI is invalid. Revise the URI and try again.
Here are the networkSettings of my Mongo Container:
"NetworkSettings": {
"Bridge": "",
"SandboxID": "fd848692fbdb9ba61c5bbbe774d05955bef0cca850e2bb137e54323fede0b5c1",
"HairpinMode": false,
"LinkLocalIPv6Address": "",
"LinkLocalIPv6PrefixLen": 0,
"Ports": {
"27017/tcp": [
{
"HostIp": "0.0.0.0",
"HostPort": "27017"
}
]
},
"SandboxKey": "/var/run/docker/netns/fd848692fbdb",
"SecondaryIPAddresses": null,
"SecondaryIPv6Addresses": null,
"EndpointID": "",
"Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"IPAddress": "",
"IPPrefixLen": 0,
"IPv6Gateway": "",
"MacAddress": "",
"Networks": {
"oms_dev_setup_default": {
"IPAMConfig": null,
"Links": null,
"Aliases": [
"mongodb",
"1eb3634590ea"
],
"NetworkID": "0f740644bac8ee38bf6e237c31e718d524d1cb74932c4426744f1820aa51abb1",
"EndpointID": "349f1b5717f0fd8a401b518d62846c53166e4d729843e2fd162c05baec32b82a",
"Gateway": "172.18.0.1",
"IPAddress": "172.18.0.4",
"IPPrefixLen": 16,
"IPv6Gateway": "",
"GlobalIPv6Address": "",
"GlobalIPv6PrefixLen": 0,
"MacAddress": "02:42:ac:12:00:04",
"DriverOpts": null
}
}
}
I think the problem is related with the configuration of the SQL EXTERNAL DATA SOURCE.
ALTER EXTERNAL DATA SOURCE MongoDBSource
SET LOCATION = 'mongodb://172.18.0.1:27017'', --- Host CREDENTIAL = MongoDBConnection,CONNECTION_OPTIONS = "ssl=false;"';
I did try several combinations but nothing is working
Related
I'm trying to get the aggregate agent to work with timescale db.
https://volttron.readthedocs.io/en/main/developing-volttron/developing-agents/specifications/aggregate.html
I have a file aggregation.config
{
"connection": {
"type": "postgresql",
"params": {
"dbname": "volttrondb",
"host": "127.0.0.1",
"port": 5432,
"user": "user",
"password": "password",
"timescale_dialect": true
}
},
"aggregations":[
# list of aggregation groups each with unique aggregation_period and
# list of points that needs to be collected
{
"aggregation_period": "1h",
"use_calendar_time_periods": true,
"utc_collection_start_time":"2016-03-01T01:15:01.000000",
"points": [
{
"topic_names": ["campus/building/fake/EKG_Cos", "campus/building/fake/EKG_Sin"],
"aggregation_topic_name":"campus/building/fake/avg_of_EKG_Cos_EKG_Sin",
"aggregation_type": "avg",
"min_count": 2
}
]
}
]
}
And run the following command
vctl install services/core/SQLAggregateHistorian/ --tag aggregate-historian -c config/aggregation.config --start
It starts correctly - the vctl status shows it running and there are no errors in the log.
I do not see the point campus/building/fake/avg_of_EKG_Cos_EKG_Sin in the topics table.
Any suggestions?
I'm using Azure Data factory,
I'm using SQLServer as source and Postgres as target. Goal is to copy 30 tables from SQLServer, with transformation, to 30 tables in Postgres. Hard part is I have 80 databases from and to, all with the exact same layout but different data. Its one database per customer so 80 customers each with their own databases.
Linked Services doesn't allow parameters for Postgres.
I have one dataset per source and target using parameters for schema and table names.
I have one pipeline per table with SQLServer source and Postgres target.
I can parameterize the SQLServer source in linked service but not Postgres
Problem is how can I copy 80 source databases to 80 target databases without adding 80 target linked services and 80 target datasets? Plus I'd have to repeat all 30 pipelines per target database.
BTW I'm only familiar with the UI, however anything else that does the job is acceptable.
Any help would be appreciated.
There is simple way to implement this. Essentially you need to have a single Linked Service, which reads the connection string out of KeyVault. You can then parameterize source and target as keyvault secret names, and easily switch between data sources by just changing the secret name. This relies on all connection related information being enclosed within a single connection string.
I will provide a simple overview for Postgresql, but the same logic applies to MSSQL servers as source.
Implement a Linked Service for Azure Key Vault.
Add a Linked Service for Azure Postgresql that uses Key Vault to store access url in format: Server=your_server_name.postgres.database.azure.com;Database=your_database_name;Port=5432;UID=your_user_name;Password=your_password;SSL Mode=Require;Keepalive=600; (advise to use server name as secret name)
Pass this parameter, which is essentially correct secret name, in the Pipeline (you can also implement a loop that would accept immediately array of x elements, and parse n elements at a time into separate pipeline)
Linked Service Definition for KeyVault:
{
"name": "your_keyvault_name",
"properties": {
"description": "KeyVault",
"annotations": [],
"type": "AzureKeyVault",
"typeProperties": {
"baseUrl": "https://your_keyvault_name.vault.azure.net/"
}
}
}
Linked Service Definition for Postgresql:
{ "name": "generic_postgres_service".
"properties": {
"type": "AzurePostgreSql",
"parameters": {
"pg_database": {
"type": "string",
"defaultValue": "your_database_name"
}
},
"annotations": [],
"typeProperties": {
"connectionString": {
"type": "AzureKeyVaultSecret",
"store": {
"referenceName": "KeyVaultName",
"type": "LinkedServiceReference"
},
"secretName": "#linkedService().secret_name_for_server"
}
},
"connectVia": {
"referenceName": "AutoResolveIntegrationRuntime",
"type": "IntegrationRuntimeReference"
}
}
}
Dataset Definition for Postgresql:
{
"name": "your_postgresql_dataset",
"properties": {
"linkedServiceName": {
"referenceName": "generic_postgres_service",
"type": "LinkedServiceReference",
"parameters": {
"secret_name_for_server": {
"value": "#dataset().secret_name_for_server",
"type": "Expression"
}
}
},
"parameters": {
"secret_name_for_server": {
"type": "string"
}
},
"annotations": [],
"type": "AzurePostgreSqlTable",
"schema": [],
"typeProperties": {
"schema": {
"value": "#dataset().schema_name",
"type": "Expression"
},
"table": {
"value": "#dataset().table_name",
"type": "Expression"
}
}
}
}
Pipeline Definition for Postgresql:
{
"name": "your_postgres_pipeline",
"properties": {
"activities": [
{
"name": "Copy_Activity_1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "7.00:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
...
... i skipped definition
...
"inputs": [
{
"referenceName": "your_postgresql_dataset",
"type": "DatasetReference",
"parameters": {
"secret_name_for_server": "secret_name"
}
}
]
}
],
"annotations": []
}
}
I'm trying to get IdentityServer 6 in Blazor WASM (an Asp.Net hosted solution) to get to work with a Microsoft id but am running into the error message "AADSTS90023: Public clients can't send a client secret". I feel like I've tried every configuration I can think off but hopefully I still missed something.
The IdentityServer configuration uses the 'SPA' profile which I believe is correct for my scenario:
"IdentityServer": {
"Clients": {
"Blazor.Client": {
"ClientId": "Blazor.Client",
"ClientName": "Blazor.Client",
"Profile": "SPA",
"RedirectUri": "https://localhost:15601",
"LogoutUri": "https://localhost:15601"
}
}
}
The configuration code follows the simplest example:
.AddMicrosoftAccount(options =>
{
options.ClientId = <clientId>;
options.ClientSecret = <secret>
})
Because these code snippets are basically the simplest way of doing this I'm assuming something is wrong with my AzureAD application registration but I can't figure out what. I've included the manifest:
{
"id": "<id>",
"acceptMappedClaims": null,
"accessTokenAcceptedVersion": 2,
"addIns": [],
"allowPublicClient": false,
"appId": "<apiId>",
"appRoles": [],
"oauth2AllowUrlPathMatching": false,
"createdDateTime": "2021-12-10T09:21:08Z",
"certification": null,
"disabledByMicrosoftStatus": null,
"groupMembershipClaims": null,
"identifierUris": [
"api://<apiId>"
],
"informationalUrls": {
"termsOfService": null,
"support": null,
"privacy": null,
"marketing": null
},
"keyCredentials": [],
"knownClientApplications": [],
"logoUrl": null,
"logoutUrl": null,
"name": "<name>",
"oauth2AllowIdTokenImplicitFlow": false,
"oauth2AllowImplicitFlow": false,
"oauth2Permissions": [],
"oauth2RequirePostResponse": false,
"optionalClaims": null,
"orgRestrictions": [],
"parentalControlSettings": {
"countriesBlockedForMinors": [],
"legalAgeGroupRule": "Allow"
},
"passwordCredentials": [
{
"customKeyIdentifier": null,
"endDate": "2023-12-10T09:21:45.315Z",
"keyId": "<keyId>",
"startDate": "2021-12-10T09:21:45.315Z",
"value": null,
"createdOn": "2021-12-10T09:21:57.2927675Z",
"hint": "H1f",
"displayName": "<displayName>"
}
],
"preAuthorizedApplications": [],
"publisherDomain": "<publisherDomain>.onmicrosoft.com",
"replyUrlsWithType": [
{
"url": "https://localhost:15602/signin-microsoft",
"type": "Spa"
}
],
"requiredResourceAccess": [
{
"resourceAppId": "00000003-0000-0000-c000-000000000000",
"resourceAccess": [
{
"id": "<someId>",
"type": "Scope"
}
]
}
],
"samlMetadataUrl": null,
"signInUrl": null,
"signInAudience": "AzureADandPersonalMicrosoftAccount",
"tags": [],
"tokenEncryptionKeyId": null
}
Is this scenario not supported or am I doing something wrong?
EDIT: ultimately the problem lied in the redirect uri platform which should not be set to 'SPA' but to 'Web' because it isn't the client doing the authentication but the IdentityServer web service. The relevant part would be:
"replyUrlsWithType": [
{
"url": "https://localhost:15602/signin-microsoft",
"type": "Web"
}
The
client secret actually must be kept secret, i.e; you cannot put it in the website
and use it from a public front-end .Client credentials flow design says the same.
Blazor webassembly applications are called a 'public
application' in oAuth/openid terms.
Note: According to microsoft docs:
Public clients (native applications and single page apps) must not use
secrets or certificates when redeeming an authorization code - always
ensure that your redirect URIs correctly indicate the type of
application
So try to disable the requirement for the client secret on the IdP and refresh tokens since they can't also be handled in a secure/safe way.
The recommendation is that you use code+PKCE for public clients,
which happens automatically when you set the response_type to code.
Reference: Microsoft identity platform and OAuth 2.0 authorization code flow - Microsoft identity platform | Microsoft Docs
Setup: I have CDC enabled on MS SQL Server and the CDC events are fed to Kafka using debezium kafka connect(source). Also more than one table CDC events are routed to a single topic in Kafka.
Question: Since I have more than one table data in the kafka topic, I would like to have the table name and the database name in the CDC data.
I am getting the table name and database name in MySQL CDC but not in MS SQL CDC.
Below is the Debezium source connector for the SQL Server
curl -i -X POST -H "Accept:application/json" -H "Content-Type:application/json" localhost:8083/connectors/ -d '{
"name": "cdc-user_profile-connector",
"config": {
"connector.class": "io.debezium.connector.sqlserver.SqlServerConnector",
"tasks.max": "1",
"database.hostname": "<<hostname>>",
"database.port": "<<port>>",
"database.user": "<<username>>",
"database.password": "<<password>>",
"database.server.name": "test",
"database.dbname": "testDb",
"table.whitelist": "schema01.table1,schema01.table2",
"database.history.kafka.bootstrap.servers": "broker:9092",
"database.history.kafka.topic": "digital.user_profile.schema.audit",
"database.history.store.only.monitored.tables.ddl": true,
"include.schema.changes": false,
"event.deserialization.failure.handling.mode": "fail",
"snapshot.mode": "initial_schema_only",
"snapshot.locking.mode": "none",
"transforms":"addStaticField,topicRoute",
"transforms.addStaticField.type":"org.apache.kafka.connect.transforms.InsertField$Value",
"transforms.addStaticField.static.field":"source_system",
"transforms.addStaticField.static.value":"source_system_1",
"transforms.topicRoute.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.topicRoute.regex":"(.*)",
"transforms.topicRoute.replacement":"digital.user_profile",
"errors.tolerance": "none",
"errors.log.enable": true,
"errors.log.include.messages": true,
"errors.retry.delay.max.ms": 60000,
"errors.retry.timeout": 300000
}
}'
I am getting the below output (Demo data)
{
"before": {
"profile_id": 147,
"email_address": "test#gmail.com"
},
"after": {
"profile_id": 147,
"email_address": "test_modified#gmail.com"
},
"source": {
"version": "0.9.4.Final",
"connector": "sqlserver",
"name": "test",
"ts_ms": 1556723528917,
"change_lsn": "0007cbe5:0000b98c:0002",
"commit_lsn": "0007cbe5:0000b98c:0003",
"snapshot": false
},
"op": "u",
"ts_ms": 1556748731417,
"source_system": "source_system_1"
}
My requirement is to get as below
{
"before": {
"profile_id": 147,
"email_address": "test#gmail.com"
},
"after": {
"profile_id": 147,
"email_address": "test_modified#gmail.com"
},
"source": {
"version": "0.9.4.Final",
"connector": "sqlserver",
"name": "test",
"ts_ms": 1556723528917,
"change_lsn": "0007cbe5:0000b98c:0002",
"commit_lsn": "0007cbe5:0000b98c:0003",
"snapshot": false,
"db": "testDb",
"table": "table1/table2"
},
"op": "u",
"ts_ms": 1556748731417,
"source_system": "source_system_1"
}
This is planned as a part of https://issues.jboss.org/browse/DBZ-875 issue
Debezium Kafka-Connect generally puts data from each table in a separate topic and the topic name is of the format hostname.database.table. We generally use the topic name to distinguish between the source table & database name.
If you are putting the data from all the tables manually into one topic then you might have to add the table and database name manually as well.
I am trying out Opserver to monitor SQL Server instances. No issue with configuring standalone instances, but when I tried to configure SQL Server clusters using the method documented here: http://www.patrickhyatt.com/2013/10/25/setting-up-stackexchanges-opserver.html
I am confused about where to put SQL Server cluster named instance and Windows node servers:
In the JSON code below:
{
"defaultConnectionString": "Data Source=$ServerName$;Initial Catalog=master;Integrated Security=SSPI;",
"clusters": [
{
"name": "SDCluster01",
"nodes": [
{ "name": "SDCluster01\\SDCluster01_01" },
{ "name": "SDCluster02\\SDCluster01_02" },
]
},
],
I assume SDCLuster01 is the instance DNS name and SDCluster01_01 and SDCluster01_02 are Windows node server names.
But what if I have a named instance (clustered) like SDCluster01\instance1?
I tried to configure it like this:
{
"defaultConnectionString": "Data Source=$ServerName$;Initial Catalog=master;Integrated Security=SSPI;",
"clusters": [
{
"name": "SDCluster01\instance1",
"nodes": [
{ "name": "SDCluster01\\SDCluster01_01" },
{ "name": "SDCluster02\\SDCluster01_02" },
]
},
],
But after deploying to Opserver it gave me this error message:
[NullReferenceException: Object reference not set to an instance of an object.]
Any ideas on how to configure the JSON file correctly for SQL Server clusters?