How to write to Azure Search Index from Synapse Spark?
Synapse has a Linked service connection to Azure Search but I am unable to find documentation on how to use it.
Azure search has a provider for Synapse but it does not follow the regular pattern.
Here is the code.
dataFrame \
.withColumn('searchAction', lit('upload')) \
.writeToAzureSearch(subscriptionKey=TokenLibrary.getSecret('your-key-vault','azure-search-key'),
actionCol="searchAction",
serviceName='your-search-service',
indexName='your-search-index',
batchSize='1000',
keyCol="id");
The magic column 'searchAction' tell Azure Search what do to like "upload", "delete", "merge", or "mergeOrUpload".
The keyCol is used to upsert or delete.
Related
So first I wanted to import a table from Powerapps dataverse to use it on Azure ML, for this I created an Azure Synapse link for the table into the Azure Portal. Now from the Storage account which has my table I tried creating a datastore and then select the path. But I am facing an issue because whenever I click on select path it appears blank and shows error that-
Error when accessing datastore: Unable to access data because it does not exist, is behind a virtual network, or you do not have Storage Blob Data Reader role on this storage account.
Even I tried using the Import Data function to directly link the Synapse link which couldn't happen.
So can anyone help me out in this?
I have few tables created in Databricks workspace and I want to give read access to these tables for certain members.
Currently we can create groups in Databricks Workspace and provide SQL Workspace access, Cluster Creation permission.
Is it possible that I can add an Azure AD security group as a user in Databricks and control access to it rather than creating groups in databricks workspace?
You can't add AAD group as a user of the workspace - you just need to sync necessary AAD groups and users from them into the Databricks workspace. You can do that using one of the tools:
Azure Databricks SCIM Connector that will sync groups & users automatically
Combine user & group resources of the Databricks Terraform Provider with Azure AD Terraform provider - with it you can easily pull groups & users from AAD, and apply these data to create users & groups in Databricks.
Use Databricks SCIM REST API to provision users - you'll need to script it.
P.S. It's also usually not recommended to give users cluster creation permissions, otherwise the costs will be out of the control. Instead, just create shared clusters, and SQL endpoints. Or at least create a cluster policy with predefined rules about cluster size, etc.
Have a table in SQL Server with a column "ModifiedBy" with Azure AD users GUIDs.
How can I get back the usernames - in an SQL query?
Actually need this for a Power BI report - so a conversion function in Power BI would also work.
You can use PowerShell script to get the list of Azure AD users with GUID, Username and Display name. This data can be exported to a CSV File.
Please use the below PowerShell script :
Connect-AzureAD
get-azureaduser | Select-Object ObjectId,UserPrincipalName,DisplayName | export-csv csv-file-path
This script will fetch all the users in the Azure AD with GUID, Username and Display name and export it to a CSV file.
Now you can import the data from your CSV file to SQL Server or Azure SQL Database by following any of the methods described in the below document :
Import data from Excel to SQL - SQL Server | Microsoft Docs
You can use JOIN to query the two tables and get the usernames
They are trying to migrate Oracle database into snowflake. THey are going to convert all existing oracle accounts to snowflake accounts. Now, if AD LDAP logon is enabled, does new users have to only be created in LDAP end, and will those be able to be tagged to accounts while conencting to snowflake? say currently there are 10 oracle accounts which was converted into 10 snowflake acccounts. If LDAP is enabled, can multiple employee IDs be added to use one of 10 snowflake account, by creating LDAP account.
Sorry if I sound dumb. I am not too experienced i LDAP/AD/admin work
You usually have one Azure AD and need configure every snowflake account to use it.
Which employee can access which snowflake account is controlled on the snowflake side, where you need to create a database user.
Say "Martina" needs access to Snowflake Account A and B. You need her in the Azure AD, as well as create her user in Account A and B.
You can find more details how to do this here:
https://docs.snowflake.com/en/user-guide/admin-security-fed-auth-configure-snowflake.html
Just had a query that can WSO2 Identity Server be integrated with both LDAP and DB at the same time. To elaborate my query more, say we have a group of users defined in LDAP and another group of users defined in DB and I want Identity Server to act authorization gateway for both of these groups. Is it possible?
Also, while integrating with DB WSO2 adds its own tables in our existing DB. Is there any way we can add custom attributes(claims) support in DB without altering the WSO2 tables?
WSO2 Identity Server supports only one active user realm at a given time. But if your requirement is to use WSO2 IS as an authorization gateway which connects to a LDAP server as well to a DB for populating claims, etc then it is possible to use XACML support in WSO2 IS with a custom PIP(policy information point). In that case, you can connect to the LDAP server as the primary user store and write a custom PIP to connect to the DB to read the required claims.
Following two posts on PIPs will be helpful.
[1] - http://xacmlinfo.com/2011/12/18/understanding-pip/
[2] - http://blog.facilelogin.com/2011/04/xacml-policy-information-point.html
Answering the second query, with the default configuration in the JDBC mode, WSO2 IS uses its own set of tables for maintaining its user store. But if required, you can write your own user store implementation which connects to your database.