I want to create autosuggestion within Azure Cosmos Db.
I can see that there is Azure cognitive search and Elastic stack services that can be used to achieve this. However, I have couple 1000 records only, and they are not expected to grow very large.
Now Cognitive search and Elastic stack feels like overkill.
So How can I create autosuggestion (index and queries) with Azure cosmos DB?
The Source of my NOSQL Cosmos DB data is SQL DB. I switched from SQL to NOSQL only for autosuggestion thing only because FullText search and LIKE on SQL side are not faster. dotnet core api is responsible for syncing of SQL to NOSQL Db Data.
Related
I am new to Azure and have no prior experience or knowledge regarding working with Azure data warehouse systems (now Azure Synapse Analytics Framework)
I have access to a "read only" data warehouse (not in Azure) that looks like this:
I want to replicate this data warehouse as it is on Azure cloud. Can anyone point me to the right direction (video tutorials or documentation) and the number of steps involved in this process? There are around 40 databases in this warehouse. And what if I wanted to replicated only specific ones?
We can't do that you only have the read only permisson. No matter which data warehouse, we all need the server admin or database owner permission to do the database replicate.
You can easily get this from the all documents relate to the database backup/migrate/replicate, for example: https://learn.microsoft.com/en-us/sql/t-sql/statements/backup-transact-sql?view=sql-server-ver15#permissions,
If you have enough permission then you can to that. But for Azure SQL datawarehouse, now we called SQL pool (formerly SQL DW), we can't replicate other from on-premise datawarehouse to Azure directly.
The official document provide a way import the data into to Azure SQL pool((formerly SQL DW)):
Once your dedicated SQL pool is created, you can import big data with
simple PolyBase T-SQL queries, and then use the power of the
distributed query engine to run high-performance analytics.
You also could use other ETL tool to achieve the data migration from on-premise datawarehouse to Azure. For example using Data Factory, combine these two tutorials:
Copy data to and from SQL Server by using Azure Data Factory
Copy and transform data in Azure Synapse Analytics by using Azure
Data Factory
We have Stage database, which contains stored procedures, and transfers data into an OLTP Database.
Do Elastic SQL Databases reside on same server, and give ability to conduct cross-db stored procedure transactions?
Would Elastic databasepool allow this?
https://learn.microsoft.com/en-us/azure/sql-database/sql-database-elastic-pool
You can achieve elastic transactions spanning across databases using .NET client applications. This is currently under preview.
Elastic transactions at server side are planned in future.
for more information
I have gone through many documents where indices are created but the APIs used are SQL API for Azure cosmos but no example for Gremlin API. Any help would be appreciated. Thanks!!
Azure Cosmos DB is a schema-agnostic database that allows you to iterate on your application without having to deal with schema or index management. By default, Azure Cosmos DB automatically indexes every property for all items in your container without having to define any schema or configure secondary indexes.
For more details, we can see Indexing policies in Azure Cosmos DB.
Azure Cosmos DB support Gremlin API. You can use the indices for your Gremlin queries.
Reference: Querying with indexes
Hope this helps.
We are in the process of developing a web application that we want to host on Azure, where the storage size of the SQL database database is expected to reach 100 TB.
The issue is the max size of the Azure SQL database is currently 4 TB which too small in our case.
Knowing the fact that we are using DTU pricing model, is there any workaround for this issue?
You need to scale out by using SQL elastic pool. You will be using shard database, in other words creating different Azure SQL databases. Azure SQL elastic pool will make it easier to manage the database and query the data in a simple way using one Azure SQL Database elastic query.
I recommend reading the following documents from Microsoft docs:
sql-database-elastic-scale-introduction
sql-database-elastic-scale-shard-map-management
sql-database-elastic-query-overview
I have an existing SQL Server database that uses Full Text Search and Semantic search for the UI's primary searching capability. The tables used in the search contain around 1 million rows of data.
I'm looking at using Azure Search to replace this, however my database relies upon the Full Text Enabled tables for it's core functionality. I'd like to use Azure Search for the "searching" but still have my current table structure in place to be able to edit records and display the detail record when something has been found.
My thoughts to implement this is to:
Create the Azure indexes
Push all of the searchable data from the Full Text enabled table in SQL Server to Azure Search
Azure Search to return ID's of documents that match the search criteria
Query the existing database to fetch the rows that contain those ID's to display on the front end
When some data in the existing database changes, schedule an update in Azure Search to ensure the data stays in sync
Is this a good approach? How do hybrid implementations work where your existing data is in an on-prem database but you want to take advantage of Azure Search?
Overall, your approach seems reasonable. A couple of pointers that might be useful:
Azure SQL now has support for Full Text Search, so if moving to Azure SQL is an option for you and you still want to use Azure Search, you can use Azure SQL indexer. Or you can run SQL Server on IaaS VMs and configure the indexer using the instructions here.
With on-prem SQL Server, you might be able to use Azure Data Factory sink for Azure Search to sync data.
I actually just went through this process, almost exactly. Instead of SQL Server, we are using a different backend data store.
Foremost, we wrote an application to sync all existing data. Pretty simple.
For new documents being added, we made the choice to sync to Azure Search synchronously rather than async. We made this choice because we measured excellent performance when adding to and updating the index. 50-200 ms response time and no failures over hundreds of thousands of records. We couldn't justify the additional cost of building and maintaining workers, durable queues, etc. Caveat: Our web service is located in the same Azure region as the Azure Search instance. If your SQL Server is on-prem, you could experience longer latencies.
We ended up storing about 80% of each record in Azure Search. Obviously, the more you store in Azure Search, the less likely you'll have to perform a worst-case serial "double query."