I have an existing SQL Server database that uses Full Text Search and Semantic search for the UI's primary searching capability. The tables used in the search contain around 1 million rows of data.
I'm looking at using Azure Search to replace this, however my database relies upon the Full Text Enabled tables for it's core functionality. I'd like to use Azure Search for the "searching" but still have my current table structure in place to be able to edit records and display the detail record when something has been found.
My thoughts to implement this is to:
Create the Azure indexes
Push all of the searchable data from the Full Text enabled table in SQL Server to Azure Search
Azure Search to return ID's of documents that match the search criteria
Query the existing database to fetch the rows that contain those ID's to display on the front end
When some data in the existing database changes, schedule an update in Azure Search to ensure the data stays in sync
Is this a good approach? How do hybrid implementations work where your existing data is in an on-prem database but you want to take advantage of Azure Search?
Overall, your approach seems reasonable. A couple of pointers that might be useful:
Azure SQL now has support for Full Text Search, so if moving to Azure SQL is an option for you and you still want to use Azure Search, you can use Azure SQL indexer. Or you can run SQL Server on IaaS VMs and configure the indexer using the instructions here.
With on-prem SQL Server, you might be able to use Azure Data Factory sink for Azure Search to sync data.
I actually just went through this process, almost exactly. Instead of SQL Server, we are using a different backend data store.
Foremost, we wrote an application to sync all existing data. Pretty simple.
For new documents being added, we made the choice to sync to Azure Search synchronously rather than async. We made this choice because we measured excellent performance when adding to and updating the index. 50-200 ms response time and no failures over hundreds of thousands of records. We couldn't justify the additional cost of building and maintaining workers, durable queues, etc. Caveat: Our web service is located in the same Azure region as the Azure Search instance. If your SQL Server is on-prem, you could experience longer latencies.
We ended up storing about 80% of each record in Azure Search. Obviously, the more you store in Azure Search, the less likely you'll have to perform a worst-case serial "double query."
Related
I have a client that does not want the documents to permanently live in Azure. They are ok with moving files up to Azure to be indexed but after indexing they want the files to be removed from Azure and results to point to their on-prem storage.
Is this use-case possible with Azure Cognitive Search?
you can push any data into a search index that you want via the service's REST API as long as you have network connectivity to the search service.
I'm not sure why your client doesn't want to store documents in Azure, but you should make sure they're aware that the ingested document data exists in the search index independently of any source data. That is, if he's concerned about his data being stored in Azure, the indexed data will always be stored in azure, since that's how the search service works.
If you're asking whether it's possible to point an azure search indexer to a datasource that is not hosted in Azure, then no, that's not generally supported. There are some third party organizations (eg: Accenture, BA Insight) that will host a connector to a non-azure datasource on your behalf though.
I want to move all data from one Azure SQL Server to different Azure SQL Server which more than 90 days old, and after moving need to delete moved data from first Azure SQL Server.
I want to run these steps on daily basis.
I am new to Azure and able to do same with Azure Data Factory. Can you please suggest any other best suited approach?
You are already using the best approach.
Azure Data Factory is an easy to use when it comes to extract and copy the data between the services. It also provide scheduling the triggers, i.e., triggering the copy pipeline after specific interval of time or any event. Refer Create a trigger that runs a pipeline on a schedule.
If the volume of data is large, you can re-configure the Integration Runtime (IR) resources (Compute type and Core count) to overcome the performance issue, if required. Refer below image.
I want to sync any DML operations on Azure SQL DB to azure search index immediately.
I have gone through this question.
How does Auto-indexing/sync of Azure SQL DB with Azure Search Works?
No answer in this question. This was posted almost 5 yrs back.
With integrated change policy in place do we have auto sync feature by any means now.
Function app does not have a SQL trigger event attached.
I don't want to do a while true loop or any timer or call indexer when data gets update.
Please suggest if there are any other best approach or any build feature.
Azure Functions don't have a SQL trigger but Logic Apps do: https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-sqlazure Logic Apps can also trigger functions and custom APIs, so you should be able to trigger an indexing operation from a SQL operation this way. Do keep in mind however that the indexing process itself may be delayed or take time once triggered so your index may not immediately reflect the changes, depending on payload.
There are two ways you can integrate Azure SQL data with Azure Search.
Built-in indexer. You have the built-in index support for Azure SQL. It supports incremental updating of the index, but with a limited refresh ratio. Currently, you can run incremental indexing every 5 minutes at the most. See Connect to and index Azure SQL content using an Azure Cognitive Search indexer
Push API. To support immediate updates, you have to push data via the Push API. In this case you only create the index, not the indexer. Your code that pushes content to Azure SQL is responsible for pushing content to Azure Search. Check out this example Tutorial: Optimize indexing with the push API
I'm aware of the various options in place for migrating a single database up to Azure. My problem is that these all only seem to cater for a single database at a time. However, I have a db per tenant model with over 2000 databases to migrate and not a lot of time to play with.
Can anyone point me in towards the best (ie fastest) way of doint this?
In the end we accomplished this with Powershell and the Azure API. Essentially batch creating bacpacs on the source server, uploading them to blob storage then importing them into Azure SQL server pools.
If I was facing the same challenge now I'd take a look at the Azure Database Migration Service - https://azure.microsoft.com/en-gb/services/database-migration/
I am also facing this problem and am going down the route of using the Visual Studio data compare tool.
All my tenant databases have the same schema so I made an empty template database in Azure, and just use the CREATE AS COPY command to make a new one each time ready for receiving the migration.
Then I ask Visual Studio to compare the empty database with the live database and automatically insert the data for me.
Seems to be working well so far, there's very little manual steps needed and it doesn't involve using the Azure Portal, or blob storage or creating databases outside of the elastic pool which is great. But the overall time will be slow to migrate data for all the databases.
It seems easy to apply an Azure Search index to an SQL Azure database. I undertand that you query the search index using REST APIs and that the index then needs to be maintained/updated. Now, consider a web server running IIS, with an underlying SQL Server database.
What is considered best practice; querying and updating the index from the web server or from SQL Server, e.g. from within a CLR stored procedure? Are there specific design considerations here?
I work on Azure Search team and will try to help.
Querying and updating the index are two different use cases. Presumably, you want to query the index in response to user input in your Web app. (It is also possible that you have a SQL stored procedure with some complex logic that needs full test search, but that seems less likely).
Updating the index can be done in multiple ways. If you can tolerate updating your index at most every 5 minutes, use Azure Search SQL indexer automagically update the index for you - see http://azure.microsoft.com/en-us/documentation/articles/search-howto-connecting-azure-sql-database-to-azure-search-using-indexers-2015-02-28/ for details on how to do it. That article describes creating indexers using REST API, but we now have support for that in .NET SDK as well.
OTOH, if you need hard real-time updates, you can update the search index at the same time you produce data to insert / update your SQL database.
Let me know if you have any follow up questions!
HTH,
Eugene