It seems easy to apply an Azure Search index to an SQL Azure database. I undertand that you query the search index using REST APIs and that the index then needs to be maintained/updated. Now, consider a web server running IIS, with an underlying SQL Server database.
What is considered best practice; querying and updating the index from the web server or from SQL Server, e.g. from within a CLR stored procedure? Are there specific design considerations here?
I work on Azure Search team and will try to help.
Querying and updating the index are two different use cases. Presumably, you want to query the index in response to user input in your Web app. (It is also possible that you have a SQL stored procedure with some complex logic that needs full test search, but that seems less likely).
Updating the index can be done in multiple ways. If you can tolerate updating your index at most every 5 minutes, use Azure Search SQL indexer automagically update the index for you - see http://azure.microsoft.com/en-us/documentation/articles/search-howto-connecting-azure-sql-database-to-azure-search-using-indexers-2015-02-28/ for details on how to do it. That article describes creating indexers using REST API, but we now have support for that in .NET SDK as well.
OTOH, if you need hard real-time updates, you can update the search index at the same time you produce data to insert / update your SQL database.
Let me know if you have any follow up questions!
HTH,
Eugene
Related
I want to move all data from one Azure SQL Server to different Azure SQL Server which more than 90 days old, and after moving need to delete moved data from first Azure SQL Server.
I want to run these steps on daily basis.
I am new to Azure and able to do same with Azure Data Factory. Can you please suggest any other best suited approach?
You are already using the best approach.
Azure Data Factory is an easy to use when it comes to extract and copy the data between the services. It also provide scheduling the triggers, i.e., triggering the copy pipeline after specific interval of time or any event. Refer Create a trigger that runs a pipeline on a schedule.
If the volume of data is large, you can re-configure the Integration Runtime (IR) resources (Compute type and Core count) to overcome the performance issue, if required. Refer below image.
I want to sync any DML operations on Azure SQL DB to azure search index immediately.
I have gone through this question.
How does Auto-indexing/sync of Azure SQL DB with Azure Search Works?
No answer in this question. This was posted almost 5 yrs back.
With integrated change policy in place do we have auto sync feature by any means now.
Function app does not have a SQL trigger event attached.
I don't want to do a while true loop or any timer or call indexer when data gets update.
Please suggest if there are any other best approach or any build feature.
Azure Functions don't have a SQL trigger but Logic Apps do: https://learn.microsoft.com/en-us/azure/connectors/connectors-create-api-sqlazure Logic Apps can also trigger functions and custom APIs, so you should be able to trigger an indexing operation from a SQL operation this way. Do keep in mind however that the indexing process itself may be delayed or take time once triggered so your index may not immediately reflect the changes, depending on payload.
There are two ways you can integrate Azure SQL data with Azure Search.
Built-in indexer. You have the built-in index support for Azure SQL. It supports incremental updating of the index, but with a limited refresh ratio. Currently, you can run incremental indexing every 5 minutes at the most. See Connect to and index Azure SQL content using an Azure Cognitive Search indexer
Push API. To support immediate updates, you have to push data via the Push API. In this case you only create the index, not the indexer. Your code that pushes content to Azure SQL is responsible for pushing content to Azure Search. Check out this example Tutorial: Optimize indexing with the push API
I want to use Elastic search for full text searching in mu SQL server.
As i read I can index my SQL server by elastic search.
My questions is about resource usage.
As I understood by indexing I must duplicate my entire SQL server database.
and I must take both elastic and SQL server sync.
Am I wrong?
Thanks
Yes, you are right. I also maintain a sql server + elastic environment.
You need a separate instance of elasticsearch running.
You will have to index / insert all your existing data from sql server to elastic. There are libraries for almost any language to easily insert data.
You will have to sync these two in order to keep the data up to date.
In theory you only need 1 instance / node of elastic running, but it is recommended to have a full cluster, due to failover, etc.
I want to know if it's posible to share SQL Server SESSION CONTEXT variables between different Azure Sql databases using Elastic Queries.
I searched in official documentation but i can't found any information about this feature is available or not.
SESSION CONTEXT exists locally to a single server instance in SQL Server. (It's tied to a session). SQL Azure is built using SQL Server but there are some parts of the mapping that are opaque to customers (they can change based on circumstances such as what Edition you use or what version of the internal software we are using to deliver the service).
Elastic Queries is a feature to let you query from one database (source) to one or more other databases (target(s)). In such a model, you have a SQL Server session to the source database, and the elastic query has a separate connection/session to each other database being touched.
I think the question you are asking is "can I set the session context on the source connection/session and have it flow through to all the target connections when running queries there?" (That's my best guess - let me know if it is different). The answer today is "no" - the session variables do not flow from source to target as part of the elastic query. Also, since today elastic query is read-only, you can't use elastic query to set the session context individually on each target database connection/session as part of the operation.
In the future, we'll consider whether there is something like this we can do, but right now we don't have a committed timeline for something like this.
I hope this explains how things work a bit under the convers.
Sincerely,
Conor Cunningham
Architect, SQL
I have an existing SQL Server database that uses Full Text Search and Semantic search for the UI's primary searching capability. The tables used in the search contain around 1 million rows of data.
I'm looking at using Azure Search to replace this, however my database relies upon the Full Text Enabled tables for it's core functionality. I'd like to use Azure Search for the "searching" but still have my current table structure in place to be able to edit records and display the detail record when something has been found.
My thoughts to implement this is to:
Create the Azure indexes
Push all of the searchable data from the Full Text enabled table in SQL Server to Azure Search
Azure Search to return ID's of documents that match the search criteria
Query the existing database to fetch the rows that contain those ID's to display on the front end
When some data in the existing database changes, schedule an update in Azure Search to ensure the data stays in sync
Is this a good approach? How do hybrid implementations work where your existing data is in an on-prem database but you want to take advantage of Azure Search?
Overall, your approach seems reasonable. A couple of pointers that might be useful:
Azure SQL now has support for Full Text Search, so if moving to Azure SQL is an option for you and you still want to use Azure Search, you can use Azure SQL indexer. Or you can run SQL Server on IaaS VMs and configure the indexer using the instructions here.
With on-prem SQL Server, you might be able to use Azure Data Factory sink for Azure Search to sync data.
I actually just went through this process, almost exactly. Instead of SQL Server, we are using a different backend data store.
Foremost, we wrote an application to sync all existing data. Pretty simple.
For new documents being added, we made the choice to sync to Azure Search synchronously rather than async. We made this choice because we measured excellent performance when adding to and updating the index. 50-200 ms response time and no failures over hundreds of thousands of records. We couldn't justify the additional cost of building and maintaining workers, durable queues, etc. Caveat: Our web service is located in the same Azure region as the Azure Search instance. If your SQL Server is on-prem, you could experience longer latencies.
We ended up storing about 80% of each record in Azure Search. Obviously, the more you store in Azure Search, the less likely you'll have to perform a worst-case serial "double query."