Azure Search on text files - azure-cognitive-search

Question

Trying to setup Azure Search on blob container that has text files.
I have created a storage account and I'm using the Azure Data Lake gen2 stored in the blob container. I have one file as I'm testing the azure search. I have created the index, datasource and when I try to create the indexer I get
{"error":{"code":"","message":"Error with data source: The remote server returned an error: (400) Bad Request. Please adjust your data source definition in order to proceed."}}
My file has no headers, contains 16 columns, and is pipe delimited
so here is what I have tried for index
{
"name" : "test-index",
"fields": [
{ "name": "id", "type": "Edm.String", "key": true, "searchable": false },
{"name":"TransactionId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"TransactionEventId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"EventTypeId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"EventSourceId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"SourceUserId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"SourceRecordId", "type": "Edm.String", "key": false, "searchable": true },
{"name":"SourceDetails", "type": "Edm.String", "key": false, "searchable": true },
{"name":"UserGlobalId", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"CallDistributorKey", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"CreatedDateTime", "type": "Edm.DateTimeOffset", "key": false, "searchable": false },
{"name":"AccountId", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"LobId", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"StartEvent", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"EndEvent", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"OnCall", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"PresenceEventId", "type": "Edm.Int32", "key": false, "searchable": false },
{"name":"EventProcessedUtcTime", "type": "Edm.DateTimeOffset", "key": false, "searchable": false }
]
}
---also tried something much simpler
"fields": [
{ "name": "id", "type": "Edm.String", "key": true, "searchable": false },
{ "name": "content", "type": "Edm.String", "key":false, "retrievable": false , "filterable": false, "sortable": false, "facetable":false, "searchable": true}
]
--datasource
{
"name" : "test-ds",
"type" : "azureblob",
"credentials" : { "connectionString" :"DefaultEndpointsProtocol=https;AccountName=......;AccountKey=..." },
"container" : { "name" : "test" }
}
--indexer
{
"name" : "test-indexer",
"dataSourceName" : "test-ds",
"targetIndexName" : "test-index"
}
---get error
{"error":{"code":"","message":"Error with data source: The remote server returned an error: (400) Bad Request. Please adjust your data source definition in order to proceed."}}
--tried this indexer create as well
{
"name" : "test-indexer",
"dataSourceName" : "test-ds",
"targetIndexName" : "test-index",
"parameters" : { "configuration" : { "parsingMode" : "delimitedText", "delimitedTextDelimiter" : "|" , "delimitedTextHeaders" : "TransactionId
TransactionEventId,EventTypeId,EventSourceId,SourceUserIdSourceRecordId,
SourceDetails,UserGlobalId,CallDistributorKey,CreatedDateTime,AccountId,
LobId,StartEvent,EndEvent,OnCall,PresenceEventId,EventProcessedUtcTime" } }
}
---get error
{"error":{"code":"","message":"Error with data source: The remote server returned an error: (400) Bad Request. Please adjust your data source definition in order to proceed."}}
Any pointers would be great....

Answer 1

This is an existing issue of API interoperability between Azure Data Lake Storage Gen2 and Blob Storage.
Azure Search uses Blob Storage APIs and these APIs are currently not compatible with hierarchical namespaces. You can disable the hierarchical namespaces feature to enable Azure Search indexing, but you'll lose some Azure Data Lake Storage Gen2 specific features.