I have a table in SQL 2012 that I'm performing a full text search on.
One of the records has, as a part of a larger string, the text 'Trying out your system'.
The problem is that, if I search for two words in the target string which are too close together, I don't get a match.
select * from mytable where contains(*,'trying') -- match
select * from mytable where contains(*,'trying and out') -- no match
select * from mytable where contains(*,'trying and your') -- no match
select * from mytable where contains(*,'trying and system') -- match
I'm aware that I can search for an exact string by enclosing the search pattern in double quotes, however that's not really what I'm after.
Any suggestions how I can make all of the above search terms match?
Thanks.
This sounds like an issue with stopwords (common words like "the", "your", etc. that are usually filtered out of the full text index, thus you cannot search on them).
To prevent this from happening, you can modify your full text index so that it does not use a stoplist (in other words, every single word will be indexed and thus searchable).
ALTER FULLTEXT INDEX ON MyTable SET STOPLIST = OFF
Be sure to rebuild the full text catalog afterwards.
But only do this if you really need the ability to search on common words. Typically this is not necessary. Also, doing so may slow down your full text searches.
Related
I have a very strange behavior on sql server.
I have a User table with one row having BRAVO as last name.
When I use this simple request:
select * from User u where contains (u.LastName, 'BRAVO')
it finds no result.
If I update User table and set the lastname BRAVO to CRAVO (or any other letter) and call
select * from User u where contains (u.LastName, 'CRAVO')
it will work.
Is BRAVO a reserved word in SQL server? Am I missing something?
Thx
By default when you create a full text index it is associated with a system stoplist.Default stoplist has more than 150 words for english language.You can run below query and see all the stop words for english language for a particular database.
SELECT stopword,language_id FROM sys.fulltext_system_stopwords WHERE language_id = 1033
What is a stop list ? :
Stopwords are managed in databases by using objects called stoplists. A stoplist is a list of stopwords that, when associated with a full-text index, is applied to full-text queries on that index.
What is a stop word ? :
To prevent a full-text index from becoming bloated, SQL Server has a mechanism that discards commonly occurring strings that do not help the search. These discarded strings are called stopwords. During index creation, the Full-Text Engine omits stopwords from the full-text index.
You can use the below query to find the system specified stop words in English Language :
SELECT ssw.stopword, slg.name
FROM sys.fulltext_system_stopwords ssw
JOIN sys.fulltext_languages slg
ON slg.lcid = ssw.language_id
WHERE slg.lcid =1033
So, if you include these words in full text search, including BRAVO (for exact match, match giving in double quotes) it won’t give you the exact result.
I have a FULLTEXT column called 'name', which may contain multiple words per row. Using CONTAINS, I can search for a variable number of words, and it works, except when one of the words has an ampersand in it.
Example: name = 'Bob Brown AB&CD'
This works:
CONTAINS(name, '"*Bo*" AND "*Br*"')
These do not (any search of the word with the ampersand):
CONTAINS(name, '"*AB*"')
CONTAINS(name,'"*AB&CD*"')
CONTAINS(name,'"*&*"')
I realize why the last search wouldn't work, since CONTAINS only searches from the beginning of words.
Is the ampersand being processed by FULLTEXT as a word-break? If so, that might explain why a search including the ampersand would fail. But it wouldn't explain why
CONTAINS(name, '"*AB*"')
CONTAINS(name, '"*CD*"') (assuming FULLTEXT sees "CD" as a different word)
would fail.
How can I mitigate the search so it returns the row if I search for "AB&CD" or just "AB" or just "CD"?
P.S. Dynamic SQL is not an option, so I cannot concatenate a variable number of LIKEs.
I found my answer, apparently. I set
STOPLIST = OFF
on my FULLTEXT index, and I can now search by "AB" and "AB&CD" successfully.
I'm using SQL Server 2012 and have created full-text index for NAME column in COMPANY table. All the searches I've tested are of the following format (with variable number of words to search), matching by beginnings of words in any order:
select id, name from company where contains(name, '"ka*" AND "de*"')
The problem is that there are cases where this query doesn't return any results even though it should be perfect match. For example when company name is "ka de we oy", the example above returns a match but '"ka*" AND "de*" AND "we*"' does not and neither does searching with all the four 'words'.
There are also other cases where, strangely enough, the search does not return results even with exact words. This seems related to very short (two-letter) words. There are also some issues with searching with many (6+) words.
Is there some explicit restriction to the number of words in a single query or how short they can be? How can I fix or work around this?
Edit: it seems to be certain common English words which are entirely excluded from the index (like 'we' in the example). This is an issue since it's a requirement that a few of the common words definitely should be searchable. Is there any way to change which words are not indexed or e.g. change the 'language' of the indexing to apply different set of common words that are left out?
Apparently this is simply a case of defining correct stopwords / stoplist:
https://msdn.microsoft.com/en-us/library/ms142551.aspx
https://msdn.microsoft.com/en-us/library/cc280405.aspx
Or setting the full-text index language for the column to the actual language so that English words don't cause issues.
Edit: actually it was easiest to simply disable the stoplist for the table entirely:
ALTER FULLTEXT INDEX ON company SET STOPLIST = OFF
Hopefully this helps someone else
I have a table with a name field with this
Test O'neill 123
If I use
SELECT *
FROM table F
WHERE CONTAINS ( F.*, '"Test O''neill 123"' )
it works fine but if I use a wildcard * I get no results.
SELECT *
FROM table f
WHERE CONTAINS ( F.*, '"Test O''neill 123*"' )
why is this ?
I am using a parser for my search terms and this is adding the wildcard *
I checked some sites, about escaping the ' but I haven't found anything referred to this..
Thanks in Advance
The problem is due to the combination of 1) using the Neutral language 2) plus a stoplist for your full text index 3) plus unexpected behavior when using a wildcard in a search that includes stopwords.
The Neutral language doesn't cover all of the nuances of the English language, so at index-time it considers O'neill to be 2 separate words O and neill. Then your stoplist considers O to be a stopword so this "word" is not added to the index, only neill is.
At search-time, the search engine typically ignores stop words in multi-word phrases. For example, searching for Contains(*, '"we x people"') will match the text ...we the people..., x and the both being stopwords and thus automatically "matching" each other. (I use the term "matching" loosely because the search engine is not matching the stopwords, but rather it knows that people is 1 word away from we.)
So you might expect the wildcard search Contains(*, '"we the people*"') to also find its match, except that it does not when using a stoplist. If it weren't for the stopword the in the search phrase, or if the was not considered a stopword, the search would work fine. I really can't explain this behavior but I suspect it has something to do with the way the word positions are computed. I also suspect it is not the intended behavior.
So back to your case, Contains(*, '"Test O''neill 123"') will find a match but the wildcard search Contains(*, '"Test O''neill 123*"') does not. (You can even simplify the search to Contains(*, '"O''neill*"') and you'll see that it still does not find a match.) The combination of the stopword O with a wildcard runs into the problem I explained in the last paragraph. This is the crux of the problem stated in your question.
Solutions ranging from most-effective to least-effective-but-possibly-more-practical-for-your-case:
1) Change the language on your full text index to English and re-index. This will cause O'neill to be treated as 1 word and thus you'll avoid the weird wildcard behavior that I explained. You can change the language in the full text index properties via SQL Server Management Studio or by dropping and recreating the index as follows:
ALTER FULLTEXT INDEX ON MyTable DROP (Column1)
GO
ALTER FULLTEXT INDEX ON MyTable ADD (Column1 LANGUAGE [English])
-- repeat for each column in the index
2) If you need to keep using the Neutral language, consider removing O from your stoplist and re-index.
ALTER FULLTEXT STOPLIST MyStoplist DROP 'o' LANGUAGE 'Neutral';
3) Or don't use a stoplist if you don't need one.
ALTER FULLTEXT INDEX ON MyTable SET STOPLIST = OFF
4) If none of the above solutions are practical, consider removing stopwords from the search phrase, or at least the O' prefix in surnames.
After taking a look at this SO question and doing my own research, it appears that you cannot have a leading wildcard while using full text search.
So in the most simple example, if I have a Table with 1 column like below:
TABLE1
coin
coinage
undercoin
select COLUMN1 from TABLE1 where COLUMN1 LIKE '%coin%'
Would get me the results I want.
How can I get the exact same results with FULL TEXT SEARCH enabled on the column?
The following two queries return the exact same data, which is not exactly what I want.
SELECT COLUMN1 FROM TABLE1 WHERE CONTAINS(COLUMN1, '"coin*"')
SELECT COLUMN1 FROM TABLE1 WHERE CONTAINS(COLUMN1, '"*coin*"')
Full text search works on finding words or stems of words. Thus, it does not find the word "coin" anywhere in "undercoin". What you seek is the ability search suffixes using full text searches and it does not do this natively. There are some hacky workarounds like creating a reverse index and searching on "nioc".