I am working with SQL Server full text search. The issue is SQL Server is returning the wrong records.
For example: I am searching for was word in article's table column striptitle
SELECT
TitleStripped
FROM
[pastic_com].[dbo].[Psa_Articles]
WHERE
FREETEXT (TitleStripped, 'was')
With this query, I found 8 records; for reference two of them are pasted below:
Seasonal dynamics and relative abundance of AM fungi in rhizosphere of rice (Oryza sativa L. cv. Basmati supper).
Seasonal dynamics of AM fungi in sugarcane (Saccharum officinarum L.CV.SPF-213) in relation to red rot (Colletotrichum falcatum) disease from Punjab, Pakistan.
You will notice title column does not contain "was" word .
For more reference here's a screenshot:
[1]: https://i.stack.imgur.com/w0gdI.png
The full text search depends on thesaurus files and stoplist objects. Please double check your configuration for entries related to was.
Also, note the difference between FREETEXT and CONTAINS. If you look for exact matches of the word was then try CONTAINS instead of FREETEXT for the reason below.
Snippet from the documentation for FREETEXT, you probably want to avoid these actions.
Is a predicate used in the Transact-SQL WHERE clause of a Transact-SQL
SELECT statement to perform a SQL Server full-text search on full-text
indexed columns containing character-based data types. This predicate
searches for values that match the meaning and not just the exact
wording of the words in the search condition. When FREETEXT is used,
the full-text query engine internally performs the following actions
on the freetext_string, assigns each term a weight, and then finds the
matches:
Separates the string into individual words based on word boundaries
(word-breaking).
Generates inflectional forms of the words (stemming).
Identifies a list of expansions or replacements for the terms based on
matches in the thesaurus.
Related
I'm trying to figure out where the matches were found when using FREETEXT so I can extract the paragraph in which they appear.
I can do this using CONTAINS by searching the the exact phrase in the document. However, because FREETEXT uses a more "fuzzy" approach and uses synonyms, I have no idea what it matched on.
For example, assume I have a column that includes the text "...this will help marginalized communities..."
select #CONTAINS(some_column, '"marginalized groups"')
will return the example column above. But, I can't find the paragraph is appears in because I'm looking for one search term, but FREETEXT is smart enough to find similar terms.
Is there any way to find what the actual match was?
I have a very strange behavior on sql server.
I have a User table with one row having BRAVO as last name.
When I use this simple request:
select * from User u where contains (u.LastName, 'BRAVO')
it finds no result.
If I update User table and set the lastname BRAVO to CRAVO (or any other letter) and call
select * from User u where contains (u.LastName, 'CRAVO')
it will work.
Is BRAVO a reserved word in SQL server? Am I missing something?
Thx
By default when you create a full text index it is associated with a system stoplist.Default stoplist has more than 150 words for english language.You can run below query and see all the stop words for english language for a particular database.
SELECT stopword,language_id FROM sys.fulltext_system_stopwords WHERE language_id = 1033
What is a stop list ? :
Stopwords are managed in databases by using objects called stoplists. A stoplist is a list of stopwords that, when associated with a full-text index, is applied to full-text queries on that index.
What is a stop word ? :
To prevent a full-text index from becoming bloated, SQL Server has a mechanism that discards commonly occurring strings that do not help the search. These discarded strings are called stopwords. During index creation, the Full-Text Engine omits stopwords from the full-text index.
You can use the below query to find the system specified stop words in English Language :
SELECT ssw.stopword, slg.name
FROM sys.fulltext_system_stopwords ssw
JOIN sys.fulltext_languages slg
ON slg.lcid = ssw.language_id
WHERE slg.lcid =1033
So, if you include these words in full text search, including BRAVO (for exact match, match giving in double quotes) it won’t give you the exact result.
I have a bunch of articles in one table that I'd like to query for search results. Using Full Text Search I can return a list of items that have the search keywords "near" each other.
Full text search does not seem to allow thesaurus (FORMSOF) with the NEAR delimiter.
What I'd like to do, in SQL, is create a query, or a number of queries, which search the same data, in different ways, and return a score (or RANK if using Full Text Search), then I would like to merge these results so there are no duplicates, and total up the ranks/scores, so that I can ORDER BY those scores.
Add in that I would also like to search a separate link table of "tags" that the documents have been assigned, and also assign extra score for those with corresponding tags.
What is the best practice way of fulfilling these requirements?
Full-text search can do search like ('"word*" near "another*"') in CONTAINSTABLE statement. The asterisk will help to search any words started with 'word' and 'another' near each other with ranking.
On the other side you can launch FORMSOF(Thesaurus, word) AND FORMSOF(Thesaurus, another) search with CONTAINSTABLE statement.
Then MERGE the results and use ORDER BY to sort by both given RANKs.
I'm using SQL Server 2012 and have created full-text index for NAME column in COMPANY table. All the searches I've tested are of the following format (with variable number of words to search), matching by beginnings of words in any order:
select id, name from company where contains(name, '"ka*" AND "de*"')
The problem is that there are cases where this query doesn't return any results even though it should be perfect match. For example when company name is "ka de we oy", the example above returns a match but '"ka*" AND "de*" AND "we*"' does not and neither does searching with all the four 'words'.
There are also other cases where, strangely enough, the search does not return results even with exact words. This seems related to very short (two-letter) words. There are also some issues with searching with many (6+) words.
Is there some explicit restriction to the number of words in a single query or how short they can be? How can I fix or work around this?
Edit: it seems to be certain common English words which are entirely excluded from the index (like 'we' in the example). This is an issue since it's a requirement that a few of the common words definitely should be searchable. Is there any way to change which words are not indexed or e.g. change the 'language' of the indexing to apply different set of common words that are left out?
Apparently this is simply a case of defining correct stopwords / stoplist:
https://msdn.microsoft.com/en-us/library/ms142551.aspx
https://msdn.microsoft.com/en-us/library/cc280405.aspx
Or setting the full-text index language for the column to the actual language so that English words don't cause issues.
Edit: actually it was easiest to simply disable the stoplist for the table entirely:
ALTER FULLTEXT INDEX ON company SET STOPLIST = OFF
Hopefully this helps someone else
I'm using SQL Server fulltext to search text in large varchar/varbinary columns that may contain IP addresses. I understand that the dots in the address aren't in the index, but I thought a phrase search such as this would work:
select * from myFTETable where contains(myFTEcolumn, "192 168 100 101")
It doesn't. What am I doing wrong? Is there a way to search for IP addresses, or more generally is there a way to do a phrase search when the phrase in the original data contains punctuation?
Thanks.
You're right, the LIKE operator doesn't take advantage of the fulltext index resulting in long query runtimes as your database grows.
Have you tried querying the internal index table to see what numbers are being indexed? This can be accomplished by running -
SELECT * FROM sys.dm_fts_index_keywords(db_id('{database}'), object_id('{table}'))
Inserting 192.168.100.101 in a full-text indexed table shows up as 8-distinct entries internally(4 numeric, 4 character) and running a CONTAINS('"192 168 100 101"') brings up the relevant row.
As a caveat, fulltext will strip some of the lower numbers as part of its stoplists mechanism. This can be overridden by specifying STOPLIST OFF during index creation or removing the matching strings from the internal stoplist.