I want to search a full-text column for the following two terms:
2011
J Vineyards
where I construct the predicate as
"2011*" and "j vineyards*"
no rows are returned.
A record which should match is
2011 j vineyards viognier alexander valley united states
After experimentation, it seems to be related to the single "j" character.
EDIT:
Here is the select statement for the full-text column BeverageSearchData.
Declare #test nvarchar(100);
Set #test='""2011*" and "j vineyards*"';
Select * from bv_beverage WHERE CONTAINS (BeverageSearchData,#test)
There are a couple of possibilities going on here. The full query and not just the predicate would help you get a better answer.
One thing that is probably going on is that SQL Full-text search eliminates single characters (As in J) when building its index.
If using CONTAINS, you may need to change your noise file and restart the SQL Server FullText Search service.
If using LIKE, you may be able try adding an additional single character wildcard. Play with it and see if it works without the 2011 and then add it back in.
WHERE myColumn like 'j_vineyards%'
WHERE myColumn like 'j%vineyards%'
An additional thing to note is that CONTAINS does not supports leading wildcards.
You're looking for %, not *.
Try this instead:
"%2011%" and "%j vineyards%"
What language did you choose when creating the index? SQL Server associates the system full-text stoplist by default when creating an index which is probably what is happening in your case.
Try building the index with STOPLIST OFF like so -
CREATE FULLTEXT INDEX on [table]([column]) ON [catalog] WITH STOPLIST OFF;
Alternatively, you can modify the stoplists to exclude certain words such as 'j' in the example shown above.
I have been struggling with the following query:
select * from table_name where contains((field1,field2),'"S.E.N.S"');
Following is the text I have in field 1:
"S.E.N.S Productions"
If I search for "production" I get the result, but not for "S.E.N.S".
Any idea regarding how to get the desired result ? Thanks.
Update: Sql Server version is 2005 with SP3.
Update: Well, it is quite weird. When I set the full-text to use noiseENG.txt the query in my question works fine. But everything stored in the database is in Turkish and all the settings are set accordingly, including noiseTRK.txt. "S.E.N.S" is not a word in Turkish nor in English as far as I know. I can set it to noiseENG.txt to get it work, but I doubt that would be appropriate. Can anyone know/think of a reason why noiseTRK.txt would break in above query ? Thanks.
P.S. I have tested with unmodified noise files as well as single-letters-removed version of Turkish noise file.
Out of the box, it may not work for SQL 2005.
As I recall, for the string "S.E.N.S Productions", the SQL 2005 FTS (Full Text Search) word breaker will ignore the punctiation, and break the string into individual letters (S E N S) and the word "Productions". Single letters are, by default, noise words, and therefore won't be included in the index.
You can do a couple things:
Edit your noise word list and remove the single letters and rebuild your index
Upgrade to a later version of SQL, as the word breaking behaviour changes to corretly return your result in 2008.
I have full text search on my database.
Is it possible to search in the middle of a word for some text?
For example, I have a column Description that contains the following text:
Revolution
Is it possible to search for 'EVO' and have it find it in the word Revolution or am I stuck doing a LIKE:
SELECT * FROM Table WHERE Description LIKE '%EVO%'
Is there a FTS equivalent of the above query?
EDIT
I want to make it clear what I am trying to ask because it appear a few people might be confused. I believe that SQL Server FTS can only search at the beginning of the word (prefix search). So if I query like:
SELECT * FROM Table WHERE CONTAINS(Description, '"Revo*"')
Then it will find the word Revolution. I want to know if it is possible at all to search something in the MIDDLE of the word. Not at the end. Not at the beginning. From what it looks like this is not possible and it makes sense because how would SQL server index this, but I just wanted to be certain.
This looks like it has come up before and the short answer was "No".
Previous thread
You can use CONTAINS. See this link
Full text catalog/index search for %book%
The only way to do this search is to add a "rotational" break down of the words.
As an exemple, the word "locomotion" will be break down into 9 new "word" like :
"ocomotion"
"comotion"
"omotion"
"motion"
"otion"
"tion"
"ion"
"on"
"n"
So now you can put this table into the Full Text Search (or create à new columns with all these parts of the word) to find it quickly.
I wrote a paper to do that without FTS (but it is in french) :
https://blog.developpez.com/sqlpro/p13123/langage-sql-norme/like-mot-ou-les-index-rotatifs
You can use CONTAINS instead of LIKE.
SELECT *
FROM Table
WHERE CONTAINS(Description, '"EVO*"')
I'm trying to wrap my head around how to search for something that appears in the middle of a word / expression - something like searching for "LIKE %book% " - but in SQL Server (2005) full text catalog.
How can I do that? It almost appears as if both CONTAINS and FREETEXT really don't support wildcard at the beginning of a search expression - can that really be?
I would have imagined that FREETEXT(*, "book") would find anything with "book" inside, including "rebooked" or something like that.
unfortunately CONTAINS only supports prefix wildcards:
CONTAINS(*, '"book*"')
SQL Server Full Text Search is based on tokenizing text into words. There is no smaller unit as a word, so the smallest things you can look for are words.
You can use prefix searches to look for matches that start with certain characters, which is possible because word lists are kept in alphabetical order and all the Server has to do is scan through the list to find matches.
To do what you want a query with a LIKE '%book%' clause would probably be just as fast (or slow).
If you want to do some serious full text searching then I would (and have) use Lucene.Net. MS SQL Full Text search never seems to work that well for anything other than the basics.
Here's a suggestion that is a workaround for that wildcard limitation. You create a computed column that contains the same content but in reverse as the column(s) you are searching.
If, for example, you are searching on a column named 'ProductTitle', then create a column named ProductsRev. Then update that field's 'Computed Column Specification' value to be:
(reverse([ProductTitle]))
Include the 'ProductsRev' column in your search and you should now be able to return results that support a wildcard at the beginning of the word. Good luck!!
Full text has a table that lists all the words the engine has found. It should have orders-of-magnitude less rows than your full-text-indexed table. You could select from that table " where field like '%book%' " to get all the words that have 'book' in them. Then use that list to write a fulltext query. Its cumbersome, but it would work, and it would be ok in the speed department. HOWEVER, ultimately you are using fulltext wrong when you are doing this. It might actually be better to educate the source of these feature requests about what fulltext is doing. You want them to understand what it WANTS to do, so they can get high value from fulltext. Example, only use wild cards at the end of a word, which means think of the words in an ordered list.
why don't program an assembly in C# to compute all the non repeated sufixes. For example if you have the Text "eat the red meat" you can store in a field "eat at t the he e red ed d meat" (note that is not necesary to add eat at and t again) ind then in this field use full text search. A function for doing that can easily written in Csharp
x) I know it seems od... it's a workarround
x) I know I'm adding overhead in the insert / update .... only justified if this overhead is insignificant besides the improvement in the search function
x) I know there is also an overhead in the size of the stored data.
But I'm pretty conffident that will be quite fast
Note: I am using SQL's Full-text search capabilities, CONTAINS clauses and all - the * is the wildcard in full-text, % is for LIKE clauses only.
I've read in several places now that "leading wildcard" searches (e.g. using "*overflow" to match "stackoverflow") is not supported in MS SQL. I'm considering using a CLR function to add regex matching, but I'm curious to see what other solutions people might have.
More Info: You can add the asterisk only at the end of the word or phrase. - along with my empirical experience: When matching "myvalue", "my*" works, but "(asterisk)value" returns no match, when doing a query as simple as:
SELECT * FROM TABLENAME WHERE CONTAINS(TextColumn, '"*searchterm"');
Thus, my need for a workaround. I'm only using search in my site on an actual search page - so it needs to work basically the same way that Google works (in the eyes on a Joe Sixpack-type user). Not nearly as complicated, but this sort of match really shouldn't fail.
Workaround only for leading wildcard:
store the text reversed in a different field (or in materialised view)
create a full text index on this column
find the reversed text with an *
SELECT *
FROM TABLENAME
WHERE CONTAINS(TextColumnREV, '"mrethcraes*"');
Of course there are many drawbacks, just for quick workaround...
Not to mention CONTAINSTABLE...
The problem with leading Wildcards: They cannot be indexed, hence you're doing a full table scan.
It is possible to use the wildcard "*" at the end of the word or phrase (prefix search).
For example, this query will find all "datab", "database", "databases" ...
SELECT * FROM SomeTable WHERE CONTAINS(ColumnName, '"datab*"')
But, unforutnately, it is not possible to search with leading wildcard.
For example, this query will not find "database"
SELECT * FROM SomeTable WHERE CONTAINS(ColumnName, '"*abase"')
To perhaps add clarity to this thread, from my testing on 2008 R2, Franjo is correct above. When dealing with full text searching, at least when using the CONTAINS phrase, you cannot use a leading , only a trailing functionally. * is the wildcard, not % in full text.
Some have suggested that * is ignored. That does not seem to be the case, my results seem to show that the trailing * functionality does work. I think leading * are ignored by the engine.
My added problem however is that the same query, with a trailing *, that uses full text with wildcards worked relatively fast on 2005(20 seconds), and slowed to 12 minutes after migrating the db to 2008 R2. It seems at least one other user had similar results and he started a forum post which I added to... FREETEXT works fast still, but something "seems" to have changed with the way 2008 processes trailing * in CONTAINS. They give all sorts of warnings in the Upgrade Advisor that they "improved" FULL TEXT so your code may break, but unfortunately they do not give you any specific warnings about certain deprecated code etc. ...just a disclaimer that they changed it, use at your own risk.
http://social.msdn.microsoft.com/Forums/ar-SA/sqlsearch/thread/7e45b7e4-2061-4c89-af68-febd668f346c
Maybe, this is the closest MS hit related to these issues... http://msdn.microsoft.com/en-us/library/ms143709.aspx
One thing worth keeping in mind is that leading wildcard queries come at a significant performance premium, compared to other wildcard usages.
Note: this was the answer I submitted for the original version #1 of the question before the CONTAINS keyword was introduced in revision #2. It's still factually accurate.
The wildcard character in SQL Server is the % sign and it works just fine, leading, trailing or otherwise.
That said, if you're going to be doing any kind of serious full text searching then I'd consider utilising the Full Text Index capabilities. Using % and _ wild cards will cause your database to take a serious performance hit.
Just FYI, Google does not do any substring searches or truncation, right or left. They have a wildcard character * to find unknown words in a phrase, but not a word.
Google, along with most full-text search engines, sets up an inverted index based on the alphabetical order of words, with links to their source documents. Binary search is wicked fast, even for huge indexes. But it's really really hard to do a left-truncation in this case, because it loses the advantage of the index.
As a parameter in a stored procedure you can use it as:
ALTER procedure [dbo].[uspLkp_DrugProductSelectAllByName]
(
#PROPRIETARY_NAME varchar(10)
)
as
set nocount on
declare #PROPRIETARY_NAME2 varchar(10) = '"' + #PROPRIETARY_NAME + '*"'
select ldp.*, lkp.DRUG_PKG_ID
from Lkp_DrugProduct ldp
left outer join Lkp_DrugPackage lkp on ldp.DRUG_PROD_ID = lkp.DRUG_PROD_ID
where contains(ldp.PROPRIETARY_NAME, #PROPRIETARY_NAME2)
When it comes to full-text searching, for my money nothing beats Lucene. There is a .Net port available that is compatible with indexes created with the Java version.
There's a little work involved in that you have to create/maintain the indexes, but the search speed is fantastic and you can create all sorts of interesting queries. Even indexing speed is pretty good - we just completely rebuild our indexes once a day and don't worry about updating them.
As an example, this search functionality is powered by Lucene.Net.
Perhaps the following link will provide the final answer to this use of wildcards: Performing FTS Wildcard Searches.
Note the passage that states: "However, if you specify “Chain” or “Chain”, you will not get the expected result. The asterisk will be considered as a normal punctuation mark not a wildcard character. "
If you have access to the list of words of the full text search engine, you could do a 'like' search on this list and match the database with the words found, e.g. a table 'words' with following words:
pie
applepie
spies
cherrypie
dog
cat
To match all words containing 'pie' in this database on a fts table 'full_text' with field 'text':
to-match <- SELECT word FROM words WHERE word LIKE '%pie%'
matcher = ""
a = ""
foreach(m, to-match) {
matcher += a
matcher += m
a = " OR "
}
SELECT text FROM full_text WHERE text MATCH matcher
% Matches any number of characters
_ Matches a single character
I've never used Full-Text indexing but you can accomplish rather complex and fast search queries with simply using the build in T-SQL string functions.
From SQL Server Books Online:
To write full-text queries in
Microsoft SQL Server 2005, you must
learn how to use the CONTAINS and
FREETEXT Transact-SQL predicates, and
the CONTAINSTABLE and FREETEXTTABLE
rowset-valued functions.
That means all of the queries written above with the % and _ are not valid full text queries.
Here is a sample of what a query looks like when calling the CONTAINSTABLE function.
SELECT RANK , * FROM TableName ,
CONTAINSTABLE (TableName, *, '
"*WildCard" ') searchTable WHERE
[KEY] = TableName.pk ORDER BY
searchTable.RANK DESC
In order for the CONTAINSTABLE function to know that I'm using a wildcard search, I have to wrap it in double quotes. I can use the wildcard character * at the beginning or ending. There are a lot of other things you can do when you're building the search string for the CONTAINSTABLE function. You can search for a word near another word, search for inflectional words (drive = drives, drove, driving, and driven), and search for synonym of another word (metal can have synonyms such as aluminum and steel).
I just created a table, put a full text index on the table and did a couple of test searches and didn't have a problem, so wildcard searching works as intended.
[Update]
I see that you've updated your question and know that you need to use one of the functions.
You can still search with the wildcard at the beginning, but if the word is not a full word following the wildcard, you have to add another wildcard at the end.
Example: "*ildcar" will look for a single word as long as it ends with "ildcar".
Example: "*ildcar*" will look for a single word with "ildcar" in the middle, which means it will match "wildcard". [Just noticed that Markdown removed the wildcard characters from the beginning and ending of my quoted string here.]
[Update #2]
Dave Ward - Using a wildcard with one of the functions shouldn't be a huge perf hit. If I created a search string with just "*", it will not return all rows, in my test case, it returned 0 records.