I am trying to implement a fulltext search on two columns which I created a view for: VendorName, ProductName. I have the full text index etc working but the actual query is what is causing some issues for me.
I want users to be able to use some standard search conventions, AND OR NOT and grouping of terms by () which is fine but I want to apply the search over both the columns so for example if I were to run a query such as:
SELECT * FROM vw_Search
WHERE CONTAINS((VendorName, ProductName), "Apple AND iTunes")
It seems to be applying the query to each column individually i.e. checking vendor name for both terms and then checking product name for both terms which wont match unless the vendor was called "Apple iTunes".
If I change the query to :
SELECT * FROM vw_Search
WHERE CONTAINS(VendorName, "Apple OR iTunes")
AND CONTAINS(ProductName, "Apple OR iTunes")
then it works but breaks other search queries (such as searching for just apple) and from user writing the query it doesn't make much sense as what they are likely to write is AND but it requires an OR to work.
What I want is it to return if between the two the search term was valid so it would match all vendors named apple with a product name itunes for example.
Should I create a separate field in the view that concatenates the Vendor and Product fields and performs the first query on that new field or is there something I am missing out?
Aside from that would anyone know of an existing method of validating the queries?
In earlier versions of SQL Server, the queries matched across multiple columns.
However, this was considered a bug.
To match across multiple columns, you should concatenate them in a computed column and create an index over that column.
Related
I'm in the process of writing a SuiteTalk integration, and I've hit an interesting data transformation issue. In the target system, we have a sort of notes table which has a category column and then the notes column. Data going into that table from NetSuite could be several different fields on a single entity in NetSuite terms, but several records of different categories in our terms.
If you take the example of a Sales Order, you might have two text fields that we need to bring across as notes. For each of those fields I need to create a row, with both the notes field in the same column but separate rows. This would allow me to add a dynamic column that give the category for each of those fields.
So instead of
SO number notes 1 notes 2
SO1234567 some text1 some text2
You’d get
SO Number Category Text
SO1234567 category 1 some text1
SO1234567 category 2 some text2
The two problems I’m really trying to solve here are:
Where can I store the category name? It can’t be the field name in NetSuite. It needs to be configurable per customer as the number of notes fields in each record type might vary across implementations. This is currently my main blocker.
Performance – I could create a saved search for each type of note, and bring one row across each time, but that’s not really an acceptable performance hit if I can do it all in one call.
I use Saved Searches in NetSuite to provide a configurable way of filtering the data to import into the target system.
If I were writing a SQL query, i would use the UNION clause, with the first column being a dynamic column denoting the category and the second column being the actual data field from NetSuite. My ideal would be if I could somehow do a similar thing either as a single saved search, or as one saved search per entity, without having to create any additional fields within NetSuite itself, so that from the SuiteTalk side I can just query the search and pull in the data.
As a temporary kludge, I now have multiple saved searches in NetSuite, one per category, and within the ID of the saved search I expect the category name and an indicator of the record type. I then have a parent search which gives me the searches for that record type - it's very clunky, and ultimately results in far too many round trips for me to be satisfied.
Any idea if something like this is at all possible?? Or if not, is there a way of solving this without hard-coding the category values in the front end? Even if I can bring back multiple recordsets in one call, that would be a performance enhancement.
I've asked the same question on the NetSuite forums but to no avail.
Thanks
At first read it sounds like you are trying to query a set of fields from entities. The fields may be custom fields or built in fields. Can you not just query the entities where your saved search has all the potential category columns and then transform the received data into categories?
Otherwise please provide more specifics in Netsuite terms about what you are trying to do.
I've been searching but haven't found how to do this in refine.
I've got two columns of unique IDS. For each a in A, I want to find the top 10 closest matches in B.
My backup plan is to just use Levenshtein to iterate ... but Refine has such a nice iterface and many more algorithms implemented that I was hoping to be able to do some of the work using it.
Or is there another tool for doing this?
Did you know you can use clustering algorithm like fingerprint or ngramFingerprint (source) out of the clustering interface in Refine?
Using you IDS field, create a new column based on this column with the following expression: ngramFingerprint(value)
You can now cross with your other data set on this new column. This might help to get more matches.
I am a bit confused as to where SOLR usage ends and where it begins.
I use php with a relational mysql db for a shopping site where all tables are related to the product table joining the tables as theyre queried. Needless to say its too slow!
e.g.
Category table - catid, catname, catdesc
Brand table - brandid, brandname, branddesc
Product table - productid, productname, productdesc, catid, brandid
(I also use ranges for price ranges etc)
I am wondering whether I should use SOLR to index the whole relational schema or whether just to index the product table alone and let my application work as it currently does.
If I just switch the product table to use SOLR are there any caveats to this?
e.g. in mysql I can do a fulltext search while joining the brand table. This will allow brands to also be searched upon. Is it possible to achieve the same thing just by switching the product table to SOLR? Are there any other caveats I should be looking out for.
I also would like to create a new table for "searches". This would allow me to use keywords in a mysql table in the following way:
Searches table - searchterm (e.g. lipstick), synonyms (e.g. lipstick, lips etc.)
ie. this would allow me to search upon multiple terms at the same time - a good time to use SOLR facets maybe instead of storing searches in mysql?, or should I just use mysql to store the searches and pull the products from SOLR?
Any help is gladly appreciated
NO NEED TO SWITCH
You don't want to "switch" -- just like using full-text indexing in MySQL (or using something like Sphinx), the full-text index is separate from the database tables.
What you want to do is figure out what you're searching for and index that in Solr -- it may well be just products. That's certainly an easy first step.
Basically you'll:
index the appropriate column(s) into Solr
use Solr for the searching
use the Solr results to point back to the records from the database
I'm more Ruby and Java than PHP, but you'll basically be talking to Solr for the full-text search and using that to find the records you want to display.
Probably a noob question, but I'll go for it nevertheless.
For sake of example, I have a Person table, a Tag table and a ContactMethod table. A Person will have multiple Tag records and multiple ContactMethod records associated with them.
I'd like to have a forgiving search which will search among several fields from each table. So I can find a person by their email (via ContactMethod), their name (via Person) or a tag assigned to them.
As a complete noob to FTS, two approaches come to mind:
Build some complex query which addresses each field individually
Build some sort of lookup table which concatenates the fields I want to index and just do a full-text query on that derived table.
(Feel free to edit for clarity; I'm not in it for the rep points.)
If your sql server supports it you can create an indexed view and full text search that; you can use containstable(*,'"chris"') to read all the columns.
If it doesn't support it as the fields are all coming from different tables I think for scalability; if you can easily populate the fields into a single row per record in a separate table I would full text search that rather than the individual records. You will end up with a less complex FTS catalog and your queries will not need to do 4 full text searches at a time. Running lots of separate FTS queries over different tables at the same time is a ticket to query performance issues in my experience. The downside with doing this is you lose the ability to search for Surname on its own; if that is something you need you might need to look at an alternative.
In our app we found that the single table was quicker (we can't rely on customers having enterprise sql at hand); so we populate the data with spaces into an FTS table through an update sp then our main contact lookup runs a search over the list. We have two separate searches to handle finding things with precision (i.e. names or phone numbers) or just for free text. The other nice thing about the table is it is relatively easy and low cost to add further columns to the lookup (we have been asked for social security number for example; to do it we just added the column to the update SP and we were away with little or no impact.
One possibility is to make a view which has these columns: PersonID, ContentType, Content. ContentType would be something like "Email", "PhoneNumber", etc... and Content would hold that. You'd be searching on the Content column, and you'd be able to see what the person's ID is. I'm not 100% sure how full text search works though, so I'm not sure if you could use that on a view.
The FTS can search multiple fields out-of-the-box. The CONTAINS predicate accepts a list of columns to search. Also CONTAINSTABLE.
So my goal here is to have a single search field in an application that will be able to search multiple tables and return results.
For example, two of these tables are "performers" and "venues" and there are the following performers: "John Andrews","Andrew Smith","John Doe" and the following venues: "St. Andrew's Church","City Hall". Is there a way to somehow return the first two performers and the first venue for a search of "Andrew"?
My first thought was to somehow get all the tables aggregated into a single table with three columns; "SearchableText","ResultType","ResultID". The first column would contain whatever I want searched (e.g. Performer name), the second would say what is being shown (e.g. Performer) and the third would say the item's ID (note: all my tables have auto-incrementing primary keys for ease). The question for this idea is it possible to somehow do this dynamically or do I have to add code to have a table that automatically fills whenever a new row is updated/added/deleted from the performers and venues table (perhaps via trigger?).
My application is written in MSAccess (I know, I know, but I have no choice) on top of a SQL Server backend. I'd prefer this happen through MSAccess so I don't have to have a "searchme" table sitting on my SQL Server but any good result is acceptable :)
I think you are looking for the "union" sql keyword
I'd use full text indexing in SQL server, have a single table with your searchable text, and forign keys in your main tables that link to the search table. This way you can order your results by relevance.
I think you have a schema problem. Querying a UNION is almost always evidence of that (though not in all cases).
The question to me is:
What are you returning as your result?
If you find a person, are you displaying a list of people?
Or if you find a venue, a list of venues?
Or a mix of both?
I would say that if you want to return a list of both, then you'd want something like this:
SELECT tblPerson.PersonID, tblPerson.LastName & ", " & tblPerson.FirstName, "Person"
FROM tblPerson
WHERE tblPerson.LastName LIKE "Andrew*"
OR tblPerson.FirstName LIKE "Andrew*"
UNION
SELECT tblVenue.Venue, tblVenue.Venue, "Venue"
FROM tblVenue
WHERE tblVenue.Venue LIKE "Andrew*"
ORDER BY Venue
This will give a list of the matches indicating which is a person and which a venue, and allow you to then select one of those and open a detail view (by checking the value in the third field).
What you definitely don't want to do is this:
SELECT tblPerson.PersonID, tblPerson.LastName & ", " & tblPerson.FirstName, "Person"
FROM tblPerson
UNION
SELECT tblVenue.Venue, tblVenue.Venue, "Venue"
FROM tblVenue
then saving that and trying to query it on the 2nd column. That will be extremely inefficient. You want your WHERE clause to be on fields that can be searched via the index, and that means each subquery of your UNION needs to have an appropriate WHERE clause.