How do you search Google App Engine logs in the new Cloud Console using regular expressions?
This blog post suggests you just need to type regex:my.*query to search, but that does not seem to work in the logging console. When I do that, it auto-corrects to the following query text:regex:my.*query.
Although, I am late here but you can do it in stackdriver.
= # equal
!= # not equal
> < >= <= # numeric ordering
: # "has" matches any substring in the log entry field
In case, you want to find all GET response with 500 in textPayload , you need to add in filters:
textPayload:"500"
To search all zone with central1 in it:
resource.labels.zone:"-central1-"
That's it. You can refer this link for more advance filters
Nowadays Google Cloud Operations Logging has support for regular expressions. This feature was published on 2020-09-17, see https://cloud.google.com/blog/products/management-tools/cloud-logging-gets-regular-expression-support. You regex works in this format:
textPayload=~"my.*query"
You can query regular expression matches with operator =~ and non-matches with operator !~:
=~ # regular expression search for a pattern
!~ # regular expression search not for a pattern
More info on syntax and examples can be found in the Google Cloud Operations Suite Logging query language reference: https://cloud.google.com/logging/docs/view/logging-query-language#regular-expressions
The Stackdriver Logging product does not currently support regular expressions. It was originally supported a while back (as you saw in the blog post), but we found that it was rarely used and that many of those uses were for simple patterns that had simpler solutions without the performance and other penalties of regexes.
In basic filter mode (the default), text searches automatically are case-insensitive and match substrings of field values, and you can use ".." to represent numerical ranges. In advanced filter mode, the "has" operator accomplishes the same thing through using a : instead of an = in your filter expression, e.g. path.to.field: "value". (See also: Write effective advanced filters)
If these operators don't accomplish your goal, consider filing feedback through the speech bubble button in the top right of the Cloud Console providing details of your use case and what you're ultimately trying to accomplish, and we'll incorporate that feedback as we plan the future direction of the product.
Related
I'd like to perform a partial text / phrase search against a Datastore record field using Ruby.
I've figured out how to do it with a conditional constraint using >= <"\ufffd" condition, but that only works from the beginning of the field.
This works; querying for "Ener" returns "Energizer AA Batteries" but querying for "AA" does not return the same.
In the docs for the Python Google Client's Search API, it documents the ability to manually create indexes which allow for both atomic and partial word searches.
https://cloud.google.com/appengine/docs/standard/python/search/ says:
Tokenizing string fields When an HTML or text field is indexed, its
contents are tokenized. The string is split into tokens wherever
whitespace or special characters (punctuation marks, hash sign, etc.)
appear. The index will include an entry for each token. This enables
you to search for keywords and phrases comprising only part of a
field's value. For instance, a search for "dark" will match a document
with a text field containing the string "it was a dark and stormy
night", and a search for "time" will match a document with a text
field containing the string "this is a real-time system".
In the docs for Ruby and PHP, I cannot find such an API reference to enable me to do the same thing. Is it possible to perform this type of query in Ruby / PHP with Cloud Datastore?
If so, can you point me to the docs, and if not, is there a workaround, for example, create indexes with the Python Search API, and then configure the PHP/Ruby client to execute it's queries against those indexes?
We are using the .net library for azure search, I have successfully built the index and stored data in the index. One of our fields is called Tags which is a collection of strings and it is marked as searchable. So we put values in this field such as C# .NET.
The problem is when searching the search service will not hit on C#, it will on C, nor will it hit on .NET but it will on NET. I can see thru fiddler that the search term is encoding the # and also the ., but it doesn't seem like it's getting decoded on the azure side.
The behavior you're seeing is the result tokenization performed by the standard analyzer used by Azure Search. By default it breaks on many punctuation characters like # and . (you can get all the details of text analysis in Azure Search here).
We're looking into adding support for custom analyzers that would let you exclude characters such as # and . from word-breaking, but this is still in the planning stages. In the meantime, as a workaround we suggest encoding these characters in your application before indexing and querying (e.g. -- C# -> CSharp, .NET -> dotNET).
Thanks Bruce, for now I have just created a function in our search implementation that removes punctuation from the search term provided by the end user. This way I don't have to go thru and update all search index/records.
private string SanitizeValue(string value)
{
return Regex.Replace(value, #"[^a-zA-Z0-9\s]", "");
}
You could try using Regex search, like searching for this string: /.*c\#.*/. Also make sure you set SearchParameters.QueryType = QueryType.Full.
Let's say that one of my fields in the index contains the word entrepreneurial. When I search for the word entrepreneur I don't get that document. But entrepreneur* does.
Is there a mode/parameter in which queries search for document that have words that contain a word token in search text?
Another example would be finding a doc that has Matthew when you're looking for Matt.
Thanks
We don't currently have a mode where all input terms are treated as prefixes. You have a few options depending of what exactly are you looking for:
Set the target searchable field to a language specific analyzer. This is the nicest option from the linguistics perspective. When you do this, if appropriate for the language we'll do stemming which helps with things such as "run" versus "running". It won't help with your specific sample of "entrepreneurial" but generally speaking this helps significantly with recall.
Split search input before sending it to search and add "" to all. Depending on your target language this is relatively easy (i.e. if there are spaces) or very hard. Note that prefixes don't mix well with stemming unless take them into account and search both (e.g. something like search=aa bb -> (aa | aa) (bb | bb*))
Lean on suggestions. This is more of a different angle that may or may not match your scenario. Search suggestions are good at partial/prefix matching and they'll help users land on the right terms. You can read more about this here.
perhaps this page might be of interest..?
https://msdn.microsoft.com/en-us/library/azure/dn798927.aspx
search=[string]
Optional. The text to search for. All searchable fields are searched by
default unless searchFields is specified. When searching searchable fields, the search text itself is tokenized, so multiple terms can be separated by white space (e.g.: search=hello world). To match any term, use * (this can be useful for boolean filter queries). Omitting this parameter has the same effect as setting it to *. See Simple query syntax in Azure Search for specifics on the search syntax.
I have created Document, added it to Index and used the GAE Search API to search for a text successfully. Please find the sample code below.
search.Document(
fields=[search.TextField(name='id', value=id),
search.TextField(name='search', value=searchT)])
options = search.QueryOptions(returned_fields=['id'])
results = search.Index(name=_D_INDEX_NAME).search(search.Query(searchTxt, options=options))
Now I am unable to understand to to achieve these mentioned below: Some sample code would be really appreciated.
To search for plural variants of an exact query, use the ~ operator:
~"car" # searches for "car" and "cars"
To build queries that reference specific fields, use both field and value in your query, separated by a colon:
field:value
field:"value as a string"
When you add a document, you specify its schema by defining the fields of the document. In your case id and search.
To search for a term that only appears in a specific field you use the notation field:term
search.Index(name=_D_INDEX_NAME).search('search:programming')
For searching plural variants of a term you use the operator ~
search.Index(name=_D_INDEX_NAME).search('~car')
Note however that this won't work in the dev_appserver.
Is there any way I can user a NOT or other negation operator before a text search keyword for example,
NOT program
When I do such a search there are 0 records returned.
Please let me know some way to achieve this option.
In Solr you can use the '-' minus sign as a NOT operator, so you would change your query to be
*:* - program
If you are using SolrNet, since that is how your question is tagged, you can do the following
solr.Query(new SolrQuery("*:*") && !new SolrQuery("program"));
Please see Querying in SolrNet for more details.
Updated: Per comment from Mauricio Scheffer
There aren't any search operators that perform the search parameters you are describing. My advice is to use the Google advanced search features. You can make your search much more specific in a number of ways, its really advanced.