I was trying to optimize a BigQuery query that I use to search through my AppEngine application logs (exported to BigQuery automatically through Google Cloud Logging) but I got an error that I don't understand.
SELECT
protoPayload.requestId,
protoPayload.line.logMessage
FROM (
SELECT
protoPayload.requestId AS matchingRequestId
FROM
TABLE_DATE_RANGE(MyProject_Logs.appengine_googleapis_com_request_log_, DATE_ADD(CURRENT_TIMESTAMP(), -1, 'HOUR'), CURRENT_TIMESTAMP())
WHERE
protoPayload.resource CONTAINS '/url'
AND protoPayload.line.logMessage CONTAINS 'criteria'
LIMIT 50)
WHERE
protoPayload.requestId = matchingRequestId
results in
Query Failed
Error: Field 'protoPayload.requestId' not found.
Job ID: myProject:job_DZpCc0u52LBFh8DFL0nDCsizo8o
This error does not make sense to me because when I try to execute just the sub-query that also use the protoPayload.requestId field, it works fine.
Just as a side note, this SO answers better what I am trying to achieve but I am still curious what cause the error in my query.
This error makes sense to me for that particular example in the question:
outside of subselect - protoPayload.requestId is not visible anymore - it is a matchingRequestId based on alias in protoPayload.requestId AS matchingRequestId
Please note, after you fix outside (two) references to protoPayload.requestId, next error will be about protoPayload.line.logMessage
It is also not visible to outer select because a) it is not part of subselect and b) reference to table from subselect
it looks like you oversimplified your example as even if/after above fixed - it still makes no much sense to me - especisaaly because of WHERE matchingRequestId = matchingRequestId
Related
Basically I have a client request to implement:
Need to show data from the following fields : PaymMode, BankChequeNum, LedgerDimensionName, JournalNum from ledgerJournalTrans table on the LedgerTransVoucher form but have been unable to do so have tried nearly all of the possible queries that I could think of but none of them are working the way I expect them to: either the query is doing a cartesian product and duplicating the records or it is displaying no data in those fields.
Below is the query that I have recently tried:
public display MH_AccountTitle displayBeneficiaryName(GeneralJournalAccountEntry _accountEntry)
{
select SubledgerVoucher, AccountingDate from journalEntry
where journalEntry.RecId == _accountEntry.GeneralJournalEntry
join Voucher, MH_AccountTitle, RecId, AmountCurDebit, AmountCurCredit, TransDate from LedgerTrans
where LedgerTrans.Voucher == journalEntry.SubledgerVoucher
&& LedgerTrans.TransDate == journalEntry.AccountingDate
&& LedgerTrans.PaymReference == _accountEntry.PaymentReference
&& (abs(_accountEntry.TransactionCurrencyAmount) == LedgerTrans.AmountCurDebit
|| abs(_accountEntry.TransactionCurrencyAmount) == LedgerTrans.AmountCurCredit);
return ledgerTrans.MH_AccountTitle;
}
I know this query is logically incorrect because joins can't be applied on the basis of date and amount but this was suggested by a senior of mine after all else failed, and it did work, records were returned correctly but it failed where there were multiple transactions with same TransactionCurrencyAmount,TransDate and voucher
join with PaymentReference also failed where the method of payment was not Check and hence there was no BankChequeNum/Payment reference resulting in the same problem
Anyone who has any idea of what could be work around for this?
Note:
work has been done on a custom form of LedgerTransVoucher
display method approach was used because simply applying joins on the form's data source didn't work
Also code has been written on form's datasource GeneralJournalAccountEntry
What I usually do to debug these kinds of issues is get the query string that X++ creates when using inline SQL. Often there is something that X++ translates into SQL that is not expected.
Secondly, I always write my queries in SSMS first to avoid having unwanted results because of the X++ translation to SQL query.
You can get the query string by either making a query object and using the .ToString() method or you can take a trace and use Traceparser to view the query that was sent to the SQL Server.
I am thinking the abs() functions will be the issue here.
I have a table in Snowflake where one of the fields, called 'value' is sometimes plain text sometimes JSON, this field is stored as string in Snowflake
I created this view to get only the rows where there is a Json format
CREATE OR REPLACE VIEW tmp_events AS
SELECT PARSE_JSON(value) as json_data,
id
FROM SessionEvent
WHERE session_event_type_id=7;
Then I flatten the rows to create a new field
CREATE OR REPLACE VIEW tmp_events_step2 AS
SELECT id,
json_data:meta:selected::string AS choice
from tmp_events ,
LATERAL FLATTEN(input => tmp_events.json_data)
WHERE choice IS NOT NULL
Everything runs fine until now, I can preview data from these two views, no error and I get the results I was expecting.
The error comes when I try to get distinct values from choice
SELECT DISTINCT choice from tmp_events_step2;
Error parsing JSON: unknown keyword "brain", pos 6
This name Brain seems to come from my initial table without the WHERE statement.
If I run the query without DISTINCT there is no error.
Weird thing I noticed while trying to debug: when I put a limit in tmp_events_step2, the code works fine again, even though I put a limit that's bigger than the number of rows in the table
CREATE OR REPLACE VIEW tmp_events_step2 AS
SELECT id,
json_data:meta:selected::string AS choice
from tmp_events ,
LATERAL FLATTEN(input => tmp_events.json_data)
WHERE choice IS NOT NULL
LIMIT 10000000;
SELECT DISTINCT choice from tmp_events_step2;
What's the catch? Why does it work only with the limit?
The very simple answer to this is the built-in function TRY_PARSE_JSON()
Er, not. You seem to have problems with the Query optimizer that may do incorrect predicate pushdowns. One way to prevent the optimizer from doing this is to use the secure view option:
CREATE SECURE VIEW tmp_events_step2 ...
and file a support ticket...
We reported this error two years ago and they said they where not going to fix, because by hoisting the JSON access prior to running the filters in the WHERE clause that makes the cast valid/safe, impacted performance.
create table variant_cast_bug(num number, var variant);
insert into variant_cast_bug
select column1 as num, parse_json(column2) as var
from values (1, '{"id": 1}'),
(1, '{"id": 2}'),
(2, '{"id": "text"}')
v;
select * from variant_cast_bug;
select var:id from variant_cast_bug;
select var:id from variant_cast_bug where num = 1;
select var:id::number from variant_cast_bug where num = 1; -- <- broken
select TRY_TO_NUMBER(var:id) from variant_cast_bug where num = 1; -- <- works
Sometimes you can nest the select and it will work, and then you can add another SELECT layer around it, and do some aggregation and the cost explodes again.
The only two safe solutions are SERCURE VIEW as Hans mentions, but that is a performance nightmare.
Or to understand this problem and use TRY_TO_NUMBER or it's friends.
At the time this was made bad worse because JSON boolean values where not valid values to pass to TRY_TO_BOOLEAN..
One of the times we got burnt by this was after a snowflake release when code that had been running for a year, started getting this error, because it was complex enough the hoisting did not impact, and then after release it did. This is where Snowflake are rather responsive, and then rolled the release back, and we put TRY_TO on a chunk of already working SQL just to play it safe.
Please submit a support case for this issue.
I am trying to count all the results that match multiple Where conditions in Salesforce. All these Where conditions exist under the same object that I am selecting from. It seems like it should be a simply query but my SQL and SOQL experience is limited.
Here's my code right now:
SELECT count() FROM Account
WHERE Success__c='yes'
AND Active__c='true'
AND Days__c>'30'
AND Days__c<'37'
It'd be useful to see the actual error message, but at a guess, you have quotes around things that shouldn't have them, e.g. you want
SELECT count() FROM Account
WHERE Success__c = 'yes'
AND Active__c = true
AND Days__c > 30
AND Days__c < 37
Also there are tools like SoqlX, the Force.com IDE and Workbench that'll let you run queries, so if Geckoboard is hiding the actual error message, you can work through getting a good query in one of these tools first.
I have following query:
Article.joins(:themes => [:users]).where(["articles.user_id != ?", current_user.id]).order("Random()").limit(15).uniq
and gives me the error
PG::Error: ERROR: for SELECT DISTINCT, ORDER BY expressions must appear in select list
LINE 1: ...s"."user_id" WHERE (articles.user_id != 1) ORDER BY Random() L...
When I update the original query to
Article.joins(:themes => [:users]).where(["articles.user_id != ?", current_user.id]).order("Random()").limit(15)#.uniq
so the error is gone... In MySQL .uniq works, in PostgreSQL not. Exist any alternative?
As the error states for SELECT DISTINCT, ORDER BY expressions must appear in select list.
Therefore, you must explicitly select for the clause you are ordering by.
Here is an example, it is similar to your case but generalize a bit.
Article.select('articles.*, RANDOM()')
.joins(:users)
.where(:column => 'whatever')
.order('Random()')
.uniq
.limit(15)
So, explicitly include your ORDER BY clause (in this case RANDOM()) using .select(). As shown above, in order for your query to return the Article attributes, you must explicitly select them also.
I hope this helps; good luck
Just to enrich the thread with more examples, in case you have nested relations in the query, you can try with the following statement.
Person.find(params[:id]).cars.select('cars.*, lower(cars.name)').order("lower(cars.name) ASC")
In the given example, you're asking all the cars for a given person, ordered by model name (Audi, Ferrari, Porsche)
I don't think this is a better way, but may help to address this kind of situation thinking in objects and collections, instead of a relational (Database) way.
Thanks!
I assume that the .uniq method is translated to a DISTINCT clause on the SQL. PostgreSQL is picky (pickier than MySQL) -- all fields in the select list when using DISTINCT must be present in the ORDER_BY (and GROUP_BY) clauses.
It's a little unclear what you are attempting to do (a random ordering?). In addition to posting the full SQL sent, if you could explain your objective, that might be helpful in finding an alternative.
I just upgraded my 100% working and tested application from 3.1.1 to 3.2.7 and now have this same PG::Error.
I am using Cancan...
#users = User.accessible_by(current_ability).order('lname asc').uniq
Removing the .uniq solves the problem and it was not necessary anyway for this simple query.
Still looking through the change notes between 3.1.1 and 3.2.7 to see what caused this to break.
I am currently working on a query that is registered for Query Notifications. In accordance w/ the rules of Notification Serivces, I can only use Deterministic functions in my queries set up for subscription. However, GetDate() (and almost any other means that I can think of) are non-deterministic. Whenever I pull my data, I would like to be able to limit the result set to only relevant records, which is determined by the current day.
Does anyone know of a work around that I could use that would allow me to use the current date to filter my results but not invalidate the query for query notifications?
Example Code:
SELECT fcDate as RecordDate, fcYear as FiscalYear, fcPeriod as FiscalPeriod, fcFiscalWeek as FiscalWeek, fcIsPeriodEndDate as IsPeriodEnd, fcPeriodWeek as WeekOfPeriod
FROM dbo.bFiscalCalendar
WHERE fcDate >= GetDate() -- This line invalidates the query for notification...
Other thoughts:
We have an application controls table in our database that we use to store application level settings. I had thought to write a small script that keeps a record up to date w/ teh current smalldatetime. However, my join to this table is failing for notificaiton as well and I am not sure why. I surmise that it has something to do w/ me specifitying a text type (the column name), which is frustrating.
Example Code 2:
SELECT fcDate as RecordDate, fcYear as FiscalYear, fcPeriod as FiscalPeriod, fcFiscalWeek as FiscalWeek, fcIsPeriodEndDate as IsPeriodEnd, fcPeriodWeek as WeekOfPeriod
FROM dbo.bFiscalCalendar
INNER JOIN dbo.xApplicationControls ON fcDate >= acValue AND acName = N'Cache_CurrentDate'
Does anyone have any suggestions?
EDIT: Here is a link on MSDN that gives the rules for Notification Services
As it turns out, I figured out the solution. Basically, I was invalidating my query attempts because I was casting a value as a DateTime which marks it as Non-Deterministic. Even though you don't specifically call out a cast but do something akin to:
RecordDate = 'date_string_value'
You still end up w/ a Date Cast. Hopefully this will help out someone else who hits this issue.
This link helped me quite a bit.
http://msdn.microsoft.com/en-us/library/ms178091.aspx
A good way to bypass this is simply to create a view that just says "SELECT GetDate() AS Now", then use the view in your query.
EDIT : I see nothing about not using user-defined functions (which is what I've used the 'view today' bit in). So can you use a UDF in the query that points at the view?