I am trying to run an asynchronous query. When I submit it to Bigquery it answers me that the table does not exist (status is "DONE"). What is wrong?
jobData = {'configuration': {'query': {'query': query, 'createDisposition': 'CREATE_NEVER'}}}
logger.info('jobdata: %s', jobData)
insertResponse = self.service.jobs().insert(projectId = PROJECT_ID, body = jobData).execute()
if 'errorResult' in str(insertResponse):
logger.error('query insert failed: %s', insertResponse)
This worked a few weeks back but no more. The error I get seems to indicate that the table cannot be found (or is it some kind of temporary table and all is well?)
jobdata: {'configuration': {'query': {'createDisposition': 'CREATE_NEVER', 'query': 'SELECT TIMESTAMP_TO_MSEC(timestamp) AS timestamp, location, branch, platform, description, original, metrics.group, metrics.id, metrics.name, metrics.value, DATE(timestamp) AS date FROM [mm.autotest] WHERE DATEDIFF(CURRENT_TIMESTAMP(), timestamp) < 14'}}}
query insert failed: {u'status': {u'state': u'DONE', u'errors': [{u'reason': u'notFound', u'message': u'Not found: Table ava-backend:_24bc4b39cef74439ece7e6f9a41399c21fccacd8.anonev_c7HQskNXMdaxCBDXdnAtOzlSJ6M'}], u'errorResult': {u'reason': u'notFound', u'message': u'Not found: Table ava-backend:_24bc4b39cef74439ece7e6f9a41399c21fccacd8.anonev_c7HQskNXMdaxCBDXdnAtOzlSJ6M'}}, u'kind': u'bigquery#job', u'statistics': {u'endTime': u'1390899670652', u'creationTime': u'1390899670344', u'startTime': u'1390899670652'}, u'jobReference': {u'projectId': u'ava-backend', u'jobId': u'job_EowCD-kiOirnlyRAQjFJNFoRFUY'}, u'etag': u'"11dTZYgUnUwbk8emYQU9mVRTTLs/0mRkexpB4dgaqONWphRujyRJhqM"', u'configuration': {u'query': {u'createDisposition': u'CREATE_NEVER', u'query': u'SELECT TIMESTAMP_TO_MSEC(timestamp) AS timestamp, location, branch, platform, description, original, metrics.group, metrics.id, metrics.name, metrics.value, DATE(timestamp) AS date FROM [mm.autotest] WHERE DATEDIFF(CURRENT_TIMESTAMP(), timestamp) < 14'}}, u'id': u'ava-backend:job_EowCD-kiOirnlyRAQjFJNFoRFUY', u'selfLink': u'https://www.googleapis.com/bigquery/v2/projects/ava-backend/jobs/job_EowCD-kiOirnlyRAQjFJNFoRFUY'}
The query works fine in Bigquery web interface and if run synchronously with 'query' instead of 'insert'
Could it possibly be an authentication issue? When I run the query with "query" it works but I get a warning message:
Checking for id_token.
id_token verification failed: Can't parse header: '\xc9\xad\xbd'
Checking for oauth token.
Client ID is not allowed: 173893847593-pdkumgr5bhobh3tv6qhqqfvel2tvnns0.apps.googleusercontent.com
URL being requested: https://www.googleapis.com/bigquery/v2/projects/ava-backend/queries?alt=json
The issue is with 'createDisposition': 'CREATE_NEVER'. This tells BigQuery that you never want to create a table for the query results. So it fails when the query result table doesn't exist. If you remove that create disposition, it should work.
Related
I ran a query using google big query but I get an error as a result. I don't know what could be wrong.
SELECT
usertype,
CONCAT(start_station_name, " to ", end_station_name) AS route,
COUNT(*) as num_trips,
ROUND(AVG(CAST(tripduration AS INT64)/60),2) AS duration
FROM
`bigquery-public-data.new_york_citibike.citibike_stations`
GROUP BY
start_station_name, end_station_name, usertype
ORDER BY
num_trips DESC
LIMIT 10
You must be getting this error: Unrecognized name: usertype . As the error suggests bigquery-public-data.new_york_citibike.citibike_stations dataset doesn't have any column usertype.
As #samuel mentioned, usertype column is present in new_york_citibike.citibike_trips. Do cross check your dataset before querying.
I am running executing raw sql to delete some records that I added for the test. If I run the same query in management studio it works fine but when I run that query EF Core 2.0 it throws below error
System.Data.SqlClient.SqlException: 'Conversion failed when converting the nvarchar value '1,2' to data type int.'
Code
var idList = await Context.User.ToListAsync();
var ids = string.Join(",",idList.Select(x=>x.Id));
await _context.Database.ExecuteSqlCommandAsync($"Delete from User where Id in ({ids}) and RoleId = {contact.RoleId}");
Query executing
Delete from sale.WatchList where OfferId in (1,2) and UserId = 9
Could anybody please advise on what wrong with the above code.
Thanks
EF Core will transform interpolated strings into queries with parameters to create reusable queries and protect against SQL Injection vulnerabilities. See: Raw SQL Queries - EF Core - Passing Parameters
So
$"Delete from User where Id in ({ids}) and RoleId = {contact.RoleId}"
is transformed into
Delete from User where Id in (#ids) and RoleId = #RoleId
With SqlParameters bound.
If that's not what you want, just build the SQL Query on a previous line.
This will not work. You have to write dynamic query. Please try like below one
var idList = await _dataContext.User.ToListAsync();
var ids = string.Join(",", idList.Select(x => x.Id));
await _dataContext.Database.ExecuteSqlCommandAsync($"execute('Delete from User where Id in ({ids}) and RoleId = {contact.RoleId}')");
although accepted answer does work, it creates lot of warnings so for now I am using what #Abu Zafor suggested with small change/fix
await _dataContext.Database.ExecuteSqlCommandAsync($"execute('Delete from User where Id in ({ids}) and RoleId = {contact.RoleId}')",ids,contact.RoleId);
Maybe it is a case of me looking at this for too long. But I have this Oracle update query I am trying to run, I have verified the query works with hardcoded values on SQL Developer, howver when I run it from my flow Mule it fails.can anybody tell me what I am doing wrong?
Here is the query:
<db:update config-ref="DBConf" doc:name="abcd">
<db:dynamic-query><![CDATA[UPDATE myTable
SET TYPE= 'Entry',
ENTERED_DATE=SYSDATE,
ENTRY_BY= 2345,
ENTRY_DATE=TO_DATE('#[flowVars.entryDate]','YYYY-MM-DD')
WHERE ID = 'abcd1234']]>
</db:dynamic-query>
</db:update>
the flowVars.entryDate value is '2017-05-10'
This throws the following Error:
Message : ORA-01841: (full) year must be between -4713 and +9999, and not be 0
(java.sql.SQLDataException). Message payload is of type: Integer
Now the same query works like I said in SQL Developer but not in Mule, Can anybody provide any input
You can find the same problem answer in the following link:
Oracle: year must be between -4713 and +9999, and not be 0
Try this once. TO_DATE('2012-05-12','yyyy-mm-dd').
Delete the quotes for the #[flowvar.entrydate]
I create table with this statement:
CREATE TABLE event(
date Date,
src UInt8,
channel UInt8,
deviceTypeId UInt8,
projectId UInt64,
shows UInt32,
clicks UInt32,
spent Float64
) ENGINE = MergeTree(date, (date, src, channel, projectId), 8192);
Raw data looks like:
{ "date":"2016-03-07T10:00:00+0300","src":2,"channel":18,"deviceTypeId ":101, "projectId":2363610,"shows":1232,"clicks":7,"spent":34.72,"location":"Unknown", ...}
...
Files with data loaded with the following command:
cat *.data|sed 's/T[0-9][0-9]:[0-9][0-9]:[0-9][0-9]+0300//'| clickhouse-client --query="INSERT INTO event FORMAT JSONEachRow"
clickhouse-client throw exception:
Code: 117. DB::Exception: Unknown field found while parsing JSONEachRow format: location: (at row 1)
Is it possible to skip fields from JSON object that not presented in table description?
The latest ClickHouse release (v1.1.54023) supports input_format_skip_unknown_fields user option which eneables skipping of unknown fields for JSONEachRow and TSKV formats.
Try
clickhouse-client -n --query="SET input_format_skip_unknown_fields=1; INSERT INTO event FORMAT JSONEachRow;"
See more details in documentation.
Currently, it is not possible to skip unknown fields.
You may create temporary table with additional field, INSERT data into it, and then do INSERT SELECT into final table. Temporary table may have Log engine and INSERT into that "staging" table will work faster than into final MergeTree table.
It is relatively easy to add possibility to skip unknown fields into code (something like setting 'format_skip_unknown_fields').
I am using a camel jdbc component to insert a record into an Oracle table. the insert uses a sequence to populate a primary key ID column.
INSERT INTO my_table (id, data) VALUES (my_seq.nextval, 'some data')
The relevant part of the route looks like below:
from("some end point here")
.process(preInsertProcessor)
.to("jdbc:myDataSource")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
LOGGER.info("Extracting database generated id");
// This list is null
List<Integer> ids = exchange.getIn().getHeader(
JdbcConstants.JDBC_GENERATED_KEYS_DATA, List.class);
});
Inside the preInsertProcessir I set the message body to be my insert statement and also set some two header values to instruct camel I want the generated ID back:
message.setBody("INSERT INTO my_table (id, data) VALUES (my_seq.nextval, ?:data)");
message.setHeader("data", "some data");
message.setHeader(JDBC_RETRIEVE_GENERATED_KEYS, true);
message.setHeader(JDBC_GENERATED_COLUMNS, new String[]{"ID"});
End if I look at the logs I can see:
[DEBUG] org.apache.camel.component.bean.MethodInfo - Setting bean invocation result on the OUT message: [Message: INSERT INTO my_table(id, data)VALUES (my_seq.nextval, :?data]
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction begin (0x1de4bee0) redelivered(false) for (MessageId: ID-MELW1TYGC2S-62650-1438583607644-0-8 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-9))
[INFO ] au.com.nab.cls.router.non.repudiation.GeneratedIdExtractor - Extracting database generated id
[DEBUG] org.apache.camel.processor.MulticastProcessor - Done sequential processing 1 exchanges
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction commit (0x1de4bee0) redelivered(false) for (MessageId: ID:414d5120445041594855423120202020027844552045b302 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-7))
If I get it well from the look of the logs the insert would be executed and my final processor should be able to get the generated ID. In reality what happens is that no record gets inserted and no ID is present in the header of the message. Without the final processor everything works fine.
Obviously I am doing something wrong here but I cannot figure out what. I am aware I could use a message en-richer to get the ID before the insert but I would prefer to avoid an extra database trip.
Thank you in advance for your inputs.
UPDATE
I put a break point in org.apache.camel.component.jdbc.JdbcProducer and found out the reason for not having the INSERT executed and consequently not getting the ID back.
// JdbcProducer code; creating a prepared statement part
if (shouldRetrieveGeneratedKeys) {
...
} else if (expectedGeneratedColumns instanceof String[]) {
// Execution gets herestatement
ps = conn.prepareStatement(preparedQuery, (String[]) expectedGeneratedColumns);
...
}
// Expected count returned here is 2
int expectedCount = ps.getParameterMetaData().getParameterCount();
if (expectedCount > 0) {
...
// And here I get the crash:
// java.sql.SQLException: Number of parameters mismatch. Expected: 2, was:1
getEndpoint().getPrepareStatementStrategy().populateStatement(ps, it, expectedCount);
}
This is where my research stopped as digging too much in the various three parties code is not really easy. I suspect one of the following two options are the cause:
I am still not doing it the right way
A camel bug which does not work as expected when header contains both named parameters and retrieve generated keys settings
Please advise about any fix or work around.
Thanks again
I also ran into this. My workaround:
Use camel-sql instead of camel-jdbc. Add parametersCount option to the endpoint URL (in adddition to setHeader(SqlConstants.SQL_RETRIEVE_GENERATED_KEYS, constant(true)) and setHeader(SqlConstants.SQL_GENERATED_COLUMNS, constant(new String[] {"ID_COLUMN_NAME"})).
Update: Works with 11.2.0.4 jdbc driver (does not work with 12.2.0.1 jdbc driver).