from("{{my.app.source}}")
.unmarshal()
.bindy(BindyType.Csv, EmployeeCsvRecord.class)
.split(body())
.streaming()
.bean("employeeService", "getMap")
.aggregate(constant(true), new EmployeeAggregationStrategy())
.completionSize(500)
.log("data ready to insert into database")
.to("{{sql.insertEmployee}}")
.log("data inserted into database");
"sql:insert = into employee (employeeName, employeeAge, employeeGender, employeeDepartment, employeeSalary) values (:#employeeName, :#employeeAge, :#employeeGender, :#employeeDepartment, :#employeeSalary);batch=true"
Cannot find key [employeeName] in message body or headers to use when setting named parameter in query
I tried with batch false it is working fine.
I have to do it only with sql component otherwise there are multiple ways available.
Related
I have a form that is linked to an ODBC (MS SQLServer), that is displaying a result of a VIEW, when I try to delete a record, I'm invoking a function at VBA level
Form_BeforeDelConfirm(Cancel As Integer, Response As Integer){
If (IsNull(Text104.value) = False) Then
Dim deleteLabel As DAO.QueryDef
Set deleteLabel = CurrentDb.CreateQueryDef("")
deleteLabel.Connect = CurrentDb.TableDefs("KontrollWerte").Connect
If (InStr(QuerySave, "KontrollWerteVIEW") <> 0) Then
deleteLabel.sql = "delete from KontrollWerte_Label where Kontrolle_Label_ID = " & Text104.value
End If
Close
End If
}
Error shown:
[Microsoft][SQL Server Native Client 11.0][SQL Server] View or function 'dbo.KontrollwerteVIEW' is not updatable because the modification affects multiple base tables. (#4450)
This error is okay, as the view is a select with multiple tables.
It seems the function is not called at all, and the "default" delete function of the MS Access is being called, there is a way to say to MS Access don't do the default delete and instead execute my sql statement inside of the Form_BeforeDelConfirm function?
Thanks!
I tried to change when the call of function is called, but no luck.
You have to ensure that the view allows updates.
When you link a table, and it is a view?
The during the creating of the table link, then you will see this prompt:
First, we select the "view" when linking that "view/table"
and note I also checked the Save password option.
Then next we get/see this:
DO NOT skip the above prompt. If you don't select the PK column for that view, then it will be read only.
FYI:
The above prompt to save password, and the above prompt to select the PK when linking to that view?
It DOES NOT appear on re-link, ONLY when adding a new link!!!!
So, that's why I stated to delete the link to sql server (not just re-link and not just re-fresh table link, but Adding for first time are the key instructions here).
So, yes, views are and can be up-datable, but you MUST answer the above prompt during the linking process in Access, or the result is a read-only table.
I have created a Python function which creates multiple query statements.
Once it creates the SQL statement, it executes it (one at a time).
Is there anyway to way to bulk run all the statements at once (assuming I was able to create all the SQL statements and wanted to execute them once all the statements were generated)? I know there is an execute_stream in the Python Connector, but I think this requires a file to be created first. It also appears to me that it runs a single query statement at a time."
Since this question is missing an example of the file, here is a file content that I have provided as extra that we can work from.
//connection test file for python multiple queries
import snowflake.connector
conn = snowflake.connector.connect(
user = 'xxx',
password = '',
account = 'xxx',
warehouse= 'xxx',
database= 'TEST_xxx'
session_parameters = {
'QUERY_TAG: 'Rachel_test',
}
}
while(conn== true){
print(conn.sfqid)import snowflake.connector
try:
conn.cursor().execute("CREATE WAREHOUSE IF NOT EXISTS tiny_warehouse_mg")
conn.cursor().execute("CREATE DATABASE IF NOT EXISTS testdb_mg")
conn.cursor().execute("USE DATABASE testdb_mg")
conn.cursor().execute(
"CREATE OR REPLACE TABLE "
"test_table(col1 integer, col2 string)")
conn.cursor().execute(
"INSERT INTO test_table(col1, col2) VALUES " +
" (123, 'test string1'), " +
" (456, 'test string2')")
break
except Exception as e:
conn.rollback()
raise e
}
conn.close()
The reference to this question refers to a method that can be done with the file call, the example in documentation is as follows:
from codecs import open
with open(sqlfile, 'r', encoding='utf-8') as f:
for cur in con.execute_stream(f):
for ret in cur:
print(ret)
Reference to guide I used
Now when I ran these, they were not perfect, but in practice I was able to execute multiple sql statements in one connection, but not many at once. Each statement had their own query id. Is it possible to have a .sql file associated with one query id?
Is it possible to have a .sql file associated with one query id?
You can achieve that effect with the QUERY_TAG session parameter. Set the QUERY_TAG to the name of your .SQL file before executing it's queries. Access the .SQL file QUERY_IDs later using the QUERY_TAG field in QUERY_HISTORY().
I believe though you generated the .sql while executing in snowflake each statement will have unique query id.
If you want to run one sql independent to other you may try with multiprocessing/multi threading concept in python.
The Python and Node.Js libraries do not allow multiple statement executions.
I'm not sure about Python but for Node.JS there is this library that extends the original one and add a method call "ExecutionAll" to it:
snowflake-multisql
You just need to wrap multiple statements with the BEGIN and END.
BEGIN
<statement_1>;
<statement_2>;
END;
With these operators, I was able to execute multiple statement in nodejs
I'm running big dependency scan on legacy db and see that some objects have obsolete ref links, if you run this code in SSMS for View that points to not existing table like in my case, you will get your output on Results tab AND error info in Messages . Like in my case below.
I tried to check all env things I know and output of this stored procedure, but didn't see any indication.
How I can capture this event as I'm running this in looped dynamic SQL script and capture output in my table for further processing?
Updated:
it just text in Message box ,on error, you still have output on
Results tab
this is sp, it loop thru object list I took from sys.object and run this string as my sample to get all dependencies, load all into table. This call to
sql_reference_entities is the only way to get inter database
dependency on column level. So I need stick to this 100$>
--
Select *
From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')
--
----update------
This behavior was fixed in SQL Server 2014 SP3 and SQL Server 2016 SP2:
Starting from Microsoft SQL Server 2012, errors raised by
sys.dm_sql_referenced_entities (such as when an object has undergone a
schema change) cannot be caught in a TRY...CATCH Transact-SQL block.
While this behavior is expected in SQL Server 2012 and above, this
improvement introduces a new column that's called is_incomplete to the
Dynamic Management View (DMV).
KB4038418 - Update adds a new column to DMV sys.dm_sql_referenced_entities in SQL Server 2014 and 2016
----update-------
The tldr is that you can't capture these on the server side, and must use a client program in C#, PowerShell or some other client that can process info messages.
That DMV is doing something strange that I don't fully understand. It's generating errors (which a normal UDF is not allowed to do), and those errors do not trigger a TRY/CATCH block or set ##error. EG
create table tempdb.dbo.foo(id int)
go
create view dbo.v_View_Obs_Table
as
select * from tempdb.dbo.foo
go
drop table tempdb.dbo.foo
go
begin try
Select * From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')
end try
begin catch
select ERROR_MESSAGE(); --<-- not hit
end catch
However these are real errors, as you can see running this from client code:
using System;
using System.Data.SqlClient;
namespace ConsoleApp6
{
class Program
{
static void Main(string[] args)
{
using (var con = new SqlConnection("Server=.;database=AdventureWorks;integrated security=true"))
{
con.Open();
con.FireInfoMessageEventOnUserErrors = true;
con.InfoMessage += (s, a) =>
{
Console.WriteLine($"{a.Message}");
foreach (SqlError e in a.Errors)
{
Console.WriteLine($"{e.Message} Number:{e.Number} Class:{e.Class} State:{e.State} at {e.Procedure}:{e.LineNumber}");
}
};
var cmd = con.CreateCommand();
cmd.CommandText = "Select * From sys.dm_sql_referenced_entities('dbo.v_View_Obs_Table','Object')";
using (var rdr = cmd.ExecuteReader())
{
while (rdr.Read() || (rdr.NextResult() && rdr.Read()))
{
Console.WriteLine(rdr[0]);
}
}
Console.ReadKey();
}
}
}
}
outputs
Invalid object name 'tempdb.dbo.foo'.
Invalid object name 'tempdb.dbo.foo'. Number:208 Class:16 State:3 at v_View_Obs_Table:4
0
The dependencies reported for entity "dbo.v_View_Obs_Table" might not include references to all columns. This is either because the entity references an object that does not exist or because of an error in one or more statements in the entity. Before rerunning the query, ensure that there are no errors in the entity and that all objects referenced by the entity exist.
The dependencies reported for entity "dbo.v_View_Obs_Table" might not include references to all columns. This is either because the entity references an object that does not exist or because of an error in one or more statements in the entity. Before rerunning the query, ensure that there are no errors in the entity and that all objects referenced by the entity exist. Number:2020 Class:16 State:1 at :1
I am using a camel jdbc component to insert a record into an Oracle table. the insert uses a sequence to populate a primary key ID column.
INSERT INTO my_table (id, data) VALUES (my_seq.nextval, 'some data')
The relevant part of the route looks like below:
from("some end point here")
.process(preInsertProcessor)
.to("jdbc:myDataSource")
.process(new Processor() {
public void process(Exchange exchange) throws Exception {
LOGGER.info("Extracting database generated id");
// This list is null
List<Integer> ids = exchange.getIn().getHeader(
JdbcConstants.JDBC_GENERATED_KEYS_DATA, List.class);
});
Inside the preInsertProcessir I set the message body to be my insert statement and also set some two header values to instruct camel I want the generated ID back:
message.setBody("INSERT INTO my_table (id, data) VALUES (my_seq.nextval, ?:data)");
message.setHeader("data", "some data");
message.setHeader(JDBC_RETRIEVE_GENERATED_KEYS, true);
message.setHeader(JDBC_GENERATED_COLUMNS, new String[]{"ID"});
End if I look at the logs I can see:
[DEBUG] org.apache.camel.component.bean.MethodInfo - Setting bean invocation result on the OUT message: [Message: INSERT INTO my_table(id, data)VALUES (my_seq.nextval, :?data]
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction begin (0x1de4bee0) redelivered(false) for (MessageId: ID-MELW1TYGC2S-62650-1438583607644-0-8 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-9))
[INFO ] au.com.nab.cls.router.non.repudiation.GeneratedIdExtractor - Extracting database generated id
[DEBUG] org.apache.camel.processor.MulticastProcessor - Done sequential processing 1 exchanges
[DEBUG] org.apache.camel.spring.spi.TransactionErrorHandler - Transaction commit (0x1de4bee0) redelivered(false) for (MessageId: ID:414d5120445041594855423120202020027844552045b302 on ExchangeId: ID-MELW1TYGC2S-62650-1438583607644-0-7))
If I get it well from the look of the logs the insert would be executed and my final processor should be able to get the generated ID. In reality what happens is that no record gets inserted and no ID is present in the header of the message. Without the final processor everything works fine.
Obviously I am doing something wrong here but I cannot figure out what. I am aware I could use a message en-richer to get the ID before the insert but I would prefer to avoid an extra database trip.
Thank you in advance for your inputs.
UPDATE
I put a break point in org.apache.camel.component.jdbc.JdbcProducer and found out the reason for not having the INSERT executed and consequently not getting the ID back.
// JdbcProducer code; creating a prepared statement part
if (shouldRetrieveGeneratedKeys) {
...
} else if (expectedGeneratedColumns instanceof String[]) {
// Execution gets herestatement
ps = conn.prepareStatement(preparedQuery, (String[]) expectedGeneratedColumns);
...
}
// Expected count returned here is 2
int expectedCount = ps.getParameterMetaData().getParameterCount();
if (expectedCount > 0) {
...
// And here I get the crash:
// java.sql.SQLException: Number of parameters mismatch. Expected: 2, was:1
getEndpoint().getPrepareStatementStrategy().populateStatement(ps, it, expectedCount);
}
This is where my research stopped as digging too much in the various three parties code is not really easy. I suspect one of the following two options are the cause:
I am still not doing it the right way
A camel bug which does not work as expected when header contains both named parameters and retrieve generated keys settings
Please advise about any fix or work around.
Thanks again
I also ran into this. My workaround:
Use camel-sql instead of camel-jdbc. Add parametersCount option to the endpoint URL (in adddition to setHeader(SqlConstants.SQL_RETRIEVE_GENERATED_KEYS, constant(true)) and setHeader(SqlConstants.SQL_GENERATED_COLUMNS, constant(new String[] {"ID_COLUMN_NAME"})).
Update: Works with 11.2.0.4 jdbc driver (does not work with 12.2.0.1 jdbc driver).
I'm inserting data into multiple tables, and I use the mybatis component to do that. I also need to create a temporary table before I can insert the data. High-level overview is:
Get data to insert
Create temp table
Insert data to temp table
Insert into table1 select x from temp table
Insert into table2 select y from temp table
Steps 2 to 5 should be their own single transaction, in case something fails. I've got this currently:
from(initialEndpoint)
.routeId("database-appender")
.aggregate().expression(constant(true)).completionSize(100).aggregationStrategy(new LinkListAggregator())
.transacted()
.bean(CreateTmpLinksTable.class)
.to("mybatis:prepareLinks?executorType=reuse&statementType=InsertList")
.to("mybatis:insertLinks?executorType=reuse&statementType=InsertList")
.to("mybatis:insertLinkSources?executorType=reuse&statementType=InsertList")
.end()
.log("Wrote at most ${body.size} links to the database")
The CreateTmpLinksTable needs to have access to the current connection, such that the creation of the temporary table does not happen in a different transaction (targeting PostgreSQL, if it matters).
I have this currently:
public class CreateTmpLinksTable {
public void createImportTable(Exchange exchange) throws SQLException {
final Connection conn = exchange.getIn().getHeader("TransactionConnection", Connection.class);
try (final Statement stat = conn.createStatement()) {
stat.execute("CREATE TEMPORARY TABLE tmp_links(" +
"url text, hostname text, service media, service_id bigint, user_id bigint, screen_name text, harvested_at timestamp with time zone, body text" +
") ON COMMIT DROP");
}
}
}
I also haven't setup my transaction manager. My suspicion is I have to get hold of the transaction manager, in order to correctly participate in the transaction.
Questions:
How do I get the transaction manager from a regular bean? Is it just a matter of getting the context, then from the context getting the manager through the registry?
Is there a better way to do what I need? I can see at least one: move all responsibilities into a single bean and do the work there. Any other ways?
NOTE: I'm learning Camel, and I like to do things using only code. Once I know how everything is wired up, then I can transfer that knowledge to Spring.
Q1,
If you can pass the instance of bean to the camel route, you can setup the transaction manager yourself, otherwise you have to use the registry to look up the transaction manager instance.
Q2,
You can wrap the DB update work in a single bean and use the transacted DSL in camel if you have other resources need to be managed.