I'm tasked with creating a Camel route using Camel version 2.20.0 that takes a line in from a CSV file uses a value from that line in the SQL statement where clause and merges the results and outputs them again. If I hardcode the identifier in the SQL statement it works fine, if I try and use a dynamic URI I get an error.
The route is:
from("file:///tmp?fileName=test.csv")
.split()
.tokenize("\n")
.streaming()
.parallelProcessing(true)
.setHeader("userID", constant("1001"))
//.enrich("sql:select emplid,name from employees where emplid = '1001'",
.enrich("sql:select name from employees where emplid = :#userID",
new AggregationStrategy() {
public Exchange aggregate(Exchange oldExchange,
Exchange newExchange) {...
As I said if I uncomment the line with the hardcoded 1001 it queries the db and works as expected. However using the ':#userID' syntax I get an Oracle error of:
java.sql.SQLSyntaxErrorException: ORA-00942: table or view does not exist
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[route3 ] [route3 ] [file:///tmp?fileName=test.csv ] [ 43]
[route3 ] [log5 ] [log ] [ 2]
[route3 ] [setHeader2 ] [setHeader[userID] ] [ 0]
[route3 ] [enrich2 ] [enrich[constant{sql:select name from employees where emplid = :#userID] [ 40]
The table is clearly there because it works when the value is hardcoded so it's got something to do with passing in the dynamic value. I've tried lots of variations on how to pass that variable in, inside single quotes, using values from the body instead of headers, etc. and haven't found the working combination yet though I've seen lots of similar seemingly working examples.
I've turned trace on it appears the header is correctly set as well:
o.a.camel.processor.interceptor.Tracer : >>> (route3) setHeader[userID, 1001] --> enrich[constant{sql:select name from employees where emplid = :#userID}] <<< Pattern:InOnly, Headers:{CamelFileAbsolute=true, CamelFileAbsolutePath=/tmp/test.csv, CamelFileLastModified=1513116018000, CamelFileLength=26, CamelFileName=test.csv, CamelFileNameConsumed=test.csv, CamelFileNameOnly=test.csv, CamelFileParent=/tmp, CamelFilePath=/tmp/test.csv, CamelFileRelativePath=test.csv, userID=1001}, BodyType:String, Body:1001,SomeValue,MoreValues
What needs to change to make this work?
I should also note I've tried this approach, using various syntax options to refer to the header value, without any luck:
.enrich().simple("sql:select * from employees where emplid = :#${in.header.userID}").aggregate ...
From the docs:
From Camel 2.16 onwards both enrich and pollEnrich supports dynamic endpoints that uses an Expression to compute the uri, which allows to use data from the current Exchange. In other words all what is told above no longer apply and it just works.
As you are using 2.20, I think you may try this example:
from("file:///tmp?fileName=test.csv")
.split()
.tokenize("\n")
.streaming()
.parallelProcessing(true)
.setHeader("userID", constant("1001"))
//.enrich("sql:select emplid,name from employees where emplid = '1001'",
.enrich("sql:select name from employees where emplid = ':#${in.header.userID}'",
new AggregationStrategy() {
public Exchange aggregate(Exchange oldExchange,
Exchange newExchange) {...
Take a look at the Expression topic in docs for further examples.
To sum up, the expression could be:
"sql:select name from employees where emplid = ':#${in.header.userID}'"
EDIT:
Sorry, I've missed the :# suffix. You could see a unit test working here.
Just take care with the columns types. If it's a integer, you shouldn't need the quotes.
Cheers!
From the Camel docs:
pollEnrich or enrich does not access any data from the current
Exchange which means when polling it cannot use any of the existing
headers you may have set on the Exchange.
The recommended way of achieving what you want is to instead use the recipientList, so I suggest you read up on that.
Content Enricher
Recipient List
Edit:
As Ricardo Zanini rightly pointed out in his answer it is actually possible to achieve this with Camel-versions from 2.16 onwards. As the OP is using 2.20 my answer is invalid.
I will, however, keep my answer but want to point out that this is only valid if you're using an older version than 2.16.
I have this query and I get the error in this function:
var accounts = from account in context.Accounts
from guranteer in account.Gurantors
select new AccountsReport
{
CreditRegistryId = account.CreditRegistryId,
AccountNumber = account.AccountNo,
DateOpened = account.DateOpened,
};
return accounts.AsEnumerable()
.Select((account, index) => new AccountsReport()
{
RecordNumber = FormattedRowNumber(account, index + 1),
CreditRegistryId = account.CreditRegistryId,
DateLastUpdated = DateLastUpdated(account.CreditRegistryId, account.AccountNumber),
AccountNumber = FormattedAccountNumber(account.AccountType, account.AccountNumber)
})
.OrderBy(c=>c.FormattedRecordNumber)
.ThenByDescending(c => c.StateChangeDate);
public DateTime DateLastUpdated(long creditorRegistryId, string accountNo)
{
return (from h in context.AccountHistory
where h.CreditorRegistryId == creditorRegistryId && h.AccountNo == accountNo
select h.LastUpdated).Max();
}
Error is:
There is already an open DataReader associated with this Command which must be closed first.
Update:
stack trace added:
InvalidOperationException: There is already an open DataReader associated with this Command which must be closed first.]
System.Data.SqlClient.SqlInternalConnectionTds.ValidateConnectionForExecute(SqlCommand command) +5008639
System.Data.SqlClient.SqlConnection.ValidateConnectionForExecute(String method, SqlCommand command) +23
System.Data.SqlClient.SqlCommand.ValidateCommand(String method, Boolean async) +144
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method, DbAsyncResult result) +87
System.Data.SqlClient.SqlCommand.RunExecuteReader(CommandBehavior cmdBehavior, RunBehavior runBehavior, Boolean returnStream, String method) +32
System.Data.SqlClient.SqlCommand.ExecuteReader(CommandBehavior behavior, String method) +141
System.Data.SqlClient.SqlCommand.ExecuteDbDataReader(CommandBehavior behavior) +12
System.Data.Common.DbCommand.ExecuteReader(CommandBehavior behavior) +10
System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior) +443
[EntityCommandExecutionException: An error occurred while executing the command definition. See the inner exception for details.]
System.Data.EntityClient.EntityCommandDefinition.ExecuteStoreCommands(EntityCommand entityCommand, CommandBehavior behavior) +479
System.Data.Objects.Internal.ObjectQueryExecutionPlan.Execute(ObjectContext context, ObjectParameterCollection parameterValues) +683
System.Data.Objects.ObjectQuery`1.GetResults(Nullable`1 forMergeOption) +119
System.Data.Objects.ObjectQuery`1.System.Collections.Generic.IEnumerable<T>.GetEnumerator() +38
System.Linq.Enumerable.Single(IEnumerable`1 source) +114
System.Data.Objects.ELinq.ObjectQueryProvider.<GetElementFunction>b__3(IEnumerable`1 sequence) +4
System.Data.Objects.ELinq.ObjectQueryProvider.ExecuteSingle(IEnumerable`1 query, Expression queryRoot) +29
System.Data.Objects.ELinq.ObjectQueryProvider.System.Linq.IQueryProvider.Execute(Expression expression) +91
System.Data.Entity.Internal.Linq.DbQueryProvider.Execute(Expression expression) +69
System.Linq.Queryable.Max(IQueryable`1 source) +216
CreditRegistry.Repositories.CreditRegistryRepository.DateLastUpdated(Int64 creditorRegistryId, String accountNo) in D:\Freelance Work\SuperExpert\CreditRegistry\CreditRegistry\Repositories\CreditRegistryRepository.cs:1497
CreditRegistry.Repositories.CreditRegistryRepository.<AccountDetails>b__88(AccountsReport account, Int32 index) in D:\Freelance Work\SuperExpert\CreditRegistry\CreditRegistry\Repositories\CreditRegistryRepository.cs:1250
System.Linq.<SelectIterator>d__7`2.MoveNext() +198
System.Linq.Buffer`1..ctor(IEnumerable`1 source) +217
System.Linq.<GetEnumerator>d__0.MoveNext() +96
This can happen if you execute a query while iterating over the results from another query. It is not clear from your example where this happens because the example is not complete.
One thing that can cause this is lazy loading triggered when iterating over the results of some query.
This can be easily solved by allowing MARS in your connection string. Add MultipleActiveResultSets=true to the provider part of your connection string (where Data Source, Initial Catalog, etc. are specified).
You can use the ToList() method before the return statement.
var accounts =
from account in context.Accounts
from guranteer in account.Gurantors
select new AccountsReport
{
CreditRegistryId = account.CreditRegistryId,
AccountNumber = account.AccountNo,
DateOpened = account.DateOpened,
};
return accounts.AsEnumerable()
.Select((account, index) => new AccountsReport()
{
RecordNumber = FormattedRowNumber(account, index + 1),
CreditRegistryId = account.CreditRegistryId,
DateLastUpdated = DateLastUpdated(account.CreditRegistryId, account.AccountNumber),
AccountNumber = FormattedAccountNumber(account.AccountType, account.AccountNumber)
})
.OrderBy(c=>c.FormattedRecordNumber)
.ThenByDescending(c => c.StateChangeDate)
.ToList();
public DateTime DateLastUpdated(long creditorRegistryId, string accountNo)
{
var dateReported = (from h in context.AccountHistory
where h.CreditorRegistryId == creditorRegistryId && h.AccountNo == accountNo
select h.LastUpdated).Max();
return dateReported;
}
Use the syntax .ToList() to convert object read from db to list to avoid being re-read again.
Here is a working connection string for someone who needs reference.
<connectionStrings>
<add name="IdentityConnection" connectionString="Data Source=(LocalDb)\v11.0;AttachDbFilename=|DataDirectory|\IdentityDb.mdf;Integrated Security=True;MultipleActiveResultSets=true;" providerName="System.Data.SqlClient" />
</connectionStrings>
In my case, using Include() solved this error and depending on the situation can be a lot more efficient then issuing multiple queries when it can all be queried at once with a join.
IEnumerable<User> users = db.Users.Include("Projects.Tasks.Messages");
foreach (User user in users)
{
Console.WriteLine(user.Name);
foreach (Project project in user.Projects)
{
Console.WriteLine("\t"+project.Name);
foreach (Task task in project.Tasks)
{
Console.WriteLine("\t\t" + task.Subject);
foreach (Message message in task.Messages)
{
Console.WriteLine("\t\t\t" + message.Text);
}
}
}
}
I dont know whether this is duplicate answer or not. If it is I am sorry. I just want to let the needy know how I solved my issue using ToList().
In my case I got same exception for below query.
int id = adjustmentContext.InformationRequestOrderLinks.Where(
item => item.OrderNumber == irOrderLinkVO.OrderNumber
&& item.InformationRequestId == irOrderLinkVO.InformationRequestId)
.Max(item => item.Id);
I solved like below
List<Entities.InformationRequestOrderLink> links =
adjustmentContext.InformationRequestOrderLinks
.Where(item => item.OrderNumber == irOrderLinkVO.OrderNumber
&& item.InformationRequestId == irOrderLinkVO.InformationRequestId)
.ToList();
int id = 0;
if (links.Any())
{
id = links.Max(x => x.Id);
}
if (id == 0)
{
//do something here
}
It appears that you're calling DateLastUpdated from within an active query using the same EF context and DateLastUpdate issues a command to the data store itself. Entity Framework only supports one active command per context at a time.
You can refactor your above two queries into one like this:
return accounts.AsEnumerable()
.Select((account, index) => new AccountsReport()
{
RecordNumber = FormattedRowNumber(account, index + 1),
CreditRegistryId = account.CreditRegistryId,
DateLastUpdated = (
from h in context.AccountHistory
where h.CreditorRegistryId == creditorRegistryId && h.AccountNo == accountNo
select h.LastUpdated
).Max(),
AccountNumber = FormattedAccountNumber(account.AccountType, account.AccountNumber)
})
.OrderBy(c=>c.FormattedRecordNumber)
.ThenByDescending(c => c.StateChangeDate);
I also noticed you're calling functions like FormattedAccountNumber and FormattedRecordNumber in the queries. Unless these are stored procs or functions you've imported from your database into the entity data model and mapped correct, these will also throw excepts as EF will not know how to translate those functions in to statements it can send to the data store.
Also note, calling AsEnumerable doesn't force the query to execute. Until the query execution is deferred until enumerated. You can force enumeration with ToList or ToArray if you so desire.
In my case, I had opened a query from data context, like
Dim stores = DataContext.Stores _
.Where(Function(d) filter.Contains(d.code)) _
... and then subsequently queried the same...
Dim stores = DataContext.Stores _
.Where(Function(d) filter.Contains(d.code)).ToList
Adding the .ToList to the first resolved my issue. I think it makes sense to wrap this in a property like:
Public ReadOnly Property Stores As List(Of Store)
Get
If _stores Is Nothing Then
_stores = DataContext.Stores _
.Where(Function(d) Filters.Contains(d.code)).ToList
End If
Return _stores
End Get
End Property
Where _stores is a private variable, and Filters is also a readonly property that reads from AppSettings.
As a side-note...this can also happen when there is a problem with (internal) data-mapping from SQL Objects.
For instance...
I created a SQL Scalar Function that accidentally returned a VARCHAR...and then...used it to generate a column in a VIEW. The VIEW was correctly mapped in the DbContext...so Linq was calling it just fine. However, the Entity expected DateTime? and the VIEW was returning String.
Which ODDLY throws...
"There is already an open DataReader associated with this Command
which must be closed first"
It was hard to figure out...but after I corrected the return parameters...all was well
In addition to Ladislav Mrnka's answer:
If you are publishing and overriding container on Settings tab, you can set MultipleActiveResultSet to True. You can find this option by clicking Advanced... and it's going to be under Advanced group.
I solved this problem by changing
await _accountSessionDataModel.SaveChangesAsync();
to
_accountSessionDataModel.SaveChanges();
in my Repository class.
public async Task<Session> CreateSession()
{
var session = new Session();
_accountSessionDataModel.Sessions.Add(session);
await _accountSessionDataModel.SaveChangesAsync();
}
Changed it to:
public Session CreateSession()
{
var session = new Session();
_accountSessionDataModel.Sessions.Add(session);
_accountSessionDataModel.SaveChanges();
}
The problem was that I updated the Sessions in the frontend after creating a session (in code), but because SaveChangesAsync happens asynchronously, fetching the sessions caused this error because apparently the SaveChangesAsync operation was not yet ready.
For those finding this via Google;
I was getting this error because, as suggested by the error, I failed to close a SqlDataReader prior to creating another on the same SqlCommand, mistakenly assuming that it would be garbage collected when leaving the method it was created in.
I solved the issue by calling sqlDataReader.Close(); before creating the second reader.
Most likely this issue happens because of "lazy loading" feature of Entity Framework. Usually, unless explicitly required during initial fetch, all joined data (anything that stored in other database tables) is fetched only when required. In many cases that is a good thing, since it prevents from fetching unnecessary data and thus improve query performance (no joins) and saves bandwidth.
In the situation described in the question, initial fetch is performed, and during "select" phase missing lazy loading data is requested, additional queries are issued and then EF is complaining about "open DataReader".
Workaround proposed in the accepted answer will allow execution of these queries, and indeed the whole request will succeed.
However, if you will examine requests sent to the database, you will notice multiple requests - additional request for each missing (lazy loaded) data. This might be a performance killer.
A better approach is to tell to EF to preload all needed lazy loaded data during the initial query. This can be done using "Include" statement:
using System.Data.Entity;
query = query.Include(a => a.LazyLoadedProperty);
This way, all needed joins will be performed and all needed data will be returned as a single query. The issue described in the question will be solved.
The same error happened to me when I was looping and updating data on
IEnumerable<MyClass>
When I changed the looped-on collection to be List<MyClass>, and filled it by converting by .ToList(), it solved and updated without any errors.
I had the same error, when I tried to update some records within read loop.
I've tried the most voted answer MultipleActiveResultSets=true and found, that it's just workaround to get the next error
New transaction is not allowed because there are other threads running
in the session
The best approach, that will work for huge ResultSets is to use chunks and open separate context for each chunk as described in
SqlException from Entity Framework - New transaction is not allowed because there are other threads running in the session
Well for me it was my own bug. I was trying to run an INSERT using SqlCommand.executeReader() when I should have been using SqlCommand.ExecuteNonQuery(). It was opened and never closed, causing the error. Watch out for this oversight.
This is extracted from a real world scenario:
Code works well in a Stage environment with MultipleActiveResultSets is set in the connection string
Code published to Production environment without MultipleActiveResultSets=true
So many pages/calls work while a single one is failing
Looking closer at the call, there is an unnecessary call made to the db and needs to be removed
Set MultipleActiveResultSets=true in Production and publish cleaned up code, everything works well and, efficiently
In conclusion, without forgetting about MultipleActiveResultSets, the code might have run for a long time before discovering a redundant db call that could be very costly, and I suggest not to fully depend on setting the MultipleActiveResultSets attribute but also find out why the code needs it where it failed.
I am using web service in my tool, where those service fetch the stored procedure. while more number of client tool fetches the web service, this problem arises. I have fixed by specifying the Synchronized attribute for those function fetches the stored procedure. now it is working fine, the error never showed up in my tool.
[MethodImpl(MethodImplOptions.Synchronized)]
public static List<t> MyDBFunction(string parameter1)
{
}
This attribute allows to process one request at a time. so this solves the Issue.
In my case, I had to set the MultipleActiveResultSets to True in the connection string.
Then it appeared another error (the real one) about not being able to run 2 (SQL) commands at the same time over the same data context! (EF Core, Code first)
So the solution for me was to look for any other asynchronous command execution and turn them to synchronous, as I had just one DbContext for both commands.
I hope it helps you
I have a requirement in which I have an af:query panel which after querying populates an af:table.
Now based on the table rows when any 1 row is selected, the graphs below it should be populated based on some columns.
Now my problem is that on search when the table is populated for the first time, no row is selected.
I need the first row to be selected automatically. I have searched multiple solutions from the net forums but till now haven't found any working solution.
Please help me what code should I use to select a row programmatically. and also where should I put this code in the backing bean.
As you've correctly mentioned, at the first render of the table no row is selected. I've encountered this problem too, and i've dealt with it by calling a getting the first row of the corresponding VO on BeforePhase (i.e: on the page first rendering or refresh).
I understand that you intend to do so for the first searching.
The af:query component has a property called QueryListener. You can link it with a method inside a backing bean. The content should be like:
private QueryEvent qEvent = null;
public void queryListener(QueryEvent queryEvent) {
setQEvent(queryEvent);
JSFUtils.invokeMethodExpression("#{bindings.YourViewObjectNameViewCriteriaQuery.processQuery}", Object.class,
QueryEvent.class, getQEvent());
BindingContainer bindings = BindingContext.getCurrent().getCurrentBindingsEntry(); //Edited
DCBindingContainer bc = (DCBindingContainer)bindings;
DCIteratorBinding iterator = bc.findIteratorBinding("YourViewObject1Iterator");
Row r = iterator.getCurrentRow(); //Here you've got the very first row, and can operate with it
.....
}
public void setQEvent(QueryEvent qEvent) {
this.qEvent = qEvent;
}
public QueryEvent getQEvent() {
return qEvent;
}
With this, you should be able to get the first row when the query is executed (i.e: when the search is done).
After getting the first row, you can programmatically the graph process execution or whatever you do.
NOTE: invokeMethodExpression can be found inside JSFUtils, which is a starndard class with static methods which source code you can download here: JSFUtils.java
I am trying to create my first NHibernate project, so it is possible I am doing something stupid here but have been Googling for a couple of days and not had any joy yet.
I have an Article object which has various properties:
public class Article {
public virtual string Title { get; set; }
public virtual string Body { get; set; }
}
I am using the fluent configuration to load the mappings:
configuration.Mappings(m => m.AutoMappings.Add((AutoMap.AssemblyOf<Article>())
.Conventions.Add(DefaultCascade.All())
.UseOverridesFromAssemblyOf<SchemaConfigurationController>())
The override is:
public class ArticleOverrideMapping : IAutoMappingOverride<Article>
{
public void Override(AutoMapping<Article> mapping)
{
//mapping.Map(x => x.Body).CustomSqlType("NVARCHAR(4000)");
//mapping.Map(article => article.Body).Length(10000);
//mapping.Map(article => article.Body).Length(Int32.MaxValue);
//mapping.Map(article => article.Body).CustomSqlType("NVARCHAR(max)");
//mapping.Map(article => article.Body).CustomSqlType("NVARCHAR(max)").Length(Int32.MaxValue);
mapping.Map(article => article.Body).CustomType("StringClob").CustomSqlType("NVARCHAR(max)");
}
}
I have tried each of the commented out lines (roughly in order of when I found a possible solution online). I can get SQL Server to create the nvarchar(max) column and if I use SQL management studio I can paste a LOT (185,602 words was largest test) into the Body column. My issue is trying to get it to save from the MVC site using NHibernate.
There are two main errors I get:
String or binary data would be truncated. The statement has been terminated.
This would occur if I didn't set the ".Length(Int32.MaxValue)" override.
The second Error that occurs (when the length is set):
The length of the string value exceeds the length configured in the mapping/parameter.
I am pretty confused as to what I should be doing at this point. My goal is to be able to store a very large string (an whole article) in SQL Server (and SQLite for testing, nvarchar(max) wasn't liked by SQLite) and get that back out, (and edit it) in an MVC site.
UPDATE
As per #Cymen's link I tried
.CustomSqlType("nvarchar(max)").Length(Int32.MaxValue).Nullable();
but this lead to the error:
The length of the string value exceeds the length configured in the mapping/parameter.
when I only tried to save 1201 words (all the word "test"). When I added the length on the end of the above mapping ".Length(Int32.MaxValue)" I still the same error.
update
wanted to confirm which versions I am using:
FluentNHibernate.1.3.0.733
NHibernate.3.3.1.4000
Microsoft.AspNet.Mvc.4.0.20710.0
final update
Kieren had it correct, I had completely forgotten that I took that property and ran markdownsharp on it on the server and populated a second property on the server. So it was the second property that I hadn't mapped that was actually blowing up, sorry.
this is how i normally handle it.
mapping.Map(x => x.Problem).CustomType("varchar(MAX)");
not sure what CustomSqlType is, but i've never used it, and this works.
Ok, folks. Same stuff, so i've did some tests.
It seems that int.MaxValue cannot be used as LENGTH for fluent. If you do that, the resulting create SQL will be nvarchar(255). So if nhibernate (fluent) still generates such a script, then he is awaiting 255 chars at max.
If you hovewer use int.MaxValue / 2, then everything is ok. At least in the script.
Only reasonable explanation: unicode string, so for a single char fluent automatically takes 2 bytes. SO internally fluent hibernate may do a multiplication by 2. And if we get above 2GB space, who knows what fluent will do...
Please let me know if it works with data too.