I've read this patterns but found it not to work. I get a rare exception that an item in foreach was changed.
lock (mylist)
{
foreach(var a in myList) {}
myList = new List<>() (or myList.Clear() )
}
I also tried this
foreach(var a in myList.ToList() ) { }
And that also generated exceptions. There are some other patterns described in this thread but I want to confirm/understand why above patterns didnt work
I've read a bit about how to properly lock a list. The exception doesn't occur often-- just very rarely and there was also a memory leak at the time.
1.
Do I need to use the lock everywhere I modify myList or does the lock prevent anyone from editing mylist? This may be the source of confusion.
2.
Is there a difference in lock mylist and casting and using syncroot?
See here
Properly locking a List<T> in MultiThreaded Scenarios?
Generally, if you have a shared resource then you need to lock a mutex that protects the resource whenever you use this resource. Does not matter if it's just reading or writing. If mutex is not locked in at least one of the places that you use the shared resource, you have a problem. For example, if you only lock a shared resource while modifying it, then it is possible that some thread is reading it while another is modifying it - a situation known as race condition occurs.
In your particular case, yes, you need to lock mylist everywhere you modify it. And not only where you modify it, but also everywhere you read it.
Related
I have a problem that i can't solve.
I have to make a data structure shared by some thread, the problems are:
The thread are executed simultaneusly and they should insert data in an particular structure, but every object should be inserted in mutex esclusion, because if an object is alredy present it must not be re-inserted.
I have think of making an array where threads put the key of object they are working, if another thread wants to put the same key it should wait for the current thread finish.
so, in other words every thread execute this function for lock element:
void lock_element(key_t key){
pthread_mutex_lock(&mtx_array);
while(array_busy==1){
pthread_cond_wait(&var_array,&mtx_array);
}
array_busy=1;
if((search_insert((int)key))==-1){
// the element is present in array and i can't insert,
// and i must wait for the array to be freed.
// (i think that the problem is here)
}
array_busy=0;
pthread_cond_signal(&var_array);
pthread_mutex_unlock(&mtx_array);
}
after i finish with the object i free the key in the arry with the follow function:
void unlock_element(key_t key){
pthread_mutex_lock(&mtx_array);
while(array_busy==1){
pthread_cond_wait(&var_array,&mtx_array);
}
array_busy=1;
zeroed((int)key);
array_busy=0;
pthread_cond_signal(&var_array);
pthread_mutex_unlock(&mtx_array);
}
in this way, the result change in every execution (for example: in a first time the program insert 300 object, and in a second time insert 100 object).
Thanks for the help!
UPDATE:
#DavidSchwartz #Asthor I modified the code as follows:
void lock_element(key_t key){
pthread_mutex_lock(&mtx_array);
while((search_insert((int)key))==-1){
//wait
pthread_cond_wait(&var_array,&mtx_array);
}
pthread_mutex_unlock(&mtx_array);
}
and...
void unlock_element(key_t key){
pthread_mutex_lock(&mtx_array);
zeroed((int)key);
pthread_cond_signal(&var_array);
pthread_mutex_unlock(&mtx_array);
}
But not work.. It behaves in the same way as before.
I also noticed a strange behavior of the function search_insert(key);
int search_insert(int key){
int k=0;
int found=0;
int fre=-1;
while(k<7 && found==0){
if(array[k]==key){
found=1;
} else if(array[k]==-1) fre=k;
k++;
}
if (found==1) {
return -1; //we can't put the key in the array
}else {
if(fre==-1) exit(EXIT_FAILURE);
array[fre]=key;
return 0;
}
}
never goes in
if(found == 1)
You have a couple of options.
The simplest option is just to hold the mutex during the entire operation. You should definitely choose this option unless you have strong evidence that you need greater concurrency.
Often, it's possible to just allow more than one thread to do the work. This pattern works like this:
Acquire the mutex.
Check if the object is in the collection. If so, use the object from the collection.
Otherwise, release the mutex.
Generate the object
Acquire the mutex again.
Check if the object is in the collection. If not, add it and use the object you generated.
Otherwise, throw away the object you generated and use the one from the collection.
This may result in two threads doing the same work. That may be unacceptable in your use case either because it's impossible (some work can only be done once) or because the gain in concurrency isn't worth the cost of the duplicated work.
If nothing else works, you can go with the more complex solution:
Acquire the mutex.
Check if the object is in the collection. If so, use the object in the collection.
Check if any other thread is working on the object. If so, block on the condition variable and go to step 2.
Indicate that we are working on the object.
Release the mutex.
Generate the object.
Acquire the mutex.
Remove the indication that we are working on the object.
Add the object to the collection.
Broadcast the condition variable.
Release the mutex.
This can be implemented with a separate collection just to track which objects are in progress or you can add a special version of the object to the collection that contains a value that indicates that it's in progress.
The answer is based on assumptions as it is.
Consider this scenario. You have 2 threads trying to insert their objects. Thread 1 and thread 2 both get objects with index 0. We then present 2 possible scenarios.
A:
Thread 1 starts, grabs the mutex and proceeds to insert their object. They finish, letting the next thread through from the mutex, which is 2. Thread 1 tries to get the mutex again to release the index but is blocked as thread 2 has it. Thread 2 tries to insert their object but fails due to the index being taken, so the insert never happens. It releases the mutex and thread 1 can grab it, releasing the index. However thread 2 has already attempted and failed to insert the object it had, meaning that we only get 1 insertion in total.
B:
Second scenario. Thread 1 starts, grabs the mutex, inserts the object, releases the mutex. Before thread 2 grabs it, thread 1 again grabs it, clearing the index and releases the mutex again. Thread 2 then successfully grabs the mutex, inserts the object it had before releasing the mutex. In this scenario we get 2 inserts.
In the end, the issue lies with there being no reaction inside the if statement when a thread fails to insert an object and the thread, not doing what it is meant to. That way you get less insertions than expected.
So I was using an array to hold objects, then iterating over them to call a particular method. As the objects in this array could be enumerated/mutated on different threads I was using an NSRecursiveLock.
I then realised that as this method is always called on every object in this array, I could just use an NSNotification to trigger it, and do away with the array and lock.
I am just double checking that this is a good idea? NSNotification will be faster than a lock, right?
Thanks
First, it seems that the locking should be done on the objects, not on the array. You are not mutating the array, but the objects. So you need one mutex per object. It will give you a finer granularity and allow concurrent updates to different objects to proceed in parallel (it wouldn't with a global lock).
Second, a recursive lock is complete overkill. If you wish to have mutual exclusion on the objects, each object should have a standard mutex. If what you are doing inside the critical section is cpu bound and really short, you might consider using a spinlock (it won't even trap in the OS. Only use for short CPU bound critical sections though). Recursive mutex are meant to be used when a thread k can (because of its logic) acquire another lock on an object it has already locked himself (hence the recursive name).
Third, NSNotifications (https://developer.apple.com/library/mac/documentation/Cocoa/Reference/Foundation/Classes/NSNotification_Class/) will allocate memory, drop the notification in a notification Center, do locking to actually implement that (the adding to the center) and finally dispatch the notifications in the center and deallocate the notification. So it is heavier than a plain simple locking. It "hides" the synchronization APIs inside the center, but it does not eliminate them, and you pay the price of memory allocations/deallocations.
If you wish to modify the array (add/remove), then you should also synchronize on these operations, but that would be unfortunate as access to independent entries of the array would now collide.
Hope that helps.
I have two loops:
One loop gets data from a device and processes it. Scales received variables, calculates extra data.
Second loop visualizes the data and stores it.
There are lots of different variables that need to passed between those two loops - about 50 variables. I need the second loop to have access only to the newest values of the data. It needs to be able to read those variables any time they are needed to be visualized.
What is the best way to share such vector between two loops?
There are various ways of sharing data.
The fastest and simplest is a local variable, however that is rather uncontrolled, and you need to make sure to write them at one place (plus you need an indicator).
One of the most advanced options is creating a class for your data, and use an instance (if you create a by-ref class, otherwise it won't matter), and create a public 'GET' method.
In between you have sevaral other options:
queues
semaphores
property nodes
global variables
shared variables
notifiers
events
TCP-IP
In short there is no best way, it all depends on your skills and application.
As long as you're considering loops within the SAME application, there ARE good and bad ideas, though:
queues (OK, has most features)
notifiers (OK)
events (OK)
FGVs (OK, but keep an eye on massively parallel access hindering exec)
semaphores (that's not data comms)
property nodes (very inefficient, prone to race cond.)
global variables (prone to race cond.)
shared variables (badly implemented by NI, prone to race cond.)
TCP-IP (slow, awkward, affected by firewall config)
The quick and dirty way to do this is to write each value to an indicator in the producer loop - these indicators can be hidden offscreen, or in a page of a tab control, if you don't want to see them - and read a local variable of each one in the consumer loop. However if you have 50 different values it may become hard to maintain this code if you need to change or extend it.
As Ton says there are many different options but my suggestion would be:
Create a cluster control, with named elements, containing all your data
Save this cluster as a typedef
Create a notifier using this cluster as the data type
Bundle the data into the cluster (by name) and write this to the notifier in the producer loop
Read the cluster from the notifier in the consumer loop, unbundle it by name and do what you want with each element.
Using a cluster means you can easily pass it to different subVIs to process different elements if you like, and saving as a typedef means you can add, rename or alter the elements and your code will update to match. In your consumer loop you can use the timeout setting of the notifier read to control the loop timing, if you want. You can also use the notifier to tell the loops when to exit, by force-destroying it and trapping the error.
Two ways:
Use a display loop with SEQ (Single Element Queue)
Use a event structure with User Event. (Do not put two event structures in same loop!! Use another)
Use an enum with case structure and variant to cast the data to expected type.
(Notifier isn't reliable to stream data, because is a lossy scheme. Leave this only to trigger small actions)
If all of your variables can be bundled together in a single cluster to send at once, then you should use a single element queue. If your requirements change later such that the transmission cannot be lossy, then it's a matter of changing the input to the Obtain Queue VI (with a notifier you'd have to swap out all of the VIs). Setting up individual indicators and local variables would be pretty darn tedious. Also, not good style.
If the loops are inside of the same VI then:
The simplest solution would be local variables.
Little bit better to use shared variables.
Better is to use functional global variables (FGVs)
The best solution would be using SEQ (Single Element Queue).
Anyway for better understanding please go trough this paper.
I Am creating simple WPF test project which contains multiple UserControls(Insteda of Pages).I Am using Switcher Class to navigate between different UserControls.When i navigate to different pages,i have observed that memory consuption keep on increasing on each UserControle Navigationand GC is not invoked.
1.So am i doing something wrong in following code?
2.Which part of the code consuming more memory?
3.Do i need to invoke GC for disposing my UserControls on each new UserControle creation?
If need how can i invoke GC?
public void On_Navigate_Click()
{
UserControle newusercontrole=new UserControle();
DataSet ds = new DataSet();
ds=con.getSome_Datafrom_SQL();//Gets data from SQL via connection class
dataGrid_test.ItemsSource = ds.Tables[0].DefaultView;
Grid.SetColumn(newusercontrole, 1);//dataGrid_test is inside newusercontrole and following is the code to add "this" usercontrol to the main window.
Grid.SetRow(newusercontrole, 1);
Grid.SetZIndex(newusercontrole, 10);
Container.Children.Add(newusercontrole);
}
First off, I must point out that if garbage collection really isn't happening (as you said), it's not your fault and it does not mean you're doing something wrong. It only means that the CLR doesn't think that your system is under memory pressure yet.
Now, to manually invoke a garbage collection cycle anyway, you can use the GC.Collect() static method. If a garbage collection actually starts and your memory consumption is still unreasonably high, this means that you're probably doing something wrong: You're keeping an ever increasing number of unnecessary object references and the garbage collector cannot safely collect those objects. This is a kind of a memory leak.
As far as your code goes, I think that the problem is at the end of the method you posted:
Container.Children.Add(newusercontrole);
This seems to add a new object (on every click) to the collection Container.Children. If this is not removed elsewhere, this is probably the cause of your memory leak. I don't know what the suitable solution would be for your use case (since I don't know exactly how your UI should behave), but you'll likely need to find a way to remove the last UserControle added from the Container.Children. If you can use LINQ, then the methods OfType<T>() and Last() could be of use to find it.
In any case, don't leave the GC.Collect() line in production code. Use it only to force a collection cycle for testing purposes, like this one.
I have been having some issues with LINQ-To-SQL around memory usage. I'm using it in a Windows Service to do some processing, and I'm looping through a large amount of data that I'm pulling back from the context. Yes - I know I could do this with a stored procedure but there are reasons why that would be a less than ideal solution.
Anyway, what I see basically is memory is not being released even after I call context.SubmitChanges(). So I end up having to do all sorts of weird things like only pull back 100 records at time, or create several contexts and have them all do separate tasks. If I keep the same DataContext and use it later for other calls, it just eats up more and more memory. Even if I call Clear() on the "var tableRows" array that the query returns to me, set it to null, and call SYstem.GC.Collect() - it still doesn't release the memory.
Now I've read some about how you should use DataContexts quickly and dispose of them quickly, but it seems like their ought to be a way to force the context to dump all its data (or all its tracking data for a particular table) at a certain point to guarantee the memory is free.
Anyone know what steps guarantee that the memory is released?
A DataContext tracks all the objects it ever fetched. It won't release this until it is garbage collected. Also, as it implements IDisposable, you must call Dispose or use the using statement.
This is the right way to go:
using(DataContext myDC = new DataContext)
{
// Do stuff
} //DataContext is disposed
If you don't need object tracking set DataContext.ObjectTrackingEnabled to false. If you do need it, you can use reflection to call the internal DataContext.ClearCache(), although you have to be aware that since its internal, it's subject to disappear in a future version of the framework. And as far as I can tell, the framework itself doesn't use it but it does clear the object cache.
As Amy points out, you should dispose of the DataContext using a using block.
It seems that your primary concern is about creating and disposing a bunch of DataContext objects. This is how linq2sql is designed. The DataContext is meant to have short lifetime. Since you are pulling a lot of data from the database, it makes sense that there will be a lot of memory usage. You are on the right track, by processing your data in chunks.
Don't be afraid of creating a ton of DataContexts. They are designed to be used that way.
Thanks guys - I will check out the ClearCache method. Just for clarification (for future readers), the situation in which I was getting the memory usuage was something like this:
using(DataContext context = new DataContext())
{
while(true)
{
int skipAmount = 0;
var rows = context.tables.Select(x => x.Dept == "Dept").Skip(skipAmount).Take(100);
//break out of loop when out of rows
foreach(table t in rows)
{
//make changes to t
}
context.SubmitChanges();
skipAmount += rows.Count();
rows.Clear();
rows = null;
//at this point, even though the rows have been cleared and changes have been
//submitted, the context is still holding onto a reference somewhere to the
//removed rows. So unless you create a new context, memory usuage keeps on growing
}
}
I just ran into a similar problem. In my case, helped establish the properties of DataContext.ObjectTrackingEnabled to false.
But it works only in the case of iterating through the rows as follows:
using (var db = new DataContext())
{
db.ObjectTrackingEnabled = false;
var documents = from d in db.GetTable<T>()
select d;
foreach (var doc in documents)
{
...
}
}
If, for example, in the query to use the methods ToArray() or ToList() - no effect