We can use vpi_scan in the following way:
vpiHandle iter = vpi_iterate(property, handle);
if (iter)
while ( entry = vpi_scan(iter) )
/*code*/;
iter will be freed when vpi_scan() returns NULL.
But what if I need to scan through the loop several times? Is there a way to tell vpi_scan not to free the iterator, so that I could vpi_free_object when I'm done?
I think I can solve the problem using an additional container (to collect and keep all entries), but is it really needed?
EDIT:
1. I would not like to call vpi_iterate more than once, since it can be expensive.
2. Suppose I go with an additional container solution. Is there a way to find out the number of entries in the vpi_scan loop without actual scanning through the loop? This could make implementation simpler.
ut what if I need to scan through the loop several times? vpi_iterate returns an initialized pointer to the iterator. Every vpi_scan removes an element from the list and frees it. If vpi_scan did not run till the end, you'd better use vpi_free_object to clean the rest of the iterator list. If you need to rescan the same object again, you can call vpi_iterate again and it will return a new iterator object which you can re-scan.
s there a way to tell vpi_scan not to free the iterator, so that I could vpi_free_object when I'm done? No, (1) is the only mechanism which can be used to access and free iterator elements. There is no other exist in the standard.
I think I can solve the problem using an additional container (to collect and keep all entries), but is it really needed? -- this is a good idea if you want to re-scan your data structs. It could be much better from the performance point of view. Verilog simulator usually has a separately generated vpi database which it needs to consult, probably multiple times to provide you with vpi access information.
Is there a way to find out the number of entries in the vpi_scan loop without actual scanning through the loop? not really. there are no defined way to query this information from the iterator. There might be a work-around with using vpi_handle(vpiUse, iterator) but it depends on the underlying data and the type of iteration. It is much easier to use linked lists created by the first scanning loop.
you can get all additional information for LRM or a verilog pli handbook.
Related
I notice that if I apply a mapPartitions on an RDD, the partitions get an iterable object. Within the mapPartitions function, I then call the toArray member function of the iterable to convert that iterable object to an Array object. Does calling toArray involves copying, or does it just start referencing the same part of the memory as an Array? If it does involve copying, what are ways to prevent copying?
One important correction to your question -- the partition data structure exposed during mapPartitions is an Iterator, not Iterable. Here's the interface difference:
An Iterator has the next() and hasNext() methods, which allow you to visit each element in the collection once. Once the next() method of an iterator is called, the last element is gone (unless you've stored it in a variable).
An Iterable has the ability to produce an Iterator whenever you want. This lets you visit each element as many times as you want.
In terms of implementation, an Iterator can stream through data. You really only need to have one element in memory at a time, which is loaded when next() is called. If you're reading from a text file with Spark (sc.textFile) it does exactly this, and uses almost no memory to do simple iteration through partitions.
You're absolutely allowed to call iterator.toArray, but you probably don't want to. You end up shoving all of the data into memory (Spark can't load just one element at a time, because you've asked for all of it at once), and either copy each piece of the data (for primitives, like Int) or allocate a new reference for each piece of data (for AnyRef, like Array[_]). There is no way to prevent this copying.
There are times when converting a partition iterator to an array is what you want to do, but these use cases are rare. You risk running out of memory and slowing down your application a huge amount due to unnecessary allocation and GC, so think hard about whether it's really needed!
How do I efficiently find which array in a list of array has members.
The context is for message brokering. I have a list of queue and need to periodically push their content to consumers, but since the number of queues can be large, I want to quickly find out which ones have elements in them.
My only solution currently is to basically keep a separate list which keeps track of which array still has content in it, but was curious about alternatives, and it just bugs the hell out of me ;)
I am using javascript but am hoping for a more general idea of which algorihtm should I look into.
I'm going to make this language-agnostic and keep my answer short and simple.
Maintain a parallel data-structure: another set storing array IDs, but only those of arrays that are nonempty.
When you make a new (empty) array, add it to the first set only.
When you add to an array, add it to the second set iff it's not already there.
When you remove from an array, if the array is now empty, remove it from the second set.
When you delete an array, remove it from the the first set (and the second, if it's there).
If you're using a hashset for this second set, insertion and deletion times don't change. This is the most efficient thing one could ask for, and a hybrid datastructure of this ilk is necessary AFAICT.
You can then find all nonempty arrays by iterating over the second set.
I am writing a program in C that reads a file. Each line of the file is a string of characters to which a computation will be done. The result of the computation on a particular string may imply that strings latter on in the file do not need any computations done to them. Also if the reverse of the string comes in alphabetical order before the (current, non-reversed) string then it does not need to be checked.
My question is would it be better to put each string in a linked list and delete each node after finding particular strings don’t need to be checked or using an array and checking the last few characters of a string and if it is alphabetically after the string in the previous element skip it? Either way the list or array only needs to be iterated through once.
Rules of thumb is that if you are dealing with small objects (< 32 bytes), std::vector is better than a linked list for most of general operations.
But for larger objects, (say, 1K bytes), generally you need to consider lists.
There is an article details the comparison you can check , the link is here
http://www.baptiste-wicht.com/2012/11/cpp-benchmark-vector-vs-list/3/
Without further details about what are your needs is a bit difficult to tell you which one would fit more with your requirements.
Arrays are easy to access, specially if you are going to do it in a non sequential way, but they are hard to maintain if you need to perform deletions on it or if you don't have a good approximation of the final number of elements.
Lists are good if you plan to access them sequentially, but terrible if you need to jump between its elements. Also deletion over them can be done in constant time if you are already in the node you want to delete.
I don't quite understand how you plan to access them since you say that either one would be iterated just once, but if that is the case then either structure would give you the similar performance since you are not really taking advantage of their key benefits.
It's really difficult to understand what you are trying to do, but it sounds like you should create an array of records, with each record holding one of your strings and a boolean flag to indicate whether it should be processed.
You set each record's flag to true as you load the array from the file.
You use one pointer to scan the array once, processing only the strings from records whose flags are still true.
For each record processed, you use a second pointer to scan from the first pointer + 1 to the end of the array, identify strings that won't need processing (in light of the current string), and set their flags to false.
-Al.
Is there a faster way to do this:
Vector3* points = malloc(maxBufferCount*sizeof(Vector3));
//put content into the buffer and increment bufferCount
...
// remove one point at index `removeIndex`
bufferCount--;
for (int j=removeIndex; j<bufferCount; j++) {
points[j] = points[j+1];
}
I'm asking because I have a huge buffer from which I remove elements quite often.
No, sorry - removing elements from the middle of an array takes O(n) time. If you really want to modify the elements often (i. e. remove certain items and/or add others), use a linked list instead - that has constant-time removal and addition. In contrast, arrays have constant lookup time, while linked lists can be accessed (read) in linear time. So decide what you will do more frequently (reading or writing) and choose the appropriate data structure based upon that decision.
Note, however, that I (kindly) assumed you are not trying to commit the crime of premature optimization. If you haven't benchmarked that this is the bottleneck, then probably just don't worry about it.
Unless you know it's a bottleneck you can probably let the compiler optimize for you, but you could try memmove.
The selected answer here is pretty comprehensive: When to use strncpy or memmove?
A description is here: http://www.kernel.org/doc/man-pages/online/pages/man3/memmove.3.html
A few things to say. The memmove function will probably copy faster than you, often it is optimised by the writers of your particular complier to use special instructions which arent available in the C language without inline assembler. I believe these instructions are called SIMD instructions (Single Instruction Multiple Data)? Somebody correct me if I am wrong.
If you can save up items to be removed, then you can optimse by sorting the list of items you wish to remove and then, doing a single pass. It isnt hard but just takes some funny arithmetic.
Also you could just store each item in a linked list, removing an item is trivial, but you lose random acccess to your array.
Finally you can have an additional array of pointers, the same size of your array, each pointer pointing to an element. Then you can access the array through double indirection, you can sort the array by swapping pointers, and you can delete items by making their pointer NULL.
Hope this gives you some ideas. There usually is a way to optimise things, but then it becomes more application specific.
Im writing timers manager in C, which involves:
creating new timer
removing timers
removing dead timer
freezing timers
and all the other stuff which i did not yet think about.
The key is - amount of memory should be as small as possible.
At first i thought about linked list, but if i remove some of the middle part, i should rebuild list, which can take some time. Typical dynamic array is the same - i should be carefull with pointers to not miss some of them, when Im rebuliding that structure.
Any ideas ?
Thx for all answer
You don't need to rebuild anything when removing from a linked list. It's an O(1) op. No matter what structure you'll choose you'll probably have to be careful about pointers.
Generally an array uses minimal memory. Single block, so less allocation overhead, and no management overhead like the next/previous pointers in a linked list.
Of course it's less time-efficient to remove from the start/middle of a sufficiently large array than it is to remove a given node from a linked list, and also any pointers/indexes into the array will no longer refer to the same element once you've done that, so you have to be careful what handles you give to users of the timer API and how you find the timer data for a particular handle. But if memory use really is the only important issue then the array wins.
I've done something like this before and personally I'd start with a linked list, unless it was obviously going to fail some particular constraint. An array of pointers to "timerdata" structs might also work well, provided you can prevent the list of active timers getting too big.
If the amount of memory has to be as small as possible I would go for an array, the only downside to using an array is that resizing the array is expensive (lots of cpu cycles), However if you can set an upper limit on the number of timers then using a single array sized to this max value will be efficient. If the number of timers is extremely dynamic a list (vector collection) is the way to go.
You could use a heap/priority queue. This has the same memory requirements as an array but insertion and removal are O(lg n) ops. The items in the queue could be ordered by the time-left-to-trigger of each timer object. When a real timer triggers you'll have to adjust the trigger time of every other timer in the list. So, if timer1 is set to go off after 1 second, and timer2 is set to go off after 1.5 seconds, when timer1 triggers you'll have to adjust timer2 to go off after 0.5 seconds. Something like that maybe?
What is a timer element? What's in it? Is it just a value that is continually counted down by some timer interrupt or some complex class, perhaps with its own thread that does timing?
What operations are likely to be performed most often? - optimize access to those as a priority.
Is there any advantage to be had in ordering the list, eg. in ascending expiry-time order, so that the timer timing out next is always at the start of the list, (ie. delta-queue).
Rgds,
Martin