I want to depict something like
A::foo() { //public API
internal();
}
A::internal() {
B.bar()
}
So I tried
->A: foo()
A -> A :internal()
A -> B :bar()
But I end up with something like
,-. ,-.
|A| |B|
`+' `+'
foo()| |
-----> |
| |
|----.
| | internal()
|<---'
| |
| bar() |
| -------------->
,+. ,+.
|A| |B|
`-' `-'
Which rather ends up denoting that bar() is invoked after completion of internal. How do I get the call to bar() to show up from within internal()?
You can show this with activation boxes:
->A: foo()
activate A
A -> A :internal()
activate A
A -> B :bar()
activate B
deactivate B
deactivate A
deactivate A
Related
I haven't programmed C for quite some time and my pointer-fu had degraded. I made a very elementary mistake and it took me well over an hour this morning to find what I'd done. The bug is minimally reproduced here: https://godbolt.org/z/3MdzarP67 (I am aware the program is absurd memory-management wise, just showing what happens).
The first call to realloc() breaks because of course, the pointer it's given points to stack memory, valgrind made this quite obvious.
I have a rule with myself where any time I track down a bug, if there is a warning that could have caught it I enable it on my projects. Often times this is not the case, since many bugs come from logic errors the compiler can't be expected to check.
However here I am a bit surprised. We malloc() and then immediately reassign that pointer which leaves the allocated memory inaccessible. It's obvious the returned pointer does not live outside the scope of that if block, and is never free()'d. Maybe it's too much to expect the compiler to analyze the calls and realize we're attempting to realloc() stack memory but I am surprised that I can't find anything to yell at me about the leaking of the malloc() returned pointer. Even clang's static analyzer scan-build doesn't pick up on it, I've tried various relevant options.
The best I could find was -fsanitize=address which at least prints out some cluing information during the crash instead of:
mremap_chunk(): invalid pointer
on Godbolt, or
realloc(): invalid old size
Aborted (core dumped)
on my machine, both of which are somewhat cryptic (although yes they do show plainly that there is some memory issue occurring). Still, this compiles without issues.
Since Godbolt links don't live forever here is the critical section of the code:
void add_foo_to_bar(struct Bar** b, Foo* f) {
if ((*b)->foos == NULL) {
(*b)->foos = (Foo*)malloc(sizeof(Foo));
// uncomment for correction
//(*b)->foos[(*b)->n_foos] = *f;
// obvious bug here, we leak memory by losing the returned pointer from malloc
// and assign the pointer to a stack address (&f1)
// comment below line for correction
(*b)->foos = f; // completely wrong
(*b)->n_foos++;
} else {
(*b)->foos = (Foo*)realloc((*b)->foos, ((*b)->n_foos + 1) * sizeof(Foo));
(*b)->foos[(*b)->n_foos] = *f;
(*b)->n_foos++;
}
}
the error occurs because f is a pointer to stack memory (intentional) but we obviously can't assign something that was supposed to have been malloc()'d to that.
Try -fanalyzer if your compiler is recent enough. When running it I get:
../main.c:30:28: warning: ‘realloc’ of ‘&f1’ which points to memory not on the heap [CWE-590] [-Wanalyzer-free-of-non-heap]
30 | (*b)->foos = (Foo*)realloc((*b)->foos, ((*b)->n_foos + 1) * sizeof(Foo));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
‘main’: events 1-2
|
| 37 | int main() {
| | ^~~~
| | |
| | (1) entry to ‘main’
|......
| 45 | add_foo_to_bar(&b, &f1);
| | ~~~~~~~~~~~~~~~~~~~~~~~
| | |
| | (2) calling ‘add_foo_to_bar’ from ‘main’
|
+--> ‘add_foo_to_bar’: events 3-5
|
| 19 | void add_foo_to_bar(struct Bar** b, Foo* f) {
| | ^~~~~~~~~~~~~~
| | |
| | (3) entry to ‘add_foo_to_bar’
| 20 | if ((*b)->foos == NULL) {
| | ~
| | |
| | (4) following ‘true’ branch...
| 21 | (*b)->foos = (Foo*)malloc(sizeof(Foo));
| | ~~~~
| | |
| | (5) ...to here
|
<------+
|
‘main’: events 6-7
|
| 45 | add_foo_to_bar(&b, &f1);
| | ^~~~~~~~~~~~~~~~~~~~~~~
| | |
| | (6) returning to ‘main’ from ‘add_foo_to_bar’
| 46 | add_foo_to_bar(&b, &f2);
| | ~~~~~~~~~~~~~~~~~~~~~~~
| | |
| | (7) calling ‘add_foo_to_bar’ from ‘main’
|
+--> ‘add_foo_to_bar’: events 8-11
|
| 19 | void add_foo_to_bar(struct Bar** b, Foo* f) {
| | ^~~~~~~~~~~~~~
| | |
| | (8) entry to ‘add_foo_to_bar’
| 20 | if ((*b)->foos == NULL) {
| | ~
| | |
| | (9) following ‘false’ branch...
|......
| 30 | (*b)->foos = (Foo*)realloc((*b)->foos, ((*b)->n_foos + 1) * sizeof(Foo));
| | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| | | |
| | | (10) ...to here
| | (11) call to ‘realloc’ here
|
No, but, runtime testing can save you.
If you can spare the execution overhead, I have seen many applications add an extra layer to memory allocation to track the allocations made and find leaks/errors.
Usually they replace malloc() and free() with a macros that include FILE and LINE
One example can be seen here (check the Heap.c and Heap.h files)
https://github.com/eclipse/paho.mqtt.c/tree/master/src
Googling "memory heap debugger" will probably turn up other examples. Or you could roll your own.
Consider the following code:
#include <stdio.h>
#include <unistd.h>
void foo(char * const arg[]) {
printf("success\n");
}
int main() {
char myargs[2][64] = { "/bin/ls", NULL };
foo(myargs);
execv(myargs[0], myargs);
return 0;
}
Both foo and execv require char * const * argument, but while my foo works (I get success in the output) the system call execv fails.
I would like to know why. Does this have something to do with the implementation of execv?
Also, assuming I have a char** variable - how can I send it to execv?
A two-dimensional array looks like this:
char myargs[2][16];
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| | | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| | | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
I reduced the size from 64 to 16 to keep the diagram from being annoyingly big.
With an initializer, it can look like this:
char myargs[2][16] = { "/bin/ls", "" }
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| /| b| i| n| /| l| s|\0| | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|\0| | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
Notice I didn't try to put a null pointer in the second row. It doesn't make sense to do that, since that's an array of chars. There's no place in it for a pointer.
The rows are contiguous in memory, so if you look at a lower level, it's actually more like this:
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| /| b| i| n| /| l| s|\0| | | | | | | | |\0| | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
When you pass myargs to a function, the famous "array decay" produces a pointer. That looks like this:
void foo(char (*arg)[16]);
...
char myargs[2][16] = { "/bin/ls", "" }
foo(myargs);
+-----------+ +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| POINTER==|===>| /| b| i| n| /| l| s|\0| | | | | | | | |\0| | | | | | | | | | | | | | | |
+-----------+ +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
The pointer is arg contains a value which locates the beginning of the array. Notice there is no pointer pointing to the second row. If foo wants to find the value in the second row, it needs to know how big the rows are so it can break down this:
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| /| b| i| n| /| l| s|\0| | | | | | | | |\0| | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
into this:
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
| /| b| i| n| /| l| s|\0| | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
|\0| | | | | | | | | | | | | | | |
+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
That's why arg must be char (*arg)[16] and not char **arg or the equivalent char *arg[].
The exec family of functions doesn't work with this data layout. It wants this:
+-----------+ +-----------+-----------+
| POINTER==|===>| POINTER | NULL |
+-----------+ +-----|-----+-----------+
|
/----------------------/
|
|
| +--+--+--+--+--+--+--+--+
\--->| /| b| i| n| /| l| s|\0|
+--+--+--+--+--+--+--+--+
And when you want to add more arguments, it wants this:
+-----------+ +-----------+-----------+- -+-----------+
| POINTER==|===>| POINTER | POINTER | ... | NULL |
+-----------+ +-----|-----+-----|-----+- -+-----------+
| |
/----------------------/ |
| |
| /--------------------------------/
| |
| |
| | +--+--+--+--+--+--+--+--+
\-+->| /| b| i| n| /| l| s|\0|
| +--+--+--+--+--+--+--+--+
|
| +--+--+--+--+--+--+
\->| /| h| o| m| e|\0|
+--+--+--+--+--+--+
If you compare this to the two-dimensional array diagram, hopefully you can understand why this can't be an implicit conversion. It actually involves moving stuff around in memory.
Both foo and execv require char * const * argument,
Yes.
but while my foo works (I get success in the output) the system call execv fails.
Getting the output you expect does not prove that your code is correct. The call exhibits undefined behavior because its argument does not match the parameter type, but it is plausible that that has little practical effect because the implementation of foo() does not use the parameter in any way. More generally, your code could, in principle, exhibit absolutely any behavior at all, because that's what "undefined" means.
I would like to know why. Does this have something to do with the implementation of execv?
From the standard's perspective, both calls exhibit equally undefined behavior. As a practical matter, however, we know that execv does use its arguments, so it would be much more surprising for that call to produce the behavior you expected than it is for the call to foo to produce the behavior you expected.
The main problem is that 2D arrays are arrays of arrays, and arrays are not pointers. Thus, your 2D array myargs does not at all have the correct type for an argument to either function.
Also, assuming I have a char** variable - how can I send it to execv?
You do not have such a variable in your code, but if you did have, you could cast it to the appropriate type:
char *some_args[] = { "/bin/ls", NULL };
execv((char * const *) some_args);
In practice, most compilers would probably accept it if you omitted the cast, too, although the standard does require it. Best would be to declare a variable that has the correct type in the first place:
char * const correct_args[] = { "/bin/ls", NULL };
execv(correct_args);
Note also that although arrays are not pointers, they are converted to pointers in most contexts -- which I use in the example code -- but only the top level. An array of arrays thus "decays" to a pointer to an array, not a pointer to a pointer.
When seeing:
void my_object_release(struct kobject *kobj)
{
struct my_object *mine = container_of(kobj, struct my_object, kobj);
/* Perform any additional cleanup on this object, then... */
kfree (mine);
}
in LWN’s The zen of kobjects, it seems incorrect in the third parameter kobj. I think it should be kobject.
The given code is correct: the third argument is the name of the container structure member to which the pointer points, not its type, so kobj is right. The example is somewhat confusing since the first kobj doesn’t correspond to the same thing as the second kobj: the first is the pointer in the caller’s scope.
Here’s a diagram to hopefully clarify the parameters of container_of:
container_of(kobj, struct my_object, kobj)
| | |
| | |
\------------+----------+--------------------------------\
| | |
| | |
/-----------------/ | |
| | |
V /-------------/ |
+------------------+ | |
| struct my_object | { | |
+------------------+ V V
+------+ +------+
struct kobject | kobj |; <-- You have a pointer to this, called | kobj |
+------+ +------+
...
};
container_of allows you to pass around a kobject pointer and find the containing object (as long as you know what the containing object is) — it allows you to use your knowledge of “what” to answer “where”.
It’s worth pointing out that container_of is a macro, which is how it can do seemingly impossible things (for developers not used to meta-programming).
In a WindowAssigner, an element gets assigned to one or more TimeWindow instances. In case of a sliding event time window, this happens in SlidingEventTimeWindows#assignWindows1.
In case of a window with size=5 and slide=1, an element with timestamp=0 gets assigned into the following windows:
Window(start=0, end=5)
Window(start=-1, end=4)
Window(start=-2, end=3)
Window(start=-3, end=2)
Window(start=-4, end=1)
In one picture:
+-> Beginning of time
|
|
+----------------------------------------------+
| size = 5 +--+ element |
| slide = 1 | |
| v |
| t=[ 0,5[ Window 1 XXXXX |
| t=[-1,4[ Window 2 XXXXX |
| t=[-2,3[ Window 3 XXXXX |
| t=[-3,2[ Window 4 XXXXX |
| t=[-4,1[ Window 5 XXXXX |
| |
| time(-4 to +4) ---- |
| 432101234 |
+---------------------------+------------------+
|
|
|
+
Is there a way to tell Flink that there is a beginning of time and before, there are no windows? If not, where to start looking to change that? In the above case, Flink should have only one window (t=[4,8[ Window 1) for the first element. Like this:
+-> Beginning of time
|
|
+-----------------------------------------------+
| size = 5 +--+ element |
| slide = 1 | |
| v |
| t=[ 0,5[ Window 1 XXXXX |
| t=[ 1,6[ Window 2 XXXXX |
| t=[ 2,7[ Window 3 XXXXX |
| t=[ 3,8[ Window 4 XXXXX |
| t=[ 4,9[ Window 5 XXXXX |
| |
| time(-4 to +8) ---- |
| 4321012345678 |
+---------------------------+-------------------+
|
|
|
+
This will have no more effect once the number of windows reaches and exceeds window size. Then, in the above case, all elements are inside of 5 Windows.
Footnotes:
org.apache.flink.streaming.api.windowing.assigners.SlidingEventTimeWindows#assignWindows
At the moment there is no way to specify the valid time interval of a Flink job. This might also be a little bit problematic given that you might want to apply your job on historic data as well.
What you could do, though, is to filter windows which start before the beginning of time out manually:
val env = StreamExecutionEnvironment.getExecutionEnvironment
env.setStreamTimeCharacteristic(TimeCharacteristic.EventTime)
val startTime = 1
val windowLength = 2
val slide = 1
val input = env.fromElements((1,1), (2,2), (3,3))
.assignAscendingTimestamps(x => x._2)
val windowed = input
.timeWindowAll(Time.milliseconds(windowLength), Time.milliseconds(slide))
.apply{ (window, iterable, collector: Collector[Int]) =>
if (window.getStart >= startTime) {
collector.collect(iterable.map(_._1).reduce(_ + _))
} else {
// discard early windows
}
}
windowed.print()
env.execute()
I might found better workaround for this issue.
The idea is to set watermark to the point in future far enough so there will be enough data for your windows. Early windows will still be there, but they will be discarded.
Here is proof of concept for AssignerWithPeriodicWatermarks[T]:
class WMG[T](wait: Long) extends AssignerWithPeriodicWatermarks[T] {
var t: Option[Long] = None
var firstTime = true
override def extractTimestamp(el: T, prevTs: Long): Long = {
t = Some(prevTs)
prevTs
}
override def getCurrentWatermark(): Watermark = (t, firstTime) match {
case (None, _) => return null
case (Some(v), false) => new Watermark(v)
case (Some(v), true) => {
firstTime = false
new Watermark(v + wait)
}
}
}
`wait` is the size of your first window.
Seems to work correctly, but I don't understand flink enough to be sure.
Update: Unfortunately, it doesn't work (now I don't know why should it), there is always few keys in keyed stream with "early windows". So in the end I'm just filtering wrong windows with something like:
val s = (winSize/winStep).intValue
kstream.flatMapWithState((in: StreamOut, state: Option[Int]) =>
state match {
case None => (Seq(), Some(1))
case Some(s) => (Seq(in), Some(s))
case Some(v) => (Seq(), Some(v+1))
})
First off, I've already found a few references which might answer my question. While I plan on reading them soon (i.e. after work), I'm still asking here in case the answer is trivial and does not require too much supplementary knowledge.
Here is the situation: I am writing a shared library (let's call it libA.so) which needs to maintain a coherent internal (as in, static variables declared in the .c file) state within the same process.
This library will be used by program P (i.e. P is compiled with -lA). If I understand everything so far, the address space for P will look something like this:
______________
| Program P |
| < |
| variables, |
| functions |
| from P |
| > |
| |
| < |
| libA: |
| variables, |
| functions |
| loaded (ie |
| *copied*) |
| from shared |
| object |
| > |
| < |
| stuff from |
| other |
| libraries |
| > |
|______________|
Now P will sometimes call dlopen("libQ.so", ...). libQ.so also uses libA.so (i.e. was compiled with -lA). Since everything happens within the same process, I need libA to somehow hold the same state whether the calls come from P or Q.
What I do not know is how this will translate in memory. Will it look like this:
______________
| Program P |
| < |
| P stuff |
| > |
| |
| < |
| libA stuff, |
| loaded by P |
| > | => A's code and variables are duplicated
| |
| < |
| libQ stuff |
| < |
| libA stuff,|
| loaded by Q|
| > |
| > |
|______________|
... or like this?
______________
| Program P |
| < |
| P stuff |
| *libA |
| *libQ |
| > |
| |
| < |
| libA stuff, |
| loaded by P |
| > | => A's code is loaded once, Q holds some sort of pointer to it
| |
| < |
| libQ stuff |
| *libA |
| > |
|______________|
In the second case, keeping a consistent state for a single process is trivial; in the first case, it will require some more effort (e.g. some shared memory segment, using the process id as the second argument to ftok()).
Of course, since I have a limited knowledge on how linking and loading works, the diagrams above may be completely wrong. For all I know, the shared libraries could be at a fixed space in memory, and every process accesses the same data and code. The behaviour could also depends on how A and/or P and/or Q were compiled. And this behaviour is probably not platform independent.
The code segment of a shared library exists in memory in a single instance per system. Yet, it can be mapped to different virtual addresses for different processes, so different processes see the same function at different addresses (that's why the code that goes to a shared library must be compiled as PIC).
Data segment of a shared library is created in one copy for each process, and initialized by whatever initial values where specified in the library.
This means that the callers of a library do not need to know if it is shared or not: all callers in one process see the same copy of the functions and the same copy of external variables defined in the library.
Different processes execute the same code, but have their individual copies of data, one copy per process.