I have an R function which is essentially acting as a wrapper to a set of C functions - the R code calls the C code through .C("..."). This C code could be parallelized and compiled using some MPI implementation. However, having never used MPI before, I have no idea if such MPI-ed code would even be callable from R in a way that would let MPI work?
Does anyone have any experience with this kind of thing? I'm guessing the R MPI libraries are pointless for my purpose, given all the work is done deep within the C code. This would ultimate be run on HPC cluster, if that makes any difference?
Can you use mpicc to create a shared objected, and if such a shared object was called from R would a parallel implementation run, or would you just get the serial version (or indeed, as I suspect, would it just crash)?
I may well be missing info needed to understand the problem, so will update accordingly.
You seem confused.
You could just use the existing Rmpi package to spawn parallel execution of several R instances (on your different nodes) and each of those can use your .C()-called code as well.
Maybe you want to work through some simple examples to get a better feel about what can or cannot be done?
Related
I am currently developing an R package where most of the computational work is done in C code. In particular, I have several complex C routines for which I then wrap with an R function. These I know I can test using testthat because the namespace includes the C function called (I am using roxygen comments to generate the namespace).
The difficulty I am having is that some of these C routines have a few complicated subroutines, the output of which is non-trivial to check. These functions are never called from R directly, so there is no code in my R/ directory that has a roxygen comment that results in the namespace making these functions available. What I want to do is write an R routine that performs the same computation and then use testthat to compare against the output of the C function. I have done this and it works great when I load the entire shared library (using dyn.load), but I don't think this is the proper approach for tests in a package that I intend to submit to CRAN.
My current workaround is to create a dummy function in a file in my R/ directory that then has a roxygen comment for each of the functions I wish to load. That way when I run devtools::document(), these functions are included in the namespace.
Is this the best approach, or is there another method that is better/preferred?
Is Matlab Coder only able to produce single threaded applications?
I attempted to convert some Matlab scripts, used for image analysis, and found that the code produced by Matlab Coder was much slower. This confused me because I thought the produced C code would be at-least as fast or faster then the M code. I then checked how many threads were being used for both the M code and the produced C code. The result was 1 thread used by the C code and there were many threads being used by the Matlab code. At this point I can only assume the image processing toolkit implements its function as mex functions which are multi-threaded.
While in general the generated code can be expected to be faster, there are some exceptions. Some implementations which are used by matlab are not available for generated code. I have no reference about the technical background, but I assume that these are fortran and/or assembler written libraries. An example for such a function is eig which is known to produce different (correct) results in generated code.
The matlab coder comes with a code examples which explains how a parfor is translated to openmp code. As a first step make sure that your code contains the openmp relevant pragmas. If not try rewriting your code using a parfor loop.
In a last step, make sure that your compiler is configured to use openmp.
I'm looking for a way to do very simple profiling in a mex program triggered from matlab. I compile from matlab using:
mex -O CFLAGS="\$CFLAGS -std=c99" rrt.c and then run my program.
Really all I need is a thing to see, which of two functions runs faster. However since it all goes down in about 1/100s time(NULL) is not fast enough.
Is there a simple function in C I could call, or are there any real profiling methods for a mex program in matlab?
I saw this post beeing treated as duplicate, but what I want to know is a way to profile the C code compiled with gcc in matlab, or easier some timing functions.
I use OSX 10.7.5 and matlab 2014b. Thanks for any hints.
Edit: Actually chappjc's hint got me looking for clock(), which does, what I need for the time beeing. An actual profiling would still be nice though.
The reason not to use tic/toc or similar is, that I have a base and a modified code, which both run with random samples. Compiling 2 versions of basically the same code each time I change something and having the extra step of exporting/importing the seed for the random number generator seems like a big hustle for exactly no value to me. I write code such that I don't have to repeat myselft. Having two seperate functions would need quite some duplicate code, since the changes are easy and a few, but deeply integrated in not just one spot.
How does one call Go code in C from threads that weren't created by Go?
What do I assign to a C function pointer such that threads not created by Go can call that pointer and enter into Go code?
Update0
I don't want to use SWIG.
The callbacks will be coming from threads Go hasn't seen before. Neither cgo/life nor anything in pkg/runtime demonstrates this behaviour AFAICT.
You can do this, but the solution is relatively slow (about 22µs per call on my machine).
The answer is for the C code to use C thread primitives to communicate with another goroutine that will actually run the callback.
I have created a Go package that provides this functionality: rog-go.googlecode.com/hg/exp/callback.
There is an example package demonstrating its use here. The example demonstrates a call back to an arbitrary Go closure from a thread created outside of the Go runtime. Another example is here. This demonstrates a typical C callback interface and layers a Go callback on top of it.
To try out the first example:
goinstall rog-go.googlecode.com/hg/exp/example/looper
cd $GOROOT/src/pkg/rog-go.googlecode.com/hg/exp/example/looper
gotest
To try out the second example:
goinstall rog-go.googlecode.com/hg/exp/example/event
cd $GOROOT/src/pkg/rog-go.googlecode.com/hg/exp/example/event
gotest
Both examples assume that pthreads are available. Of course, this is just a stop-gap measure until cgo is fixed, but the technique for calling arbitrary Go closures in a C callback will be applicable even then.
Here is the documentation for the callback package:
PACKAGE
package callback
import "rog-go.googlecode.com/hg/exp/callback"
VARIABLES
var Func = callbackFunc
Func holds a pointer to the C callback function.
When called, it calls the provided function f in a
a Go context with the given argument.
It can be used by first converting it to a function pointer
and then calling from C.
Here is an example that sets up the callback function:
//static void (*callback)(void (*f)(void*), void *arg);
//void setCallback(void *c){
// callback = c;
//}
import "C"
import "rog-go.googlecode.com/hg/exp/callback"
func init() {
C.setCallback(callback.Func)
}
I'll assume you mean from C code compiled with gcc?
IIRC, this either can't be done or can't easily be done using 6g+cgo and friends. Go uses a different calling convention (as well as the segmented stacks and such).
However, you can write C code for [685]c (or even [685]a) and call into go easily using package·function() (you can even call methods IIRC). See the Source of the runtime package for examples.
Update:
Coming back to this question after the update, and giving it some more thought. This can't be done in a standard fashion using 6c or cgo. Especially because the threads are not started by the go runtime, the current implementation would fail. The scheduler would suddenly have a thread under its control that it does not know about; additionally, that thread would be missing some thread-local variables the go runtime uses for managing stacks and some other things. Also, if the go function returns a value (or several) the C code can't access it on the currently supported platforms, as go returns values on the stack (you could access them with assembly though). With these things in mind, I do believe you could still do this using channels. It would require your C code to be a little too intimate with the inner workings of the go runtime, but it would work for a given implementation. While using channels may not be the solution you're looking for, it could possibly fit more nicely with the concepts of Go than callbacks. If your C code reimplemented at least the sending methods in The channel implementation (that code is written for 6c, so it would have to be adapted for gcc most likely, and it calls the go runtime, which we've determined can't be done from a non-go thread), you should be able to lock the channel and push a value to it. The go scheduler can continue to manage it's own threads, but now it can receive data from other threads started in C.
Admittedly, it's a hack; I haven't looked close enough, but it would probably take a few other hacks to get it working (I believe the channels themselves maintain a list of the goroutines that are waiting on them [EDIT: confirmed: runtime·ready(gp);], so you'd need something in your go code to wake up the receiving channel or to warranty the go code won't receive on the channel until you've already pushed a value). However, I can't see any reason this can't work, whereas there are definite reasons that running code generated by 6g on a thread created in C can't.
My original answer still holds though: barring an addition to the language or runtime, this can't yet be done the way you'd like (I'd love to be proven wrong here).
You can find a real-world application of rog's callback package in these bindings for the PortAudio audio I/O library: http://code.google.com/p/portaudio-go/. Might make it easier to understand..
(Thanks for implementing that, rog. It's just what I needed!)
Hi I wanted to know how to use MATLAB as an external solver from a C program. Specifically in my code I wish
to solve several linear systems of the form Ax=b.
I have heard that to go the other way namely Calling C functions in a MATLAB routine one uses MEX files.But I am not really sure how to use Mex files either.
Thank you
Actually, MEX files allow you to include C code in Matlab programs, for example if you want to use external C libraries in Matlab.
What you want to do is use the Matlab Engine:
http://www.mathworks.com/help/techdoc/matlab_external/f29148.html
As an alternative, you could use linear algebra libraries that are written purely in C, such as LAPACK and BLAS. ( www.netlib.org )
you can use the matlab engine as Lagerbaer points out. However sometimes it can be convenient just calling the matlab process commandline style. I use this often when I don't want to mess with mxArrays etc, or when the amount of matlab code that needs executing gets really large. PseudoCode:
WriteArrayInMFormat( "in.m", myInputNumbers );
LaunchProcess( "matlab", "-nodesktop -r \"myFunction( 'in.m' )\" -logfile out.m" );
ReadArrayInMFormat( "out.m", myResult );
For me this is especially useful when testing things out: instead of having to recompile the C/C++ program each time I change something, I just apply all changes in the myFunction.m file. At that point I don't even need the C program, instead everything can be tested in matlab.