How we can read a whole file using multiple threads if the total number of lines are not given ?
Maybe first we calculate total numbers of lines but I don't think its a good approach.
Can Someone Help ?
I am a newbie, trying to edit a program. I have a MPI Program that divide array into subsets, the master sends the subsets to the slaves, they doo a quicksort and then return the sorted numbers to the master so he can write them in a file.
What I am trying to do is make the quick sort happen even quicker. My idea is to make the master divide the array and sends subsets to the slaves but keeping one for himself. Then dividing them again to new subsets (for example if we have numbers from 1 to 100 in the array the new subsets should be from 1 to 25, 26 to 50, 51 to 75 and 76 to 100) and then keep the first subset (1 to 25) for himself, send the second (26 to 50) to the first slave, the third one (51 to 76) to the second slave and etc. The slaves should do the same. Then it should perform a quicksort and the slave should return the sorted numbers to the master. I am hoping that this way the sort should be faster. The problem is that as I said I am a newbie and I need help with ideas, advices and even code so I can achieve my goal.
For this answer I am going to stick with the assumption that this should be done with Quicksort, and that the data is read on a single process. Just keep in mind that there are many sophisticated parallel sorting techniques.
Your idea of separating the numbers by subsets is problematic, because it makes assumptions about the shape of data. For non-uniformly distributed data sets it won't even help to know the minimum and maximum. It is better to simply send out equal amount of numbers to each process, let them sort and afterwards merge the data.
For the merge you start with ntasks sorted sub-lists and want to end up with a single one. A naive merge would repeatedly look for the minimal element in each sub-list, remove that and append it to the final list. This needs ntasks * N comparisons, N swaps and N * 2 memory. You can optimize the comparisons to log2(ntasks) * N by doing an actual merge sort, but that also needs log2(ntasks) * N swaps. You can further refine that by keeping the sub-lists (or pointers to their first element) in a priority queue, which should give you log2(ntasks) * N comparisons and N swaps.
About the usage of MPI:
Do not use MPI_Isend & MPI_Wait right after each other. In this case use MPI_Send instead. Use the immediate variants only if you can actually do something useful between the MPI_Isend and MPI_Wait.
Use collective operations whenever possible. To distribute data from the root to all slaves, use MPI_Scatter or MPI_Scatterv. The first requires all ranks to receive the same number of elements, which can also be achieved by padding. To collect data from the slaves in the master, use MPI_Gather or MPI_Gatherv.1 Collectives are more easy to get right, because they describe the high level operation. Their implementation is usually highly optimized.
To receive an unknown-size message, you can also send the message directly and use MPI_Probe at the receiver side to determine the size. You are even allowed to MPI_Recv with a buffer that is larger than the sent buffer, if you know an upper bound.
1 You could also consider the merge step as a reduction and parallelize the necessary computation for that.
In principle your solution looks very good. I don't understand completely if for the larger files you are intending to process them in chunks or as a whole. From my experience I suggest that you assign as large as possible blocks to the slaves. This way the rather expensive message passing operations are executed only very seldom.
What I cannot understand in your question is what the overall goal of your program is. Is it your intention to sort the complete input files in parallel? If this is the case you will need some sort of merge sort to be applied to the results you receive from the individual processes.
In a simulation program I am trying to print the measures to a text file. The project is a combination of Java, C and C++ but the file I am working with is in C. The code for printing is as follows:
if(sample)
fprintf(MeasureInfo->measuresFile, "%d: %f\n", count++, sample);
This works for part of the output but there are large bulks (about 100 to 1000 measures) of data that are not printed to the text file. Instead I see just a bulk of NULs in Sublime Text and 0-bytes in bless:
436: 0.851661
437: 0.043466
(Really large block of NUL all in one line).210402
751: 0.357543
752: 0.816120
I only worked with part of the code so far and thought it might be a concurrency problem. So I printed out all pids that access the function with getpid() and it gave me different ones (19036, 19037, 19038 for instance). I then tried to use pthread_mutex_lock and pthread_mutex_unlock but it produced the same output.
Another thing I tried was using sleep after every 400 measures. This actually helped but shortened the number of measures the produced by a fourth.
Do you have any idea what the actual problem might be and how to fix it? I am really sorry if this is an answered or easy question but I tried and searched a while and didn't find a solution to this.
I/O and multitasking are a dangerous combination. At the thread level, fprintf is writing into a buffer, in thread-unsafe fashion. The individual formatting components might overwrite each other, and when one call flushes the buffer while another is trying to write, the result is certain to be lost data. It could even cause a buffer overflow and crash right inside to stdio library.
At the process level, the buffers are getting flushed to particular locations in the file. If two processes try to write at the same time, they will clobber each others' results. A process typically doesn't poll the filesystem to check if a file is growing due to an outside influence, then seek to the new end before writing.
If you have a very large project and synchronizing all the I/O to this file isn't an option, consider assigning each thread or process its own file, and merge the files after the fact.
sample looks to be a float or double. You shouldn't compare a floating point variable against a fixed value, as it most likely will never have this value. For floating point variable you should always compare against a delta. Your if clause will probably always be entered.
fprintf shouldn't print NUL though. Can you show the exact output (or part of it)?
I am writing code that involved a for loop that makes calculations at each index.
The smallest of these calculations is stored in a variable and I use MPI_Allreduce at the end of the program to determine the global minimum across all processes.
However, I need a way of knowing which process has the smallest value, i.e. can MPI_Allreduce tell me which process sends the result? the process with the smallest value? there is some additional data I need to get from that process.
Thanks in advance for any help!
You can use the MPI_MINLOC operator in the reduce operation to receive the rank of the process with the minimal value (more specifically, the lowest ranked process that has a minimal value).
See http://www.netlib.org/utk/papers/mpi-book/node114.html#SECTION005103000000000000000
So, this seemed simple at first, but after crawling Google and here, the answer doesn't seem as simple as I first thought.
Basically, I'm editing a MINIX kernel as part of a practical for my Operating Systems course, and I have to add a little function that spits out the number of running processes when you hit a function key in the Information Server. I've figured out how to integrate the functionality so all the other stuff works, but for the life of me, I can not figure out how to get the current number of processes running in the system into my C code and into a variable to print out.
First I thought there'd be a nifty Syscall like SYS_NUMPROCS or something that'd return the value, but no luck.
Then, I tried piping output from a system("ps -ax | wc -l") to a file and the file wouldn't create. I tried using popen() and no luck there either - even with a simple "ls" read into a buffer, it just bombs the code and "hangs" the run of the code, so there's no output.
So now I'm truly stumped, and any help would be super awesome, because at this point I've exhausted all the obvious options.
The only two things I can think of now would be a loop counting all the processes, but first you have to get to the system's process list, and I've heard vague things said about /proc/ as a directory, but I haven't a clue how to access/run through that or how it'd link up to getting the number of processes in the first place.
Thanks a stack (lol pun), guys :)
Also, I haven't included code explicitly because nothing I've written aside from basic printf'ing for cosmetic output, because none of the things I've tried gave me any joy :/
Edit notes: Guys, this is a kernel edit - I'm writing the function to printf the information in a system C file, then recompiling the kernel and rebooting the system to test. It's a UNIX (MINIX) kernel, not a Linux kernel, and it's not a user mode program.
My code for popen(), as some of you requested, is as follows:
public void cos_dmp(){
char buffer[512];
FILE * f;
f = popen("ps -ax | wc -l","r");
fgets(buffer, sizeof(buffer),f);
//buffer should now contain result of popen()
printf(buffer);
}
That's a bit of a hacked together version from what I remember and keeping it ultra simple and showing you guys that's what I was trying to do. Again though, there must be a better way to do this aside from essentially calling the output of a system() call.
Edit again: the above code woks perfectly from a user program but won't work from the kernel function. Anybody have an idea why?:/
struct kinfo kinfo;
int nr_tasks, nr_procs;
getsysinfo(PM_PROC_NR, SI_KINFO, &kinfo);
nr_procs = kinfo.nr_pro;
This will get you the number of processes running
try looking to see what ps does. Look at its source code; it knows how many processes there are
Perhaps you could show us the code your wrote for capturing the result of system("ps -ax | wc -l"), or the code you wrote to use popen and we could help you diagnose the problem with it.
Regardless, the most efficient way I can think of to count the number of existing (not the same as running) processes on the system is to opendir("/proc") and count the number of entries that are strings of decimal digits. Each process in the system will be represented by a subdirectory of /proc, named after the decimal process id number of that process.
So, if you find "/proc/3432", for example, then you know that there exists a process with a pid of "3432". Simply count the number of subdirectories you find whose names are decimal numbers.
Assumptions:
You are asking about Linux, not MINIX.
You are writing a user-mode program, not modifiying the kernel.
So I have been having the same problem and found a solution. (MINIX 3.1) within the method to count the processes use this code:
(This is ANSI C)
It simply runs through the process table and counts the number of processes.
I know this is an old thread but it might help someone in the future.
#include "../pm/mproc.h"
/* inside function */
struct mproc *mp;
int i, n=0;
printf("Number of running processes:\n");
getsysinfo(PM_PROC_NR, SI_PROC_TAB, mproc);
for (i = 0; i<NR_PROCS; i++) {
mp = &mprocs[i];
if (mp->mp_pid == 0 && i != PM_PROCS_NR) continue;
n++;
}
printf("%d", n);
/* function end */
I had the same assignment at my university so i will post my solution if someone needs it in future. I am using Minix 3.3 and VMware player for virtual machines.
In pm server at location /usr/src/minix/servers/pm there is glo.h file which contains various global variables used by pm server. In that file, there is fortunately one variable called procs_in_use defined as EXTERN int procs_in_use;
So simple printf("%d\n",procs_in_use); from a system call will show the number of current processes running. You can test this by adding fork() function in your user space program in the middle of a loop.
One more mention : first answer that says
struct kinfo kinfo;
int nr_tasks, nr_procs;
getsysinfo(PM_PROC_NR, SI_KINFO, &kinfo);
nr_procs = kinfo.nr_procs;
didn't work out for me. SI_KINFO no more exists so you should use SI_PROC_TABLE. Also there can be problems with permissions, so you will not be able to call this function from your regular system call. There is alternative function sys_getkinfo(&kinfo) that can be called from your fresh system call and that will do the same as the above. The problem is kinfo.nr_procs will not return number of current processes but number of maximum user processes that can be in operative system which is 256 by default, and can be changed manually in file where NR_PROCS is defined. On the other hand kinfo.nr_taskswill return maximum number of kernel processes that can be held by operative system, which is 5 by default.
Check this out: http://procps.sourceforge.net/
It's got source to a number of small utilities that do these kinds of things. It'll be a good learning experience :) and I think PS is i n there as pm100 noted.
If you're editing the kernel, the most efficient way to solve this problem is to maintain a count every time a process (i.e., a task_struct entry) is created (and make sure you decrement the count every where a process terminates).
You could always loop through the list of processes in the kernel using the built-in macro (but its expensive, so you should try to avoid it):
struct task_struct *p;
unsigned int count = 0;
for_each_process(task) {
count++;
}
Check this out: http://sourceforge.net/p/readproc/code/ci/master/tree/
#include"read_proc.h"
int main(void)
{
struct Root * root=read_proc();
printf("%lu\n",root->len);
return 0;
}