I have this code:
#define SHMSIZE 8388606
int main()
{
int shmid;
void *shmPtr;
char *shm;
if ((shmid = shmget(IPC_PRIVATE, sizeof(char) * SHMSIZE , IPC_CREAT | 0666)) < 0) {
perror("shmget");
exit(1);
}
if ((shmPtr = shmat(shmid, NULL, 0)) == (char *) -1) {
perror("shmat");
exit(1);
}
shm = (char *)shmPtr;
strncpy(shm, "0\n", 2);
struct shmid_ds shmid_ds;
int rtrn = shmctl(shmid, SHM_LOCK, &shmid_ds);
if(rtrn < 0) {
perror("shmctl");
exit(1);
}
else {
printf("Nailed it\n" );
}
return 0;
}
Running it, I get the error:
shmctl: Cannot allocate memory
Defining smaller value for SHMSIZE fixes the error, but what I found weird is that this error is thrown in the "shmctl" part. My reasoning tells me that this error is supposed to be thrown in the "shmget" part.
How does this code runs successfully through the shmget() call? Did I miss something important?
Read this, It may help you to solve your problem:
The caller can prevent or allow swapping of a shared memory segment
with the following cmd values:
SHM_LOCK (Linux-specific)
Prevent swapping of the shared memory segment. The caller
must fault in any pages that are required to be present
after locking is enabled. If a segment has been locked,
then the (nonstandard) SHM_LOCKED flag of the shm_perm.mode
field in the associated data structure retrieved by
IPC_STAT will be set.
SHM_UNLOCK (Linux-specific)
Unlock the segment, allowing it to be swapped out.
In kernels before 2.6.10, only a privileged process could employ
SHM_LOCK and SHM_UNLOCK. Since kernel 2.6.10, an unprivileged
process can employ these operations if its effective UID matches the
owner or creator UID of the segment, and (for SHM_LOCK) the amount of
memory to be locked falls within the RLIMIT_MEMLOCK resource limit
(see setrlimit(2)).
Try this:
int rtrn = shmctl(shmid, IPC_STAT, &shmid_ds);
Related
int shmCreate(int id, int size)
{
int shmid
key_t key;
if ((key = ftok(".", id)) == -1) {
perror("ftok");
exit(1);
}
if ((shmid = shmget(key, size, IPC_CREAT | 0666)) == -1) {
perror("shmget");
exit(1);
}
return shmid;
}
I'm trying to write function which use shmget, where function gets size as parameter. I'm getting Invalid argument error all the time, except for size = 1. When I pass size other than 1 i got error, but the memory segment is created when i check ipcs -m, and it has size of 1. How do I get rid of this error?
According to the man page for shmget(2):
EINVAL A segment for the given key exists, but size is greater than
the size of that segment.
You cannot attach a segment that already exists with a larger size. So you need to remove it first or use a different key.
I'm using shared memory (shm_open/mmap) to keep track of some state. In my shared memory I have the struct:
typedef struct fim_t {
uint64_t num_procs;
uint64_t num_numa;
int64_t *numa_nodes[MAX_FIM_NUMA];
int64_t procs[MAX_FIM_PROC];
}fim_t;
What I want to do is load process IDs in the procs array and then have the numa_nodes array point to procs array values so I can manipulate the value in one spot and have it change across all the references. My understanding is that setting the numa_nodes references to addresses of the procs array should not be a memory access violation because their addresses are both entirely within the shared memory segment. However I get a seg fault when I try to access the value which tells me that my previous statement must be false.
Here is example code:
int main(){
int fd;
int init_flag = 0;
if((fd = shm_open("fim", O_RDWR | O_CREAT | O_EXCL, S_IRWXU)) > 0){
printf("creating shared memory\n");
init_flag = 1;
} else {
printf("opening shared memory\n");
fd = shm_open("fim", O_RDWR, S_IRWXU);
}
if (-1 == fd) {
printf("fd is negative\n");
abort();
}
if ((1 == init_flag) && -1 == ftruncate(fd, sizeof(fim_t))){
printf("ftruncate failed %d\n", errno);
abort();
}
fim_t *fim = mmap(NULL, sizeof(fim_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
if(MAP_FAILED == fim){
printf("mmap failed\n");
abort();
}
if(init_flag){
fim->num_procs = 1;
my_rank = 0;
for(int x=0;x<MAX_FIM_PROC;x++){
fim->procs[x] = 0;
}
fim->numa_nodes[0] = &(fim->procs[0]);
} else {
my_rank = __sync_fetch_and_add(&(fim->num_procs),1);
fim->procs[my_rank] = my_rank;
fim->numa_nodes[0] = &(fim->procs[my_rank]);
}
printf("my rank is: %"PRId64"\n",my_rank);
sleep(5);
printf("my numa val is %"PRId64"\n",*fim->numa_nodes[0]);
printf("rank %"PRId64" is going down\n", my_rank);
// SHUTDOWN
uint64_t active = __sync_sub_and_fetch(&(fim->num_procs),1);
printf("num active is now %"PRId64"\n", active);
close(fd);
shm_unlink("fim");
return 0;
}
What I expect/hope to happen would be that I run one process then immediately start another and the first process prints "my numa val is 1" (due to the second process setting the numa_node[0] value) and both exit cleanly. However, the second process runs fine, but in the first process seg faults (memory access) at the print statement for numa_node[0] (after the sleep).
So here's my question: Am I doing something wrong or is my approach unworkable? If it is unworkable, is there another way to achieve the result I'm looking for?
You haven't done anything to arrange for all users of the shared memory to map it at the same virtual address. Some *nix systems will do this by default, but most will not.
Either try to map your segment at a fixed address (and deal with failure - this may not succeed) - or store offsets in the shared memory, not actual pointers.
My understanding is that setting the numa_nodes references to addresses of the procs array should not be a memory access violation because their addresses are both entirely within the shared memory segment.
The problem is that different processes map the shared memory to different addresses.
fim_t *fim = mmap(NULL, sizeof(fim_t), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0);
fim will have different values in different processes. Print it out to check this.
This causes the pointers to the int64_t procs[MAX_FIM_PROC] elements to be different in different processes.
fim is <addr1> in process 1
fim is <addr2> in process 2
&fim->procs[0] will be different in two processes
&fim->procs[0] is <addr1> + <offset> in process 1
&fim->procs[0] is <addr2> + <offset> in process 2
Because these are different values, they cannot be shared between processes. Valid pointer in one process will not be valid in another process.
There are two possible solutions to this.
Force the shared memory to map to the same address in all processes. mmap has an option to accomplish this. Then you can share pointers to elements in the shared memory across processes.
Do not share pointers in shared memory. Share indexes instead.
I have a simple program that writes to a shared memory and reads from a shared memory, but while reading I am getting segmentation fault.
When I am debugging, the child process is not writing information to the shared memory, and after that, the parent process is trying to read from the shared memory which has no data and is throwing segmentation fault at 1st printf, in parent printf("%d\n",ptr->nread);
Why the child process is not able to write data to the shared memory? (it is failing at ptr->nread=20; line)
#include<sys/types.h>
#include<sys/ipc.h>
#include<sys/shm.h>
#define SIZE 5*1024
struct databuf{
int nread;
char *buf;
int xyz;
};
struct databuf* ptr;
main()
{
int shmid,pid;
shmid = shmget((key_t)1,SIZE,IPC_CREAT|0777);
pid = fork();
if(pid==0)
{
ptr = (struct databuf *)shmat(shmid,(char*)0,0);
ptr->nread=20;
ptr->buf=ptr+sizeof(ptr->nread);
strcpy(ptr->buf, "abc");
ptr->xyz=20;
}
else
{
wait(0);
ptr = (struct databuf *)shmat(shmid,(char*)0,0);
printf("%d\n",ptr->nread);
printf("%s\n",ptr->buf);
printf("%d\n",ptr->xyz);
}
return 0;
}
If ptr->nread is failing then you should put a error checking code something like this, before acessing the ptr.
ptr = (struct databuf *)shmat(shmid,(char*)0,0);
if (data == (struct databuf *)(-1)) {
perror("shmat failed");
exit(1);
}
ptr->nread=20;
Ref: http://linux.die.net/man/2/shmat
ptr->buf=ptr+sizeof(ptr->nread);
could be written as:
ptr->buf=(char*)ptr+sizeof(struct databuf)+ptr->nread;
or
ptr->buf=(char*)ptr+ptr->nread;
The string can now be accessed in the parent process.
Brief Explanation:
If you're using shared memory, you have to make sure all the data you want to access in other processes is in the shared memory segment. Leave the data at a specified offset(in your case ptr+ptr->nread) in the memory segment. And be careful not to overwrite the existing data in the shared memory. sizeof(ptr->nread) will yield the sizeof(int).
Leaving to one side all the other issues with the code, I think:
shmid = shmget((key_t)1, SIZE, IPC_CREAT|0777) ;
is probably a mistake, unless you can (somehow) guarantee that (key_t)1 is not in use elsewhere. For parent-child communication, as in this case, the simpler approach is:
shmid = shmget((IPC_PRIVATE, SIZE, IPC_CREAT|0777) ;
In general, when something apparently inexplicable happens, I find it helps to make sure I have checked for error returns. In this case: shmget() returning -1 or shmat() returning -1... and I suspect that both have happened.
Worked solution added by #WhozCraig:
The following example works, and likely does what you're trying to accomplish. Pay note to how we calculate the address we store in ptr->buf after we home ptr to our shared memory base address. We have to leave room for the structure, so we calculate the address to start the first byte past the structure back-side.
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include<sys/types.h>
#include<sys/ipc.h>
#include<sys/shm.h>
struct databuf
{
int nread;
char *buf;
int xyz;
};
#define SIZE (5*1024)
int main()
{
// acquire shared memory first (read/write access)
int shmid = shmget(IPC_PRIVATE, SIZE, IPC_CREAT|0666);
if (shmid == -1)
{
perror("Failed to acquire shared emory.");
exit(EXIT_FAILURE);
}
// fork child process
pid_t pid = fork();
// both parent and child need this. may as well do both before
// special casing child vs. parent logic.
struct databuf *ptr = shmat(shmid,(char*)0,0);
if (ptr == (void*)(-1))
{
perror("Failed to map shared memory to our process");
exit(EXIT_FAILURE);
}
// child process
if (pid==0)
{
ptr->nread = 20;
ptr->buf = ((char*)ptr) + sizeof(*ptr);
strcpy(ptr->buf, "abc");
ptr->xyz = 30;
}
// parent process
else
{
wait(NULL);
printf("ptr = %p, ptr->buf = %p\n", ptr, ptr->buf);
printf("%d\n",ptr->nread);
printf("%s\n",ptr->buf);
printf("%d\n",ptr->xyz);
}
return 0;
}
Sample Output (varies by system obviously)
ptr = 0x80000, ptr->buf = 0x8000c
20
abc
30
it is worth noting that this:
ptr->buf = ((char*)ptr) + sizeof(*ptr);
could be written as the following, using typed-pointer math:
ptr->buf = (char*)(ptr + 1);
The location where the cast is applied is important. The first applies it before we do any math, so we need to account for octet count. The second applies it after the pointer math, so simply adding one to to the typed pointer ptr will move us to the memory immediately past our ptr structure base.
Best of luck.
If you have a pointer from type T, an addition by one to the pointer will increment it by sizeof(T).
So you have to replace:
ptr->buf=ptr+sizeof(ptr->nread);
to
ptr->buf= ((char*)ptr )+sizeof(ptr->nread);
If you don't do that, your pointer will be incremented by sizeof(T)^2.
While trying to resolve some debugging issues , I added some printf-s to my code :
I used that code :
struct PipeShm
{
int init;
sem_t sema;
...
...
}
struct PipeShm * sharedPipe = NULL;
func2:
int func2()
{
if (!sharedPipe)
{
int myFd = shm_open ("/myregion", O_CREAT | O_TRUNC | O_RDWR, 0666);
if (myFd == -1)
error_out ("shm_open");
// allocate some memory in the region in the size of the struct
int retAlloc = ftruncate (myFd, sizeof * sharedPipe);
if (retAlloc < 0) // check if allocation failed
error_out("ftruncate");
// map the region and shared in with all the processes
sharedPipe = mmap (NULL, sizeof * sharedPipe,PROT_READ | PROT_WRITE,MAP_SHARED , myFd, 0);
if (sharedPipe == MAP_FAILED) // check if the allocation failed
error_out("mmap");
// put initial value
int value = -10;
// get the value of the semaphore
sem_getvalue(&sharedPipe->semaphore, &value);
if (sharedPipe->init != TRUE) // get in here only if init is NOT TRUE !
{
if (!sem_init (&sharedPipe->semaphore, 1, 1)) // initialize the semaphore to 0
{
sharedPipe->init = TRUE;
sharedPipe->flag = FALSE;
sharedPipe->ptr1 = NULL;
sharedPipe->ptr2 = NULL;
sharedPipe->status1 = -10;
sharedPipe->status2 = -10;
sharedPipe->semaphoreFlag = FALSE;
sharedPipe->currentPipeIndex = 0;
printf("\nI'm inside the critical section! my init is: %d\n" , sharedPipe->init);
}
else
perror ("shm_pipe_init");
printf("\nI'm out the critical section! my init is: %d\n" , sharedPipe->init);
}
}
return 1; // always successful
}
With that main :
int main()
{
int spd, pid, rb;
char buff[4096];
fork();
func2();
return 0;
}
And got this :
shm_pipe_mkfifo: File exists
I'm inside the critical section! my init is: 1
I'm out the critical section! my init is: 1
Output:hello world!
I'm inside the critical section! my init is: 1
I'm out the critical section! my init is: 1
It seems that the shared memory is not so shared , why ?
The segment is shared between all processes due to MAP_SHARED | MAP_ANONYMOUS , so why both processes have the same before and after values ?
It seems that each process has its own semaphore even though it was shared between them , so what went wrong ?
Thanks
Since you use the MAP_ANONYMOUS flag to mmap, the myFd argument is ignored, and you create two independent shared memory chunks, one in each process, which have no relation to each other.
MAP_ANONYMOUS
The mapping is not backed by any file; its contents are initial‐
ized to zero. The fd and offset arguments are ignored; however,
some implementations require fd to be -1 if MAP_ANONYMOUS (or
MAP_ANON) is specified, and portable applications should ensure
this. The use of MAP_ANONYMOUS in conjunction with MAP_SHARED
is only supported on Linux since kernel 2.4.
If you get rid of MAP_ANONYMOUS you'll then only have one shared memory chunk, but you then have the problem of not calling sem_init. On Linux with NPTL it will actually work, as clearing a sem_t to all 0 bytes (the initial state here) is equivalent to sem_init(&sema, anything, 0); (NPTL ignores the pshared flag), but that's not portable to other systems.
Per Karoly's comment on another answer, there's also a race condition due O_TRUNC on the open call. If the second thread calls open after the first thread has already started modifying the semaphore, that TRUNC will clobber the semaphore state. Probably the best solution is to move the code creating, opening, and mmaping the shared memory to a different function that is called BEFORE calling fork.
edit
To fix the O_TRUNC problem, you can't have more than one process calling shm_open with O_TRUNC. But if you just get rid of the O_TRUNC, then you have the startup problem that if the shared memory object already exists (from a previous run of the program), it may not be in a predictable state. On possibility is to split off the beginning of func2:
main() {
func1();
fork();
func2();
}
func1() {
int myFd = shm_open ("/myregion", O_CREAT | O_TRUNC | O_RDWR, 0666);
if (myFd == -1)
error_out ("shm_open");
// allocate some memory in the region in the size of the struct
int retAlloc = ftruncate (myFd, sizeof *sharedPipe);
if (retAlloc < 0) // check if allocation failed
error_out("ftruncate");
// map the region and shared in with all the processes
sharedPipe = mmap (NULL, sizeof *sharedPipe, PROT_READ|PROT_WRITE, MAP_SHARED, myFd, 0);
if (sharedPipe == MAP_FAILED) // check if the allocation failed
error_out("mmap");
}
func2() {
// put initial value
int value = -10;
// get the value of the semaphore
sem_getvalue(&sharedPipe->semaphore, &value);
:
Alternately you could keep the same code (just get rid of O_TRUNC) and add a cleanup before the fork:
main() {
shm_unlink("/myregion");
fork();
func2();
In all cases you'll still have problem if you run multiple copies of your program at the same time.
A few thoughts...
I think this is a fundemental misunderstanding of how POSIX semaphores work. I don't see a call to sem_init or sem_open. You shouldn't be able to use them across processes without doing so much more explicitly than you've done.
I'm not so fresh on the implementation of mmap on Linux and how MAP_ANONYMOUS might affect this, but in general writes to mapped regions can't really be instantaneous. The manpage on linux.die says:
MAP_SHARED
Share this mapping. Updates to the mapping are visible to other processes that map this file, and are carried through to the underlying file. The file may not actually be updated until msync(2) or munmap() is called.
The reason for this is that your memory access gets trapped in a page fault and at that point the kernel will fill contents from the file descriptor, then let you do your write in RAM, then at some later point the kernel will flush back to the file descriptor.
I have an issue attempting to access shared memory using mmap for complex types.
So I allocate my memory as so in my parent process:
/* Create mmap file */
fid = open(TMP_FILE_NAME, O_RDWR | O_CREAT | O_EXCL, (mode_t) 0755);
if (fid < 0)
{
printf("Bad Open of mmap file <%s>\n", TMP_FILE_NAME);
die(-1);
}
/* Make mmap file Buffer Size */
status = ftruncate(fid, INPUT_BUFFER_SIZE);
if (status)
{
printf("Failed to ftruncate the file <%s>, status = %d\n", TMP_FILE_NAME, status);
die(-1);
}
/* Initialize Shared Memory */
mmap_ptr = mmap((caddr_t) 0,
INPUT_BUFFER_SIZE, // Default Buffer Size
PROT_WRITE | PROT_READ, // R/W Permissions
MAP_SHARED, // No file backing
fid,
(off_t) 0);
if (mmap_ptr == MAP_FAILED)
{
printf("Failed to perform mmap, Exiting\n");
die(-1);
}
Now the Struct that I'm passing in memory to my child process is as follows:
/* Data structue for IPC */
typedef struct {
int current_active_id;
int consume_remaining;
Queue buffer;
} input_buffer;
where Queue is a data structure class from the following:
http://www.idevelopment.info/data/Programming/data_structures/c/Queue/Queue.shtml
In my child process it's okay when I do this, it returns the correct value:
printf("Got here... Shared Mem: %d\n", input_queue->consume_remaining);
but when I do something like:
IsEmpty(input_queue->buffer)
it crashes and in the code of the Queue it's only doing this:
return Q->Size == 0;
Any help would be appreciated, thanks!!
Queue is a pointer to struct QueueRecord, and should be allocated as such, presumably using the same shared memory segment. note that this should also be mapped at the same address in both parent and child, or you will not be able to dereference it.
The structure you are putting in the map contains pointers. The pointers are all relative to the address space of the process that created them.
If the other process doesn't mmap at the same address, or if it does but the allocations made for the queue aren't taken from inside that buffer, the pointers will be invalid in the other process.