i'm trying to modify nodemcu lua file.list functon /app/modules/file.c
to return a large string of filenames separated with newline char. currently it returns array and is very memory consuming also i'm stripping file size.
Here is what i have done (examined how some other functions return strings)
static int file_list( lua_State* L )
{
char temp[32];
unsigned st = luaL_optinteger( L, 1, 1 ); // start offset
unsigned tf = luaL_optinteger( L, 2, 100000 ); // how much files to list
tf=tf+st;
vfs_dir *dir;
if (dir = vfs_opendir("")) {
lua_newtable( L );
struct vfs_stat stat;
int i=1;
int ii=0;
while (vfs_readdir(dir, &stat) == VFS_RES_OK) {
if (i<st)
{
i++;
continue;
}
if (i>=tf)
{
break;
}
strcpy (temp,stat.name);
strcat (temp,"\n");
lua_pushstring( L, temp );
i++;
ii++;
}
vfs_closedir(dir);
return ii;
}
return 0;
}
It do not work as expected, If I request more than 40 files (at once after device reboots) I see output like this:
....
3fff0d10 already freed
3fff11b8 already freed
3fff0cc8 already freed
3fff1f88 already freed
.....
and device restart, but if I request 30 files, and every time increase them in steps of 30, manage to get 400 files at one time.
=file.list(1,30)
=file.list(1,60)
=file.list(1,90)
that way it works, If I do directly:
=file.list(1,60)
it does not work. Noticed also that memory is allocated and not set free after function finished, but also not reallocate after same command execution so it is not memory leak, just some date stays in the stack perhaps.
I am having a lot of trouble with this piece of code (I am not good at pointers :P). So here is the code.
printf("\n Enter the file name along with its extensions that you want to delete:-");
scanf("%s",fileName);
deletefile_1_arg=fileName;
printf("test\n");
result_5 = deletefile_1(&deletefile_1_arg, clnt);
if (result_5 == (int *) NULL) {
clnt_perror (clnt, "call failed");
}
else
{
printf("\n File is deleted sucessfully");
goto Menu2;
}
break;
Function that is getting called is as following.
int *
deletefile_1_svc(char **argp, struct svc_req *rqstp)
{
static int result;
printf("test2\n");
printf("%s",**argp);
if(remove(**argp));
{
printf("\nFile Has Been Deleted");
result=1;
}
return &result;
}
I am getting test2 on console but. It does not print value of argp / removes that perticular file. I am not sure what I am doing wrong. Please help me.
The argp is a pointer to a pointer char, and you are trying to use it as a pointer to char, try change your code to:
printf("%s", *argp);
You would also need to change your remove call to:
remove(*argp);
I always found drawing pictures helped understand pointers. Use boxes for memory addresses and a label for the box is the variable name. If the variable is a pointer, then the contents of the box is the address of another box (draw line to the other box).
You are using pointers when you don't need to. Your "deletefile1_svc" function doesn't manipulate the value of "argp" at all so it doesn't need a pointer-to-pointer. Plus your "result" doesn't need to be returned as a pointer since it is simply a numeric value. You also don't initialize result (it might be zero) or re-initialize it (it is static so it will remember the last value assigned to it).
int
deletefile_1_svc(const char *argp, struct svc_req *rqstp)
{
int result = 0; /* Initial value => failure */
if (remove (argp) == 0)
{
result = 1; /* 1 => success */
}
return result;
}
To call the function use:
result_5 = deletefile1_svc(filename, clnt);
if (result_5 == 0)
// Failed
else
// Success
That will make the code simpler and less prone to bugs.
I have this .c file that counts the system calls that linux calls. These are just the main functions. There were a couple of other things that I had to do, like create an array
unsigned long syscall_counts[345];
and then in another file with some assembly I incremented the array with the command:
incl syscall_counts(,%eax,4)
// This function is called each time the application calls read(). It starts the process of
// accumulating data to fill the application buffer. Return a pointer representing the current
// item. Return NULL if there are no more items.
//
static void *counter_seq_start(struct seq_file *s, loff_t *record_number)
{
if (*record_number > 347)
return NULL;
return (void*)s;
}
// This function is called to compute the next record in the sequence given a pointer to the
// current record (in bookmark). It returns a pointer to the new record (essentially, an updated
// bookmark) and updates *record_number appropriately. Return NULL if there are no more items.
//
static void *counter_seq_next(struct seq_file *s, void *bookmark, loff_t *record_number)
{
unsigned long *temp_b =(unsigned long*) bookmark;
(*temp_b)++;
if (*temp_b > 345)
return NULL;
return (void*)temp_b;
}
// This function is called whenever an application buffer is filled (or when start or next
// returns NULL. It can be used to undo any special preparations done in start (such as
// deallocating auxillary memory that was allocated in start. In simple cases, you often do not
// need to do anything in this function.
//
static void counter_seq_stop(struct seq_file *s, void *bookmark)
{
}
// This function is called after next to actually compute the output. It can use various seq_...
// printing functions (such as seq_printf) to format the output. It returns 0 if successful or a
// negative value if it fails.
//
static int counter_seq_show(struct seq_file *s, void *bookmark)
{
loff_t *bpos = (loff_t *) bookmark;
seq_printf(s, "value: %Ld\n", *bpos);
return 0;
}
// Define the only file handling function we need.
static int counter_open(struct inode *inode, struct file *file)
{
return seq_open(file, &counter_seq_ops);
}
my output is very strange:
Anyone have any idea where the issue is?
Don't you think :
static int counter_seq_show(struct seq_file *s, void *bookmark) {
unsigned long *bpos = (unsigned long *) bookmark;
seq_printf(s, "value: %Ld\n", *bpos);
return 0;
}
Or even
static int counter_seq_show(struct seq_file *s, void *bookmark) {
seq_printf(s, "value: %lu\n", *((unsigned long *)bpos));
return 0;
}
I haven't fully understood your program but I saw two different ways you cast 'bookmark'. In one function you cast it to be 'unsigned long *' and other you do 'loff_t *' (long int). Ideally they should be the same, but are you doing it this way for some reason ?
HTH
I am very much stuck in the following issue. Any help is very much appreciated!
Basically I have a program wich contains an array of structs and I am getting a segmentation error when I call an external function. The error only happens when I have more than 170 items on the array being passed.
Nothing on the function is processed. The program stops exactly when accessing the function.
Is there a limit for the size of the parameters that are passed to external functions?
Main.c
struct ratingObj {
int uid;
int mid;
double rating;
};
void *FunctionLib; /* Handle to shared lib file */
void (*Function)(); /* Pointer to loaded routine */
const char *dlError; /* Pointer to error string */
int main( int argc, char * argv[]){
// ... some code ...
asprintf(&query, "select mid, rating "
"from %s "
"where uid=%d "
"order by rand()", itable, uid);
if (mysql_query(conn2, query)) {
fprintf(stderr, "%s\n", mysql_error(conn2));
exit(1);
}
res2 = mysql_store_result(conn2);
int movieCount = mysql_num_rows(res2);
// withhold is a variable that defines a percentage of the entries
// to be used for calculations (generally 20%)
int listSize = round((movieCount * ((double)withhold/100)));
struct ratingObj moviesToRate[listSize];
int mvCount = 0;
int count =0;
while ((row2 = mysql_fetch_row(res2)) != NULL){
if(count<(movieCount-listSize)){
// adds to another table
}else{
moviesToRate[mvCount].uid = uid;
moviesToRate[mvCount].mid = atoi(row2[0]);
moviesToRate[mvCount].rating = 0.0;
mvCount++;
}
count++;
}
// ... more code ...
FunctionLib = dlopen("library.so", RTLD_LAZY);
dlError = dlerror();
if( dlError ) exit(1);
Function = dlsym( FunctionLib, "getResults");
dlError = dlerror();
(*Function)( moviesToRate, listSize );
// .. more code
}
library.c
struct ratingObj {
int uid;
int mid;
double rating;
};
typedef struct ratingObj ratingObj;
void getResults(struct ratingObj *moviesToRate, int listSize);
void getResults(struct ratingObj *moviesToRate, int listSize){
// ... more code
}
You are likely blowing up the stack. Move the array to outside of the function, i.e. from auto to static land.
Another option is that the // ... more code - array gets populated... part is corrupting the stack.
Edit 0:
After you posted more code - you are using C99 variable sized array on the stack - Bad IdeaTM. Think what happens when your data set grows to thousands, or millions, of records. Switch to dynamic memory allocation, see malloc(3).
You don't show us what listsize is, but I suppose it is a variable and not a constant.
What you are using are variable length arrays, VLA. These are a bit dangerous if they are too large since they usually allocated on the stack.
To work around that you can allocate such a beast dynamically
struct ratingObj (*movies)[listSize] = malloc(sizeof(*movies));
// ...
free(movies);
You'd then have in mind though that movies then is a pointer to array, so you have to reference with one * more than before.
Another, more classical C version would be
struct ratingObj * movies = malloc(sizeof(*movies)*listsize);
// ...
free(movies);
I have some long source code that involves a struct definition:
struct exec_env {
cl_program* cpPrograms;
cl_context cxGPUContext;
int cpProgramCount;
int cpKernelCount;
int nvidia_platform_index;
int num_cl_mem_buffs_used;
int total;
cl_platform_id cpPlatform;
cl_uint ciDeviceCount;
cl_int ciErrNum;
cl_command_queue commandQueue;
cl_kernel* cpKernels;
cl_device_id *cdDevices;
cl_mem* cmMem;
};
The strange thing, is that the output of my program is dependent on the order in which I declare the components of this struct. Why might this be?
EDIT:
Some more code:
int HandleClient(int sock) {
struct exec_env my_env;
int err, cl_err;
int rec_buff [sizeof(int)];
log("[LOG]: In HandleClient. \n");
my_env.total = 0;
//in anticipation of some cl_mem buffers, we pre-emtively init some. Later, we should have these
//grow/shrink dynamically.
my_env.num_cl_mem_buffs_used = 0;
if ((my_env.cmMem = (cl_mem*)malloc(MAX_CL_BUFFS * sizeof(cl_mem))) == NULL)
{
log("[ERROR]:Failed to allocate memory for cl_mem structures\n");
//let the client know
replyHeader(sock, MALLOC_FAIL, UNKNOWN, 0, 0);
return EXIT_FAILURE;
}
my_env.cpPlatform = NULL;
my_env.ciDeviceCount = 0;
my_env.cdDevices = NULL;
my_env.commandQueue = NULL;
my_env.cxGPUContext = NULL;
while(1){
log("[LOG]: Awaiting next packet header... \n");
//read the first 4 bytes of the header 1st, which signify the function id. We later switch on this value
//so we can read the rest of the header which is function dependent.
if((err = receiveAll(sock,(char*) &rec_buff, sizeof(int))) != EXIT_SUCCESS){
return err;
}
log("[LOG]: Got function id %d \n", rec_buff[0]);
log("[LOG]: Total Function count: %d \n", my_env.total);
my_env.total++;
//now we switch based on the function_id
switch (rec_buff[0]) {
case CREATE_BUFFER:;
{
//first define a client packet to hold the header
struct clCreateBuffer_client_packet my_client_packet_hdr;
int client_hdr_size_bytes = CLI_PKT_HDR_SIZE + CRE_BUFF_CLI_PKT_HDR_EXTRA_SIZE;
//buffer for the rest of the header (except the size_t)
int header_rec_buff [(client_hdr_size_bytes - sizeof(my_client_packet_hdr.buff_size))];
//size_t header_rec_buff_size_t [sizeof(my_client_packet_hdr.buff_size)];
size_t header_rec_buff_size_t [1];
//set the first field
my_client_packet_hdr.std_header.function_id = rec_buff[0];
//read the rest of the header
if((err = receiveAll(sock,(char*) &header_rec_buff, (client_hdr_size_bytes - sizeof(my_client_packet_hdr.std_header.function_id) - sizeof(my_client_packet_hdr.buff_size)))) != EXIT_SUCCESS){
//signal the client that something went wrong. Note we let the client know it was a socket read error at the server end.
err = replyHeader(sock, err, CREATE_BUFFER, 0, 0);
cleanUpAllOpenCL(&my_env);
return err;
}
//read the rest of the header (size_t)
if((err = receiveAll(sock, (char*)&header_rec_buff_size_t, sizeof(my_client_packet_hdr.buff_size))) != EXIT_SUCCESS){
//signal the client that something went wrong. Note we let the client know it was a socket read error at the server end.
err = replyHeader(sock, err, CREATE_BUFFER, 0, 0);
cleanUpAllOpenCL(&my_env);
return err;
}
log("[LOG]: Got the rest of the header, packet size is %d \n", header_rec_buff[0]);
log("[LOG]: Got the rest of the header, flags are %d \n", header_rec_buff[1]);
log("[LOG]: Buff size is %d \n", header_rec_buff_size_t[0]);
//set the remaining fields
my_client_packet_hdr.std_header.packet_size = header_rec_buff[0];
my_client_packet_hdr.flags = header_rec_buff[1];
my_client_packet_hdr.buff_size = header_rec_buff_size_t[0];
//get the payload (if one exists)
int payload_size = (my_client_packet_hdr.std_header.packet_size - client_hdr_size_bytes);
log("[LOG]: payload_size is %d \n", payload_size);
char* payload = NULL;
if(payload_size != 0){
if ((payload = malloc(my_client_packet_hdr.buff_size)) == NULL){
log("[ERROR]:Failed to allocate memory for payload!\n");
replyHeader(sock, MALLOC_FAIL, UNKNOWN, 0, 0);
cleanUpAllOpenCL(&my_env);
return EXIT_FAILURE;
}
if((err = receiveAllSizet(sock, payload, my_client_packet_hdr.buff_size)) != EXIT_SUCCESS){
//signal the client that something went wrong. Note we let the client know it was a socket read error at the server end.
err = replyHeader(sock, err, CREATE_BUFFER, 0, 0);
free(payload);
cleanUpAllOpenCL(&my_env);
return err;
}
}
//make the opencl call
log("[LOG]: ***num_cl_mem_buffs_used before***: %d \n", my_env.num_cl_mem_buffs_used);
cl_err = h_clCreateBuffer(&my_env, my_client_packet_hdr.flags, my_client_packet_hdr.buff_size, payload, &my_env.cmMem);
my_env.num_cl_mem_buffs_used = (my_env.num_cl_mem_buffs_used+1);
log("[LOG]: ***num_cl_mem_buffs_used after***: %d \n", my_env.num_cl_mem_buffs_used);
//send back the reply with the error code to the client
log("[LOG]: Sending back reply header \n");
if((err = replyHeader(sock, cl_err, CREATE_BUFFER, 0, (my_env.num_cl_mem_buffs_used -1))) != EXIT_SUCCESS){
//send the header failed, so we exit
log("[ERROR]: Failed to send reply header to client, %d \n", err);
log("[LOG]: OpenCL function result was %d \n", cl_err);
if(payload != NULL) free(payload);
cleanUpAllOpenCL(&my_env);
return err;
}
//now exit if failed
if(cl_err != CL_SUCCESS){
log("[ERROR]: Error executing OpenCL function clCreateBuffer %d \n", cl_err);
if(payload != NULL) free(payload);
cleanUpAllOpenCL(&my_env);
return EXIT_FAILURE;
}
}
break;
Now what's really interesting is the call to h_clCreateBuffer. This function is as follows
int h_clCreateBuffer(struct exec_env* my_env, int flags, size_t size, void* buff, cl_mem* all_mems){
/*
* TODO:
* Sort out the flags.
* How do we store cl_mem objects persistantly? In the my_env struct? Can we have a pointer int the my_env
* struct that points to a mallocd area of mem. Each malloc entry is a pointer to a cl_mem object. Then we
* can update the malloced area, growing it as we have more and more cl_mem objects.
*/
//check that we have enough pointers to cl_mem. TODO, dynamically expand if not
if(my_env->num_cl_mem_buffs_used == MAX_CL_BUFFS){
return CL_MEM_OUT_OF_RANGE;
}
int ciErrNum;
cl_mem_flags flag;
if(flags == CL_MEM_READ_WRITE_ONLY){
flag = CL_MEM_READ_WRITE;
}
if(flags == CL_MEM_READ_WRITE_OR_CL_MEM_COPY_HOST_PTR){
flag = CL_MEM_READ_WRITE | CL_MEM_COPY_HOST_PTR;
}
log("[LOG]: Got flags. Calling clCreateBuffer\n");
log("[LOG]: ***num_cl_mem_buffs_used before in function***: %d \n", my_env->num_cl_mem_buffs_used);
all_mems[my_env->num_cl_mem_buffs_used] = clCreateBuffer(my_env->cxGPUContext, flag , size, buff, &ciErrNum);
log("[LOG]: ***num_cl_mem_buffs_used after in function***: %d \n", my_env->num_cl_mem_buffs_used);
log("[LOG]: Finished clCreateBuffer with id: %d \n", my_env->num_cl_mem_buffs_used);
//log("[LOG]: Finished clCreateBuffer with id: %d \n", buff_counter);
return ciErrNum;
}
The first time round the while loop, my_env->num_cl_mem_buffs_used is increased by 1. However, the next time round the loop, after the call to clCreateBuffer, the value of my_env->num_cl_mem_buffs_used gets reset to 0. This does not happen when I change the order in which I declare the members of the struct! Thoughts? Note that I've omitted the other case statements, all of which so similar things, i.e. updating the structs members.
Well, if your program dumps the raw memory content of the object of your struct type, then the output will obviously depend on the ordering of the fields inside your struct. So, here's one obvious scenario that will create such a dependency. There are many others.
Why are you surprised that the output of your program depends on that order? In general, there's nothing strange in that dependency. If you base your verdict of on the knowledge of the rest of the code, then I understand. But people here have no such knowledge and we are not telepathic.
It's hard to tell. Maybe you can post some code. If I had to guess, I'd say you were casting some input file (made of bytes) into this struct. In that case, you must have the proper order declared (usually standardized by some protocol) for your struct in order to properly cast or else risk invalidating your data.
For example, if you have file that is made of two bytes and you are casting the file to a struct, you need to be sure that your struct has properly defined the order to ensure correct data.
struct example1
{
byte foo;
byte bar;
};
struct example2
{
byte bar;
byte foo;
};
//...
char buffer[];
//fill buffer with some bits
(example1)buffer;
(example2)buffer;
//those two casted structs will have different data because of the way they are defined.
In this case, "buffer" will always be filled in the same manner, as per some standard.
Of course the output depends on the order. Order of fields in a struct matters.
An additional explanation to the other answers posted here: the compiler may be adding padding between fields in the struct, especially if you are on a 64 bit platform.
If you are not using binary serialization, then your best bet is an invalid pointer issue. Like +1 error, or some invalid pointer arithmetics can cause this. But it is hard to know without code. And it is still hard to know, even with the code. You may try to use some kind of pointer validation/tracking system to be sure.
other guesses
by changing the order you are having different uninitialized values in the struct. A pointer being zero or not zero for example
you somehow manage to overrun an item (by casting ) and blast later items. Different items get blasted depending on order
This may happen if your code uses on "old style" initializer as from C89. For a simple example
struct toto {
unsigned a;
double b;
};
.
.
toto A = { 0, 1 };
If you interchange the fields in the definition this remains a valid initializer, but your fields are initialized completely different. Modern C, AKA C99, has designated initializers for that:
toto A = { .a = 0, .b = 1 };
Now, even when reordering your fields or inserting a new one, your initialization remains valid.
This is a common error which is perhaps at the origin of the initializerphobia that I observe in many C89 programs.
You have 14 fields in your struct, so there is 14! possible ways the compiler and/or standard C library can order them during the output.
If you think from the compiler designer's point of view, what should the order be? Random is certainly not useful. The only useful order is the order in which the struct fields were declared by you.