I need to run a program that crawls websites and I already have an algorithm and some parts of the code. Problem is, I do not know how to insert wget into my source code. Our student assistant hinted that some kind of keyword or function shall be used before the wget( system, I think or something but I'm not so sure).
when to not use system:
1.) when you want to distribute the program to different environment, where the program you call via system is not available
2.) in a security relevant environment, where you have to make sure that the program you call is really the program you want it to be
3.) when the thing you want to do can easily be accomplished in 10-20 lines of C code
4.) in performance-critical applications
so, you should use system virtually never.
instead, to accomplish the same thing, you could use libcurl, as David suggested (his answer seems to be gone...), or do some socket programming (it's C, after all).
In a real-world scenario, I'd probably just default to writing the crawler in a different language. web requests and complex string processing are not necessarily the strong sides of C, and most definitely not very convenient to use :)
You can use the system() command.
In your case (possibly):
system("/bin/wget");
But if you want really call wget with parameters, so you should use execl().
execl("/bin/wget", "http://anyadress.com/file");
Whenever , you want to run shell commands from your C program , you use system("shell command").In your case
system("wget");
Note - wget is an executable , whose location is added to the path variable, so there is no need to specify the path explicitly.
-- Example --
#include <stdio.h>
#define BUFFLEN 2500
int main()
{
char web_address[BUFFLEN] = "www.google.com";
system("wget 'web_address' ");
return 0;
}
The system command is used to execute a shell command. man system
Related
I am trying to call a do file which has loops from a program in other do file. I am getting an error.
Now, if I use do instead of include, it runs fine but I don't get to use local macros created. I used include so I can use the macros further in the program. I don't want to use global.
First do file (test.do).
forval i = 1/5 {
local val`i' = `i'
}
Second do file(call-test.do)
capture program drop test
program test
include "test.do"
di `val1'
end
test
I got error r(9611);
I using version 16.1
Response from Stata support
The -include- is designed to let you share definitions. It will not
work correctly within a program as documented in -help include-
The short answer is that -include- is usually ok to use in programs,
but not with looping commands, and if you use -include- in a program,
it probably isn't working the way you think it is.
Here's the long version of exactly what is going on:
When you use -include- in a program, your program literally includes
the -include- command in it. The program does NOT have the contents
of the include file substituted in place. That's the start of the
problem for looping commands.
In any case, when a program executes the -include- command, Stata gets
confused about whether to define a loop program on the behalf of a
looping command globally or within the program, and things go downhill
from there. Given how the code is structured, it is unlikely we could
fix -include- to behave differently, so our documentation really
should simply recommend against using -include- in programs. In
addition, at the point at which the failure occurs, Stata simply knows
that it cannot call a program that it thinks should already be in
memory, hence the 9611 return code. It has no idea at that point that
this was because it was called with -include-, unfortunately.
We could in the future introduce a true C-like "#include" for use in
programs which would simply substitute in-line the lines from whatever
was included into your program
Is it possible call a separate C program (.exe file) within a C program, like if it was a function?
I would like to be able to pass arguments of any kind (like any other function) to this separate program, and get the return value (so it can be used in the host program).
I imagine that the arguments can be passed by using int argc, char *argv[], but I don't know if it's possible to pass integers, arrays, pointers to structures and so on.
On the other hand, I've read that the return value from the main function is system specific. Since I'm using Windows, is there any limitations to this return value (type, size, etc.)? Can it be anything that could be used as a return value in any normal function?
Thanks!
What you describe, is the basic premise of the Unix operating system. Unix was designed to allow accomplishing very complex tasks by chaining several commands, piping the (text) output of a command as the input of the next one (this was pretty revolutionary back then).
As klutt already suggested, you can accomplish the same with a Windows executable. To his list, I would add learning how to redirect the input/output of a program to a file handle.
The Windows PowerShell extended this concept to allow passing different data-types other than text, to some special executables known as cmdlets, however, to write your own, you need support from the .Net Framework or the .Net Core infrastructure, so you must do so from a managed language such as C# or C++/CLI.
Keep in mind that spawning a whole process is an extremely expensive operation (compared to simply calling a linked function), so there is some significant overhead you need to be aware of.
I know how to intercept system calls with LD_PRELOAD, that occur in compiled programs I may not have source for. For example, if I want to know about the calls to int fsync(int) of some unknown program foobar, I compile a wrapper
int fsync(int)
for
(int (*) (int))dlsym(RTLD_NEXT,"fsync");
into a shared library and then I can set the environment variable LD_PRELOAD to that and run foobar. Assuming that foobar is dynamically linked, which most programs are, I will know about the calls to fsync.
But now suppose there is another unknown program foobar1 and in the source of that program was a statement like this:
execve("foobar", NULL, NULL)
that is, the environment was not passed. Now the whole LD_PRELOAD scheme breaks down?
I checked by compiling the statemet above into foobar1, when that is run, the calls from foobar are not reported.
While one can safely assume most modern programs are dynamically linked, one cannot at all assume how they may or may not be using execve?
So then, the whole LD_PRELOAD scheme, which everybody says is such a great thing, is not really working unless you have the source to the programs concerned, in which case you can check the calls to execve and edit them if necessary. But in that case, there is no need for LD_PRELOAD, if you have sources to everything. LD_PRELOAD is specifically, supposed to be, useful when you don't have sources to the programs you are inspecting.
Where am I wrong here - how can people say, that LD_PRELOAD is useful for inspecting what unknown programs are doing??
I guess I could also write a wrapper for execve. In the wrapper, I add to the original envp argument, one more string: "LD_PRELOAD=my library" . This "seems" to work, I checked on simple examples.
I am not sure if I should be posting an "answer" which may very easily exceed my level of C experience.
Can somebody more experienced than me comment if this is really going to work in the long run?
I am trying to call a C program from my Ruby script, parsing it an argument (file object) and then store some variables the C program would return.
The idea is that my Ruby script allows me to easily cycle through the files & folders of a parent folder but it is way too slow to efficiently process all the files in that folder. Hence the use of a C program that I want to call to process each file.
My problem is that I can't find a method to call that C program from Ruby (and how to parse it the file argument, I'm not even sure it is possible as I don't know if Ruby files objects and C streams are "compatible")
Thank you in advance for your help !
You say you are trying to call a program so I assume you are not trying to statically or dynamically load a library and call a function. (If you are trying to load a library to call a function then look to the DL::Importer module.)
As for calling an external program from Ruby and receiving its result (from stdout, in this case), regardless of whether it was written in C or not, an easy way to do it is:
value = `program arg1 arg2 ...`
e.g. if the program you want to call compresses a given file and outputs the compressed size.
size = `mycompressionprogram filename.txt`
puts "compressed result is: #{size}"
Note those are back ticks " ` ".
So this is one easy way to code your computationally heavy stuff in C and wrap it up in a Ruby script.
One simple traditional way for a Ruby process to interact with unrelated C code is popen, which will allow your Ruby process to invoke the (compiled) code as a separate process, passing your choice of arguments into the traditional space the operating system allocates for that (accessible in argv in your process's int main(int argc, char** argv)), and then interacting with its standard input and standard output over a pipe. However, this technique launches another process and requires that you serialize/deserialize any ongoing interprocess communication so that it can run over the pipe, which may be an impediment.
So you can also write the C code as a Ruby extension, which will allow you to return values more readily, and moreover avoids the overhead associated with having a separate process involved. However, note that if you perform extensive work with Ruby objects in your C code you may still incur the performance penalties you'd hoped to avoid. The canonical document on how to write Ruby extensions is README.EXT.
If a C program was used to send a combination of Find/Sed instructions to system, where would a system admin best find evidence of this happening, and is it possible to find the exact arguments passed to these programs? Just to say that I am mentioning that it is a C program doing this to exclude the bash history. Would really appreciate someone to give me a list of places to look. Thank you.
pseudo code:
Way in which Find/Sed invoked:
Command= find .... exec Sed.....
sprintf(command,....
system(command);
That would depend entirely on how the C program invoked find/sed, and whether or not it redirected the I/O. There may not be any record of the process having been run unless some kind of process accounting is enabled.
Either use absolute paths or use the function getcwd to get the current working directory and work it out from there where the files will be stored from the redirection.