I am not finding documentation for custom protocol support.
From what I understand, Gatling has core engine that does scheduling, thread management etc, and protocol support is designed as an Actor ?
I am trying to develop a custom protocol (thats basically a shell script that will talk to an external service). The latest reference documentation does not seem to have any reference to how to do this ? Any pointers will be greatly appreciated.
If you need to stress test something that is implemented in a shell script, then Gatling probably isn't the best fit. Gatling is designed for stress testing networking protocols. So unless you can duplicate what your shell script is doing in Gatling expressed in networking protocols, you then might want to use something else.
Secondly, if you did implement it, I would check with the core developers of Gatling if it's something that they would consider including (use a github issue to ask). Since the applications of this might not be widespread, they may choose to not include it in their project. If that's the case you would have to either run your own fork with the implementation or add some sort of plugin architecture to Gatling for 3rd part extensibility.
So my suggestions are:
Decompose your shell script into the specific network protocol parts you're interested in stress testing implementing in Gatling.
Use a different tool that's designed to running multiple shell scripts at once for stress testings. Something like GNU Parallel if you're on a Linux box.
Implement it yourself. There's no documentation on how to do this. However a good starting example would be the JMS Protocol Implementation to give you an idea of all that's involved.
Related
I'm working on a project, implementing everything in C language. As a part of the project, we need to be able to control and configure firewalld, firewall of the current system.
firewalld is implemented in Python and an interface is available. However, we don't want to make Python calls from C or vice versa.
There are command line tools to configure firewalld (e.g. firewall-cmd) but we don't want to make such calls from C either.
I recently started working on firewalld, I don't know much about its internals. I've read that it uses D-Bus, I also don't know much about D-Bus.
There is a C library developed by Thomas Woerner: libfirewall.
However, it's been more than a year and a half since the last commit so it's not maintained. Other than libfirewall, I don't know any firewalld interface in C.
I gave libfirewall a shot. It got me some problems when both compiling and running the examples and I still have problems to resolve. Is it worth to continue with libfirewall? Should I use it?
Is there any other interface that I'm not aware of?
Possibly naive question due to lack of understanding of D-Bus: I thought, maybe, with a D-Bus interface, I can issue commands to firewalld. Can it be done? (i.e. Does D-Bus work like that?) Can we write a program that mimics, say firewall-cmd, and interacts with D-Bus in the same way and at the end allows us to control firewalld?
If this is possible, how to do it and what to use? libdbus and GDBus have relatively good documentation although libdbus requires good deal of effort. They even said "If you use this low-level API directly, you're signing up for some pain." in the documentation. In any case I'll be in need of examples or any kind of text demonstrating their usage.
How should I approach this problem?
Yes, you can issue commands to firewalld via D-Bus. I haven't checked but expect that firewall-cmd is itself implemented as a D-Bus client.
The D-Bus API is extensively documented: https://firewalld.org/documentation/man-pages/firewalld.dbus.html. The documentation should give you a rough idea what can be accomplished through the API. You could try the D-Bus debugger d-feet to interact with firewalld without any code.
GDBus is definitely the easiest way use D-Bus from C but it's still not trivial and firewalld is a fairly complex API: Using it may require some expertise (completely depending on what you need to do).
I was looking for a good state machine compiler so as to test some custom networking protocols. I looked at a few tools already such as Yakindu, Ragel(compiler), SCXML(language) but I was not sure if any of them could be used for networking protocols.
SCXML(language) looks good but I could not find any compilers specifically for C (scxmlcc is for C++). Does anyone know a compiler for C based on SCXML?
Yakindu tool looks promising, but I am not sure if network protocols like BGP/OSPF could be tested. Can anyone please give any pointers on this?
Ragel also looks good, but again I am not sure if complex network protocol clients can be generated using this compiler.
The reason I mentioned network protocols specifically is that I also want to be able to perform custom routines such as packet_create/packet_send (with custom packet sizes) etc as part of 'actions' after an 'event' occurs.
Do I need to always generate code from the state graph or Is there a way to directly interact with the states?
I am very new to FSMs, any help/advice/suggestion/links will be greatly appreciated.
The default Yakindu SCT C code generator generates plain C code that is agnostic of the concrete execution environment. This means you get a piece of standard C code that you can integrate into you own application manually. This manual integration means mapping events, operations, and variables between your application and the state machine. Additionally you have to trigger the state machine execution properly.
You can find some information on that in the user guide. You could also customize the code generators in order to match your needs but that makes only sense if the integration into the application always look the same and you want to integrate more than one or two state machines.
So if you provide some more information about the application side API that i could provide some hints how to integrate with the state machine.
We just finished our SCXML -> ANSI C transformation. Currently, it does everything but invocations. Here is a sample of generated ANSI C code, with user-supplied callbacks and the general scaffolding here. Performance measurements for a single microstep on a late 2015 MacBook Pro#3.1GHz are here.
Note that the scaffolding is in C++98 as we had to connect to a data-model implementation to pass the SCXML IRP tests. Generated source is ANSI C though. If you want to transform a SCXML state-chart you can use uscxml-transform as:
$ uscxml-transform -tc -i FILE_OR_URL -o GENERATED_HERE
For example, to print the generated C code for test144 from the SCXML IRP suite on STDOUT:
$ uscxml-transform -tc -i https://raw.githubusercontent.com/tklab-tud/uscxml/master/test/w3c/ecma/test144.scxml
Generated ANSI-C code passes all SCXML IRP tests but those for invocations and custom I/O processors.
LOKI is a new application designed to provide an easy way for programmers and system admistrators to interact with BGP networks. Use it to test your OSPF/BGP connections as well
Is there any high performance (ideally evented and open source) web server in C or C++?
I'd like to be able to use it in that it calls a method/function in my application with a filled out HTTP Request class/struct, and then I can return a filled out HTTP Response class/struct to it.
If it isn't open source, I'd need built in support for long-polling connections, keep-alive, etc—otherwise, I think that I can add these things myself.
If you don't know of any such servers available, would you recommend writing my own web server to fit the task? It cannot be file-based, and must be written in high-performance C/C++.
Edit: I'm thinking something like the Ruby Mongrel for C, if that helps.
I had the very same requirements for my job, so I evaluated a number of solutions: mongoose, libmicrohttpd, libevent. And I also was thinking about writing nginx modules. Here is the summary of my findings:
nginx
nginx project page
I love this server and use it a lot. Its performance and resource usage is much better than that of Apache, which I also still use but plan migrating to nginx.
Very good tunable performance. Rich functionality. Portability.
Module API is not documented and seems to be very verbose. See this nginx hello world module as example.
Nginx does not use threads but uses multiple processes. This makes writing modules harder, need to learn nginx API for shared memory, etc.
mongoose
mongoose project page
All server's code is in single mongoose.c file (about 130K), no dependencies. This is good.
One thread per connection, so if you need concurrency you've got to configure lots of threads, ie. high RAM usage. Not too good.
Performance is good, although not exceptional.
API is simple but you have to compose all response HTTP headers yourself, ie. learn HTTP protocol in detail.
libmicrohttpd
libmicrohttpd project page
Official GNU project.
Verbose API, seems awkward to me, although much more simple than writing nginx modules.
Good performance in keep-alive mode (link to my benchmarks below), not so good without keep-alive.
libevent
libevent project page
Libevent library has built-in web server called evhttp.
It is event based, uses libevent for that.
Easy API. Constructs HTTP headers automatically.
Officially single-threaded. This is major disadvantage. I've found a hack, which makes several instances of evhttp run simultaneously accepting connections from the same socket. Not sure if it is all safe and robust.
Performance of single-threaded evhttp is surprisingly poor. Multi-threaded hack works better, but still not good.
G-WAN
G-WAN project is not open source, but I'd like to say a few words about it.
Very good performance, low memory usage, 150 KB executable.
Very convenient 'servlet' deployment: just copy .c file into csp directory, and running server automatically compiles it. Code modifications also compiled on the fly.
Simple API. Although constrained in some ways. Rich functionality (json, key-value store, etc.).
Unstable. I had segfaults on static files. Hangs on some sample scripts. (Experienced on clean install. Never mixed files of different versions).
Only 32-bit binary (not anymore).
So as you can see, none of existing alternatives have fully satisfied me. So I have developed my own server, which is ...
NXWEB
NXWEB project page
Feature highlights:
Very good performance; see benchmarks on project page
Can serve tens of thousands concurrent requests
Small memory footprint
Multi-threaded model designed to scale
Exceptionally light code base
Simple API
Decent HTTP protocol handling
Keep-alive connections
SSL support (via GNUTLS)
HTTP proxy (with keep-alive connection pooling)
Non-blocking sendfile support (with configurable small file memory cache; gzip pre-encoded file serving)
Modular design for developers
Can be run as daemon; relaunches itself on error
Open source
Limitations:
Depends on libev library (not anymore)
Only tested on Linux
I would suggest to write a FastCGI executable that can be used with many high performance web servers (even closed source ones).
mongoose: one file. simple and easy to use. not an asycn io but perfect for embedded and particular purposes.
gwan. excellent. no crashes. ultra well planned configuration. very smart and easy for c/c++ development in other words, very clean sensible api compared to nginx. provides a thread per core. or whatever you specify. a great choice. largest disadvantage (maybe im lacking in this area): cannot step thru code.
libevent: single thread is not a disadvantage on a single core machine. afterall its point is an async i/o. does have multithreads for other cores.
nginx: no personal experience. gaining serious ground on a-patchy server. (terribly confusing api)
boost asio: a c++ library for asynchio (asio). awesome. needs a friendly higher-level api for simpletons like myself. and others who come from php, java, javascript, node.js and other web languages.
python bottle: awesome 1 file lib (framework/system) that makes it easy to build python web apps. has/is a built in httpd server, like libevent and node.js
node.js: javascript asyncio server. an excellent selection. unfortunately, have to program in javascript that does become tedious. while there is something to be said for getting the job done; there is also something to be said for enjoying yourself during the process. hopefully no ones comes up with node.php
I'm going to suggest the same thing as Axel Gneiting - but have provided an answer with my reasons for taking this approach:
1) HTTP is not trivial as a protocol - writing your own server or amending an off-the-shelf solution is a very complex task - a lot more complex than using the available APIs for implementing a separate processing engine
2) Using (an unmodified) mainstream webserver should provide you with more functionality than you require (so you've got growing room).
3) Using (an unmodified) mainstream webserver will usually mean that it has been far more extensively tested and documented than a homebrew system.
4) .. and its more likely to be secure and stable.
5) Using fastCGI you can use all sorts of languages to develop your back-end processing in - including C++ and C. There are standard toolkits available to facilitate this.
6) alternatively many webservers provide support for running interpreter engines in-process (e.g. mod_php, mod_perl). I'd advise against running your own native code as a module though.
It cannot be file-based.
Eh? What does that mean?
I'm an avid nginx user; nginx is written in C; nginx seems like it could work for you. If you want the very best speed out of nginx, I would make a nginx module. Here are 3rd party modules which you can examine to get an idea of what it requires.
As for the long polling requirement, you might want to have a look at the http push modules.
Say I have fancy new algorithm written in C,
int addone(int a) {
return a + 1;
}
And I want to deploy as a web application, for example at
http://example.com/addone?a=5
which responds with,
Content-Type: text/plain
6
What is the best way to host something like this? I have an existing setup using Python mod_wsgi on Apache2, and for testing I've just built a binary from C and call as a subprocess using Python's os.popen2.
I want this to be very fast and not waste overhead (ie I don't need this other Python stuff at all). I can dedicate the whole server to it, re-compile anything needed, etc.
I'm thinking about looking into Apache C modules. Is that useful? Or I may build SWIG wrappers to call directly from Python, but again that seems wasteful if I'm not using Python at all. Any tips?
The easiest way should be to write this program as a CGI app (http://en.wikipedia.org/wiki/.cgi). It would run with any webserver that supports the Common Gateway Interface.
The output format needs to follow the CGI rules.
If you want to take full advantage of the web server capabilities then you can write an Apache module in C. That needs a bit more preparation but gives you full control.
Maybe this tiny dynamic webserver in C to be used with C language can help you.. it should be easy to use and self-contained.
Probably the fastest solution you can adopt according to the benchmarks shown on their homepage!
This article from yesterday has a good discussion on why not to use C as a web framework. I think an easy solution for you would be to use ctypes, it's certainly faster than starting a subprocess. You should make sure that your method is thread safe and that you check your input argument.
from ctypes import *
libcompute = CDLL("libcompute.so")
libcompute.addone(int(a))
I'm not convinced that you're existing general approach might not be the best one. I'm not saying that Apache/Python is necessarily the correct one but there is something compelling about separating the concerns in your architecture being composed of highly focused elements that are specialists in their functions within the overall system.
Having your C-based algorithm server being decoupled from the HTTP server may give you access to things like HTTP scalability and caching facilities that might otherwise have to be in-engineered (or reinvented) within your algorithm component if things are too tightly coupled.
I don't think performance concerns in of themselves are always the best or only reasons when designing an architecture. For example the a YAWS deployment with a C-based driver could be a very performant option.
I have just setup a web service using libmicrohttpd and have had amazing results. On a quad core I've been handling 20400 requests a second and the CPU is running only at 58%. This is probably going to be deployed on a server with 8 cores, so I'm expecting much better results. A very simple C service will be even faster!
I have tried GWAN, it is very good, but it's closed, and doesn't play well with virtual environments. I will give #Gil kudos being good at supporting it here though. We just had a few issues and found LibMicroHttpd works better for our needs.
If you go here, you may need to update your openssl if you're using CentOs from axivo
rpm -ivh --nosignature http://rpm.axivo.com/redhat/axivo-release-6-1.noarch.rpm
yum --disablerepo=* --enablerepo=axivo update openssl-devel
You can try Duda I/O, it only requires a Linux host: http://duda.io
Introduction:
I am currently developing a document classifier software in C/C++ and I will be using Naive-Bayesian model for classification. But I wanted the users to use any algorithm that they want(or I want in the future), hence I went to separate the algorithm part in the architecture as a plugin that will be attached to the main app # app start-up. Hence any user can write his own algorithm as a plugin and use it with my app.
Problem Statement:
The way I am intending to develop this is to have each of the algorithms that user wants to use to be made into a DLL file and put into a specific directory. And at the start, my app will search for all the DLLs in that directory and load them.
My Questions:
(1) What if a malicious code is made as a DLL (and that will have same functions mandated by plugin framework) and put into my plugins directory? In that case, my app will think that its a plugin and picks it and calls its functions, so the malicious code can easily bring down my entire app down (In the worst case could make my app as a malicious code launcher!!!).
(2) Is using DLLs the only way available to implement plugin design pattern? (Not only for the fear of malicious plugin, but its a generic question out of curiosity :) )
(3) I think a lot of softwares are written with plugin model for extendability, if so, how do they defend against such attacks?
(4) In general what do you think about my decision to use plugin model for extendability (do you think I should look at any other alternatives?)
Thank you
-MicroKernel :)
Do not worry about malicious plugins. If somebody managed to sneak a malicious DLL into that folder, they probably also have the power to execute stuff directly.
As an alternative to DLLs, you could hook up a scripting language like Python or Lua, and allow scripted plugins. But maybe in this case you need the speed of compiled code?
For embedding Python, see here. The process is not very difficult. You can link statically to the interpreter, so users won't need to install Python on their system. However, any non-builtin modules will need to be shipped with your application.
However, if the language does not matter much to you, embedding Lua is probably easier because it was specifically designed for that task. See this section of its manual.
See 1. They don't.
Using a plugin model sounds like a fine solution, provided that a lack of extensibility really is a problem at this point. It might be easier to hard-code your current model, and add the plugin interface later, if it turns out that there is actually a demand for it. It is easy to add, but hard to remove once people started using it.
Malicious code is not the only problem with DLLs. Even a well-meaning DLL might contain a bug that could crash your whole application or gradually leak memory.
Loading a module in a high-level language somewhat reduces the risk. If you want to learn about embedding Python for example, the documentation is here.
Another approach would be to launch the plugin in a separate process. It does require a bit more effort on your part to implement, but it's much safer. The seperate process approach is used by Google's Chrome web browser, and they have a document describing the architecture.
The basic idea is to provide a library for plugin writers that includes all the logic for communicating with the main app. That way, the plugin author has an API that they use, just as if they were writing a DLL. Wikipedia has a good list of ways for inter-process communication (IPC).
1) If there is a malicious dll in your plugin folder, you are probably already compromised.
2) No, you can load assembly code dynamically from a file, but this would just be reinventing the wheel, just use a DLL.
3) Firefox extensions don't, not even with its javascript plugins. Everything else I know uses native code from dynamic libraries, and is therefore impossible to guarantee safety. Then again Chrome has NaCL which does extensive analysis on the binary code and rejects it if it can't be 100% sure it doesn't violate bounds and what not, although I'm sure they will have more and more vulnerabilities as time passes.
4) Plugins are fine, just restrict them to trusted people. Alternatively, you could use a safe language like LUA, Python, Java, etc, and load a file into that language but restrict it only to a subset of API that wont harm your program or environment.
(1) Can you use OS security facilities to prevent unauthorized access to the folder where the DLL's are searched or loaded from? That should be your first approach.
Otherwise: run a threat analysis - what's the risk, what are known attack vectors, etc.
(2) Not necessarily. It is the most straigtforward if you want compiled plugins - which is mostly a question of performance, access to OS funcitons, etc. As mentioned already, consider scripting languages.
(3) Usually by writing "to prevent malicous code execution, restrict access to the plugin folder".
(4) There's quite some additional cost - even when using a plugin framework you are not yet familiar with. it increases cost of:
the core application (plugin functionality)
the plugins (much higher isolation)
installation
debugging + diagnostics (bugs that occur only with a certain combinaiton of plugins)
administration (users must know of, and manage plugins)
That pays only if
installing/updating the main software is much more complex than updating the plugins
individual components need to be updated individually (e.g. a user may combine different versions of plugins)
other people develop plugins for your main application
(There are other benefits of moving code into DLL's, but they don't pertain to plugins as such)
What if a malicious code is made as a DLL
Generally, if you do not trust dll, you can't load it one way or another.
This would be correct for almost any other language even if it is interpreted.
Java and some languages do very hard job to limit what user can do and this works only because they run in virtual machine.
So no. Dll loaded plug-ins can come from trusted source only.
Is using DLLs the only way available to implement plugin design pattern?
You may also embed some interpreter in your code, for example GIMP allows writing plugins
in python.
But be aware of fact that this would be much slower because if nature of any interpreted language.
We have a product very similar in that it uses modules to extend functionality.
We do two things:
We use BPL files which are DLLs under the covers. This is a specific technology from Borland/Codegear/Embarcadero within C++ Builder. We take advantage of some RTTI type features to publish a simple API similar to the main (argv[]) so any number of paramters can be pushed onto the stack and popped off by the DLL.
We also embed PERL into our application for things that are more business logic in nature.
Our software is an accounting/ERP suite.
Have a look at existing plugin architectures and see if there is anything that you can reuse. http://git.dronelabs.com/ethos/about/ is one link I came across while googling glib + plugin. glib itself might may it easier to develop a plugin architecture. Gstreamer uses glib and has a very nice plugin architecture that may give you some ideas.