Does anyone have any experience with doing this? I'm working on a Java decompiler right now in C++, but would like a higher level language to do the actual transformations of the internal trees. I'm curious if the overhead of marshaling data between languages is worth the benefit of a more expressive and language for better articulating what I'm trying to accomplish (like Haskell). Is this actually done in the "real world", or is it usually pick a language at the beginning of a project and stick with it? Any tips from those who have attempted it?
I'm a big advocate of always choosing the right programming language for each challenge. If there is another language which handles some otherwise tricky task easily, I'd say go for it.
Does it happen in the real world? Yes. I am currently working on a project which is made up of both PHP and objective-c code.
The trick is, as you pointed out, the communication between the two languages. If at all possible, let each language stick to its own domain, and have the two sections communicate in the simplest way possible. In my case, it was XML documents sent via http. In your case, some kind of formatted text file might be the answer.
Marshalling costs depend on the languages and architecture you're working with. For example, if you're on the CLR or JVM, there are low-cost interop solutions available - though I know you are working with probably unmanaged C++.
Another avenue is an embedded domain-specific language. Tree transformations are often expressible via pattern matching and application of a relatively small number of functions. You could consider writing a simple tree pattern-matcher - e.g. something that looks like Lisp s-exprs but uses placeholders to capture variables - with associated actions that are functions that transform the matched subtree.
John Ousterhout, the inventor of Tcl/Tk was a stong advocate of multi-language programming and wrote quite extensively about it. In order to do it, you need a clean interface mechanism between the languages you are using for it. There are quite a few mechanisms for this. Examples of different mechanisms for doing this are:
SWIG (Simplified Wrapper and
Interface Generator can take a c
or c++ (or several other languages)
header file and generate an
interface for a high level language
such as perl or python that allows
you to access the API. There are
other systems that use this
approach.
Java supports JNI, and various
other systems such as Python's
ctypes, VisualWorks DLL/C
connect are native mechanisms
that allow you to explicitly
construct the call to the lower
level subsystem.
Tcl/Tk was designed explicitly to be
embeded, and has a native API
for a C library to add hooks into
the language. The constructs for
this resemble argv[] structures in
C, and were designed to make it
relatively easy to interface a
command-line based C program into
Tcl. This is similar to the above
example, but coming from the opposite
direction. Many scripting languages
such as Python, Lua and Tcl support
this type of mechanism.
Explicit glue mechanisms such as
Pyrex, which are similar to a
wrapper generator, but have their
own language for defining the
interface. Pyrex is actually a
complete programming language.
Middleware such as COM or
CORBA allow a generic
interface definition to be built
externally to the application in an
interface definition language
and language bindings for the
languages concerned to use the
common interface mechanism.
Related
I want to write a library that is dynamically loadable and callable from C code, but I really don't want to write it in C - the code is security critical, so I want a language that makes it easier to have confidence that my code is correct. What are my options?
To be more specific, I want C programmers to be able to #include this, and -l that, and start using my library just as if I had written it in C. I'd like programmers in other languages to be able to use their favourite tools for linking to C libraries to link to it. Ideally I'd like that to be possible on every platform that supports C, but I'll settle for Linux, Windows and MacOS.
Anything that compiles to native code. So you might Google for that - "languages that compile to native code." See, e.g., Programming languages that compile to native code and have the batteries included
C++ is often the choice for this. Compiles to native code and provided you keep your interfaces simple, easy to write an adapter layer.
Objective C and Fortran are also possible.
It sounds like you are looking for a language with ABI compatibility or which can be described as resulting in native code. So long as it can be compiled to a valid object file (typically an .obj or .o file) which is accepted by the linker, that should be the main criteria. You also then want to write a header file as a convenience for any client code which is written in C (or a closely related language/variant thereof).
As mentioned by others, you need a pretty good reason for choosing a language other than C as it is the lingua-franco of low-level/systems software. Assembler is an option, although harder to port between platforms. D is a more portable - but less widespread - alternative which is designed to produce secure, efficient native code with a minimum of fuss. There are many others.
Almost every security critical application I know of is written in C. I don't believe that there are any other language that has higher real status in producing secure applications.
C is being said to be a poor language for security by people who don't understand.
If you want C programmers to use your library, use C. Doing anything else is tying one hand behind your back whilst trying to walk on a balance beam (the gymnastics equipment). Sure, there are dozens of other languages that are CAPABLE of interfacing to C, but it typically involves using a C layer and then stuffing the C data types into a language specific data type (Java Objects, Python Objects, etc, etc), and when the call is finished, you use the same conversion back to a C data type. Just makes it harder to work with, and potentially slower if you don't get all the design decisions right. And people won't understand the source code, so won't like to use it (see more about this below).
If you want security, then write very good code, wearing your "security aspects" hat firmly on at all times, find a security mailing list or website and post it there for review, take the review comments on board, understand the comments, and fix any comments that are meaningful to fix. Distribute the source code to the users, so people can see what your code does. Those that understand security will know what to look for and understand that you have done a good job (or a bad job, whichever is applicable) - and those who don't will hopefully trust the right pople. If it's good, people will use it. If it's "hidden", and not easy to access, you won't get many customers, no matter what language you use.
Don't worry, you won't reveal anything more from releasing source. If there is a flaw in the code, and it is popular (or important) enough, someone will find the flaw, even if you publish only binaries. For those skilled in reverse engineering, not having source code is only a small obstacle.
Security doesn't stem from using a specific language or a specific tool, it stems from good design and good basic understanding of the problems with security.
And remember security by obscurity (whether that means "hidden source code" or "unusual language" or something else obscure) is false security.
You might be interested in ATS, http://ats-lang.sourceforge.net/. ATS compiles via C, can be as efficient as C, and can be used in a way that is ABI-compatible with C. From the project website:
ATS is a statically typed programming language that unifies implementation with formal specification. It is equipped with a highly expressive type system rooted in the framework Applied Type System, which gives the language its name. In particular, both dependent types and linear types are available in ATS. The current implementation of ATS (ATS/Anairiats) is written in ATS itself. It can be as efficient as C/C++ (see The Computer Language Benchmarks Game for concrete evidence) and supports a variety of programming paradigms
ATS's dependent and linear type system helps produce static guarantees about your code, including various aspects of resource management safety.
Chris Double has been writing a series of articles exploring the power of ATS's type system for systems programming here: http://bluishcoder.co.nz/tags/ats/. Of particular note is this article: http://bluishcoder.co.nz/2012/08/30/safer-handling-of-c-memory-in-ats.html
This document covers aspects of calling back and forth between ATS and C code: https://docs.google.com/document/d/1W6DYQApEqKgyBzMbvpCI87DBfLdNAQ3E60u1hUiMoU0
The main downside is that dependently-typed programming is still a daunting prospect, even for non-systems programming. The syntax of the language is also a bit weird: consider lexical quirks such as the use of abst#ype as a keyword. Finally, ATS is to some degree a research project, and I personally don't know whether it would be sensible to adopt for a commercial endeavour.
Theoretically, it's going to be Fortran: less indirection (as in: my array is [here], not just a pointer to here, and this is true of most but not all of your data structures and variables).
However... There are many gotchas and quirks in Fortran: not, perhaps, as many as in C but you probably know your way around C rather better than Fortran. Which is the point behind most of the comments saying 'Know your code' - but do you really know what your compiler is doing?
Knowing you, I'm prepared to take it on trust that you do, for C. Most programers don't. You do not know and cannot know what a local JVM or JIT compiler does, and that's a black hole in your security model if you're using Java or C# r scripting languages.
Ignore anyone who tells you that the hairy-chested he-men of secure computing write their own assembler: they probably don't even know the security errors they're making in any and all nontrivial projects they release. Know your compiler, indeed.
You could write it in lua - providing a C API to a Lua library is relatively straight forward. C++ is also an option, though of course you'd have to write C wrappers and make sure no exceptions can escape your functions. But honestly, if it's security critical the minor inconveniences of the C language shouldn't be that much of a big deal. What you really should be doing is prove the correctness of your program where feasible, and test extensively where it's not.
You can write a library in Java. JNI is normally used to call C from Java, but it can be used the other way around.
There is finally a decent answer to this question: Rust.
There are a lot of excellent answers how can one simulate object oriented concepts with C. To name a few:
C double linked list with abstract data type
C as an object oriented language
Can you write object-oriented code in C?
When is it appropriate to use such simulation and not to use languages that support object-oriented techniques natively?
Highly related:
Why artificially limit your code to C?
https://stackoverflow.com/questions/482574/whats-the-advantage-of-using-c-over-c-or-is-there-one
I'll give you the one reason I know of because it has been the case for me:
When you are developing software for a unique platform, and the only available compiler is a C compiler. This happens quite often in the world of embedded microcontrollers.
To just give you another example: a fair amount of the x86 Linux kernel is using C as if it were C++, when object-orientation seems natural (eg, in the VFS). The kernel is written in assembly and C (if that wasn't changed in the 3.0 kernel). The kernel coders create macros and structures, sometimes even named similar to C++ terms (eg, for_each_xxx), that allow them to code as-if. As others have pointed out, you'd never choose C if you start a heavily object-oriented program; but when you're adjusting C based code to add object-oriented features, you might.
When you want a cross-platform foundation for object-oriented APIs. A case in point is Apple's Core Foundation. Being entirely C, it could be easily ported, yet provides an extremely rich set of opaque objects to use.
A nice example of its flexibility is the way many of its types are 'toll-free' bridged with those from Foundation (a set of true OO Objective-C libraries). Many types from Core Foundation can be used, fairly naturally, in Foundation APIs, and vice-versa. It's hard to see this working so well without some OO concepts being present in the Core Foundation libraries.
We have a very versatile terminal/sniffer application which can do all sorts of things with TCP, UDP and serial connections.
We are looking to make it extensible -- i.e, allow people to write their own protocol parsers, highlighters, etc.
We created a C-like language for extending the product, and then discovered that for some coders, this presents a steep learning curve.
We are now pondering the question: Should we stick to C or go with something like Ruby or Lua?
C is beautiful for low-level stuff (like parsing binary data), because it supports pointers. But for exactly that reason, it can be tough to learn.
Ruby (etc) are easy to learn, but don't have pointers, so anything that has to do with parsing binary data gets ugly very fast.
What do you think? For extending a product that parses binary data -- Ruby/Lua or C/C++?
Would be great if you could give some background when you respond -- especially if you've done something similar.
Wireshark, the "world's foremost network protocol analyzer", is also a packet sniffer/analyzer, formerly also called Ethereal. It uses Lua to enable writing custom dissectors and taps, see the manual.
However, note that I have not used it, so I cannot tell how nice/effective/easy to learn the API is.
Like TCL, Lua was designed to be tightly integrated with an application. Personally, I find Lua's syntax and idioms to be much easier to deal with than TCL.
Lua is easy to integrate with an existing system, and easy to extend. It is also fairly easy to create safe sandboxes in which user-supplied code can run without full access to the innards of your product.
If you have an API written does it make a difference? The person using the C-like API would only have to understand the difference between passing by value or reference.
Your core does one thing very good, so fine. Let it be that way. I think you should create an API based on std in/out, just like the way of good unix design. Then anyone can extend it in any language of choice.
Tcl was designed with the goal to allow scripting for C programs, so it would be much easier to implement.
http://en.wikipedia.org/wiki/Tcl#Interfacing_with_other_languages
I second Johan's idea. Although in past when I had to do something like this I stuck to
C language APIs and people were restricted to use C language only. But now I look at it,
I realize that it would have been more efficient if we would have done the way Johan describes
PS: And by coincidence it was a protocol testing app using packet sniffer
perl, sed, awk, lex, antler, ... These are languages I'm somewhat familiar with that I'd like to write something like this in. It depends on the data flow, though.
It's really hard to say what the correct thing to use is. Something that I don't think anyone else has mentioned is to keep in mind that the scripts will have bugs. It's very easy to design something like this in such a way that bugs in the scripts (especially run time errors) either just show up a "error in script" or kill the whole system.
You should keep that the scripts should be unit testable and that failures should be reproducible.
I don't think it matters what you do as long as you do one thing, drop the in-house language. It sounds like you choose to make C into a scripting language. One issue I see with this is it will look familiar to C programmers, but not be the same. I can't imagine you have mimicked the semantics of C that would make existing C programmers comfortable. And as you have mentioned, others will find it hard to learn.
The company I am working at have developed their own language. It uses XML for structure so parsing is easy. The language grows "as needed." Meaning if a feature is missing then it will be added. I'm pretty sure it went from an XML database to something that needed control flow. But my point is that if you aren't thinking about building it as a language, then you'll be limiting what users can do with it unintentionally.
Personally I've been looking at how I can get the company to start taking advantage of Lua. And specifically Lua for several reasons. Lua was developed as an extension language that was general purpose. It easily interfaces with the language, including Python and Ruby. It is small and simple for use by non-programmers (not really needed in your case). It is simple enough to replace XML, INI... for configuration settings and powerful enough to replace the need for another programming language.
http://www.lua.org/spe.html
Everyone knows how awesome C language is and how much it sucks in text processing tasks. Given these facts. Regex definitely must be part of ISO C. But it isn't. I don't understand why? Are there people who think its not essential?
Regular Expressions don't belong in the C language proper any more than a sound library, a graphics library, or an encryption library does. Doing so would reduce the general purpose nature of the language and greatly inhibit its use as a small and efficient embedded language.
The philosophy of C was to have a very small and efficient language keyword set with standardized libraries for the next layer of functionality. Since things like regex, graphics, sound, encryption, etc. don't have a single platform or standard they don't fit in with the standard C library.
They fit best as user libraries which they currently are.
Regex is defined as part of IEEE Std 1003.1:2001 (POSIX)
Here's a handly list of which headers are in which standard:
http://www.schweikhardt.net/identifiers.html
Because it is a library feature that would require standardizing on one of the regex languages. Standard bodies are commitee driven, not an easy task.
This document explains the rationalization of the standard: http://www.open-std.org/jtc1/sc22/wg14/www/docs/C99RationaleV5.10.pdf which might clarify why.
Another reason explained in the doc. is to keep the language simple.
There are quite a few downloads available, just use one.
Because regexes are not essential to a programming language. Handy? Yes, very much so, when you need them. Essential? No way.
Web developers will naturally consider regexes to be an essential feature of a language because they have to validate all that HTML form data. Developers whose experience is always with one of a few big-name relational database servers will consider SQL support to be essential. Those working in the scientific domain will require support for "big numbers" or tensors. GUI developers think a built-in GUI toolkit is essential. Some folks deal with XML all day and consider XML support to be essential.... etc. you get the idea. This list of "essentials" can get pretty big, and languages like Java have certainly taken the "kitchen sink" approach to their massive standard libraries. I appreciate that C is not a kitchen sink language in that sense.
Be careful not to assume that your favorite language feature is an essential feature for everyone else.
The point of C is to be small yet powerful. Since regular expressions are typically a large and complex topic, it belongs in a library. It is too bad though that the C committee doesn't "sponser" some well written, standard C, algorithms/data structure libraries. There is a plethora of them out there. I tend to stick with GNU "sponsored" libs whenever I can since they are available for most platforms even if they aren't necessarily the easiest or most efficient to use. They do strike a nice balance.
I have heard of Boost and ACE as two of the well known C++ libraries. What are the other good C/C++ libraries available?
Does Boost and ACE support session management for web applications written in C/C++?
EDIT: Ok I will try to be domain specific. I am looking for a C/C++ library which could help me maintain session state for a C++ based CGI web application.
When you're trying to build a web application in C++ I'd recommend Wt, a Qt-like framework for creating web applications in C++.
It handles sessions either in one process per session (when security matters) or multiple sessions per process.
You can either use the built-in webserver or use it with any webserver that supports FastCGI.
(Also, I'd recommend it over Boost.CGI as it seems to be maintained and feature-complete).
Depends if you are talking about general purpose or domain specific libraries. For general purpose Boost is best of breed (and don't forget about the good old STL), so I don't see the point of looking for something else that will cover much of the same ground, but is not as polished. As for domain specific you'd have to specify the domain :-)
If you're interested in C (not C++) as well, glib (the Gnome project's utility library) provides a number of useful data structures and constructs.
C++ has libraries for anything you could imagine, so the scope of your question is rather undefined. What interests you? Web applications, scientific programs, GUIs? Specify what you need exactly if you want a good answer.
Boost is a general-purpose library for relatively low-level things. It's rather complex and advanced though, so you should have a good grasp of C++ before you start with it. ACE is mainly for synchronization and communication between threads/processes/applications.
If web applications is what you need, I recommend you to strongly consider the language you're picking. C++ may not be the best direction to go here, unless you have very specific constraints that force your hand.
There is also GTK which is good if you need to have a gui or use unicode. (although c++0x should have better unicode support natively when the standard is complete).
Boost doesn't yet support sessions, but a CGI library has been proposed which should have sessions.
If you want to use C++ for web applications, consider using CGICC
Ok I will try to be domain specific. I am looking for a C/C++ library which could help me maintain session state for a C++ based CGI web application.
CppCMS?
Example of session management: http://cppcms.sourceforge.net/wikipp/en/page/tut_sessions
Reference: http://cppcms.sourceforge.net/wikipp/en/page/ref_cppcms_session_iface
Configuration: http://cppcms.sourceforge.net/wikipp/en/page/ref_config#sessions
Poco is an excellent C++ Library with data access, xml, networking, compression and crypto all wrapped up in once nice little package.
Boost evidently, QT for GUI (that's not clearly a library I know), Electronic Arts Standard Template Library and
Blitz++ if you want to do scientific computation :
Blitz++ is a C++ class library for
scientific computing which provides
performance on par with Fortran 77/90.
The C++ programming language offers
many features useful for tackling
complex scientific computing problems:
inheritance, polymorphism, generic
programming, and operator overloading
are some of the most important.
Unfortunately, these advanced features
came with a hefty performance
pricetag: until recently, C++ lagged
behind Fortran's performance by
anywhere from 20% to a factor of ten.
As a result, the adoption of C++ for
scientific computing has been slow.
Is there a way to soup up C++ so that
we can keep the advanced language
features but ditch the poor
performance? This is the goal of the
Blitz++ project: to develop techniques
which will enable C++ to rival -- and
in some cases even exceed -- the speed
of Fortran for numerical computing,
while preserving an object-oriented
interface. The Blitz++ Numerical
Library is being constructed as a
testbed for these techniques.
Recent benchmarks show C++ encroaching
steadily on Fortran's high-performance
monopoly, and for some benchmarks, C++
is even faster than Fortran! These
results are being obtained not through
better optimizing compilers,
preprocessors, or language extensions,
but through the use of template
techniques. By using templates
cleverly, optimizations such as loop
fusion, unrolling, tiling, and
algorithm specialization can be
performed automatically at compile
time.
Another goal of Blitz++ is to extend
the conventional dense array model to
incorporate new and useful features.
Some examples of such extensions are
flexible storage formats, tensor
notation and index placeholders.