I came from a C-Java-C# world... where the parentheses were VIP citizens and types were loved by everyone : )
Sometimes I need to develop some piece of code to automatize a repetitive task, convert some complex frame, order an input file, send programatic request to a server... you know what I mean?
But sometimes I wonder how easier could be write the same in some language like Python... or these DoItSimple languages : ) -with all respect.
So which language should I use for this kind of tasks?
Thank you very much in advance!
You should really investigate one of the more popular scripting languages, like Python/Ruby/Perl. Any one of those will set you in good stead for what you're trying to achieve. Which one of those you should use is a matter for heated debate around the water cooler - or more likely, the coffee machine.
That said, there's something to be said for learning tools like 'awk' and 'sed', not to mention some of the extended capabilities in 'vim' for achieving day-to-day tasks
For a long time, you will always be faster with the languages you know. However, if you just want to experience something new and eventually have another useful tool in your belt, I'd try python. It's very handy, and is not too far of a jump from the C# world (if you've ever done BASIC, it feels VERY similar).
BTW - AlBlue's suggestion about awk/sed/vim(macros/etc) is A+. I will often hop onto a Linux box at work to handle any repetitive file tasks (on Windows you can get cygwin, we just don't have it available at work).
I love PHP and perl for these things (having grown out of sed and awk, for the most part), but if don't rule out the bash shell (I use Cygwin's on windows systems), and even the Windows scripting language (yes, batch files still exist) can do more than I once thought.
Why don't you just take a list of the languages you find attractive and write some small pieces of code. Then you'll be able to select the one that is "yours".
I Say Ruby or Python
My favorite is Ruby for its high-purity object model and generally awesome syntax and power.
However, Python has the same general capabilities and is about twice as popular.
Traditionally, Perl is used for sysadmin automation and other scripting tasks, plus obviously sed, awk, and sh.
But languages like Perl and PHP started as ad-hoc scripting interpreters, whereas Python and Ruby were real object-oriented languages. Perl and PHP have since grown object-oriented revisions, but now they have all the complexity of the first "nice try" coupled with the "oh we should make it a real language" extensions.
Ruby, in particular, is the product of a real language designer who combined the ultra-pure object model of Smalltalk with the populist syntax of Perl.
If you are coming from the C world, I would say Lua because not only is Lua a fine scripting language, but it also comes with a super API for binding it to C programs, which means in the long run you'll be able to use the same technology to script your C programs.
Lua also stands well on its own merits; I have written elsewhere about why I prefer Lua to Python.
Google for something like "perl vs ruby vs python" or "best general purpose scripting language" ... You should find some comparisons. I personally like ruby most cause it lacks the nasty $'s in front of all variables. If you're familiar with c# and the .net framework, IronRuby and IronPython would give you access to all the classes you already know.
I'll probably get downvoted for recommending this, but I suggest starting with "writing C in Perl", and working (or not) towards whatever level of idiomatic Perl that suits you.
My reason for suggesting this is that even if your Perl code is structured much like C, and even if you don't avail yourself of all of Perl's syntactic sugar, you'll still see an increase in productivity. Starting with C-style Perl just allows you to make use of idioms that you are already familiar with (ie C-style for loops vs Perl's foreach) while learning the language.
As I know, PHP is one of the scripting language to do tasks such as: order an input file, send programatic request to a server... and PHP is C-like syntax, and said to be the easiest to learn(you don't even have to learn before you using it, you just use it while learning it). But PHP is not a general purpose scripting language.
Python is a general purpose scripting language and has a strong type checking, bug it's not C-like, I'm afraid you would not like it.
I myself use PHP for the most time for text processing, network(socket, HTTP...) tesing. ^_^
These are all fine scripting languages, and IMHO it probably won't make a huge difference which one you choose. I hope I won't be downvoted for injecting a realistic answer in what is mostly a religious argument. It's basically like when you agonize over which college to go to, and someone points out that as long as they're both good schools, it's not likely to make a big difference in your life.
I use Perl for writing CLI programs because it's the first scripting language I learned (20+ years ago when it was the lingua franca for writing CGI scripts on the nascent WWW). I now use PHP for writing web apps because my last job used it for our intrantet web pages, and it has lots of nice built-in features for web and SQL programming; but I'm sure that with the write CPAN modules I could be just as productive with Perl.
If there's a language that's popular with other programmers you work with and respect, that's probably one of the best reasons to choose it, since you'll have a convenient support community. SO is nice, but even better is being able to learn from a local expert.
Related
I need some tips on efficiently programming in Maple (as in the computer algebra software). By efficient I do not mean making my program/code more efficient (performancewise) but rather efficiently working with Maple codes. This is ironic, but I am quite spoiled by software and tools developed for programming in languages such as C/C++ e.g. : IDE, debuging tools (watches, breakpoints, etc. etc.), compilers and the language structure itself (object orientedness, more flexibility with procedures, overloading, passing by reference or value etc.). In this sense I feel more comfortable programming in C/C++ (I use visual studio) or even in script languages like python (no real debugging but I feel more in control and discipline working).
Maple on the other hand, is one of the few languages (somehow I do have similar problems with other CAS but maybe to a lesser degree) where I feel I do not have discipline. I tried just simply working with some text file (most people call it .mpl files) and just read the text file in maple and just load this text file from maple e.g. read("some.mpl"), this allows me to diff my changes when using a version control like svn or git (the .mws or other maple file format aren't so version control friendly). On the fly programming can be really cumbersome the IDE often takes a long time loading and feels very heavy. So I just write my codes in Notepad++ (working in windows) and costumized a "Run" in Notepad++ menu to call cmaple (the console version of maple) and read my .mpl files on the fly. This are the few steps I took to help me work more efficiently. But I think I can do more and I would appreciate any tips. Right now for instance, I feel like maple (unlike in programming languages in C, python, java etc.) does not work very well when one has many procedures. In C/C++ or any C-like languages, I can practically inline all my code as procedure and call them individually , sometimes you get weird behaviour if you do this in Maple. Sometimes even, functions and maple commands behave very differently when they are within a procedure (I have experience such a thing when for instance wanting to do SVD or LU decomposition within a procedure). One has also to be careful when working with nested loops (e.g. print command have levels where they do not appear if you reach a certain level of your nested loop in a procedure and other things that I am sure I still do not know) and very many other things that gets irritating in the long run.
So I think I have an idea what tools/softwares to use to make programming easier when working in maple. But I still do not know how to program properly and more efficiently in maple. To the initiated C programmer many things are very unintuitive and even annoying in maple. Is there any other things that I should be cautious about when programming in maple (e.g. unusual behaviours when reading several .mpl files in another .mpl file, different types of global/local variables, saving all of the variables in a worksheet, backward compatibility between maple versions etc.). It is probably better to also ask for a reference/book that does provide these advises. Just for the record: I have been programming in maple (seriously) for more than 2 years and I still do not feel as comfortable as in the first 6 months when I first learned to program in C.
If you're looking for a written guide, the Maple Programming Guide is probably the best resource out there at this moment for getting started (though if you've been programming in Maple for a few years, you may have moved past it already). It is distributed by Maplesoft and is available in their documentation center. I've also heard good things about 'Understanding Maple'.
That said, there's probably much more that you can learn from anything tagged Maple on StackExchange or more importantly from the Maple user community at
mapleprimes.com. You cover a variety of topics in your post, so what you might try is to simply add a short code example at MaplePrimes in order to further explore the topic with the Maple experts there.
From my experience, the most efficient flow is to solve mathematically heavy parts in Maple and then use these results for producing C code you can embed everywhere, from for C++ numeric simulator of your scientific project to code fragment in your JavaScript computer game.
The strongest sides of Maple are symbolic mathematics and code generation, it nicely complements your expertise in C++ or say even Java without ambitions to replace these languages where they are better.
We have a very versatile terminal/sniffer application which can do all sorts of things with TCP, UDP and serial connections.
We are looking to make it extensible -- i.e, allow people to write their own protocol parsers, highlighters, etc.
We created a C-like language for extending the product, and then discovered that for some coders, this presents a steep learning curve.
We are now pondering the question: Should we stick to C or go with something like Ruby or Lua?
C is beautiful for low-level stuff (like parsing binary data), because it supports pointers. But for exactly that reason, it can be tough to learn.
Ruby (etc) are easy to learn, but don't have pointers, so anything that has to do with parsing binary data gets ugly very fast.
What do you think? For extending a product that parses binary data -- Ruby/Lua or C/C++?
Would be great if you could give some background when you respond -- especially if you've done something similar.
Wireshark, the "world's foremost network protocol analyzer", is also a packet sniffer/analyzer, formerly also called Ethereal. It uses Lua to enable writing custom dissectors and taps, see the manual.
However, note that I have not used it, so I cannot tell how nice/effective/easy to learn the API is.
Like TCL, Lua was designed to be tightly integrated with an application. Personally, I find Lua's syntax and idioms to be much easier to deal with than TCL.
Lua is easy to integrate with an existing system, and easy to extend. It is also fairly easy to create safe sandboxes in which user-supplied code can run without full access to the innards of your product.
If you have an API written does it make a difference? The person using the C-like API would only have to understand the difference between passing by value or reference.
Your core does one thing very good, so fine. Let it be that way. I think you should create an API based on std in/out, just like the way of good unix design. Then anyone can extend it in any language of choice.
Tcl was designed with the goal to allow scripting for C programs, so it would be much easier to implement.
http://en.wikipedia.org/wiki/Tcl#Interfacing_with_other_languages
I second Johan's idea. Although in past when I had to do something like this I stuck to
C language APIs and people were restricted to use C language only. But now I look at it,
I realize that it would have been more efficient if we would have done the way Johan describes
PS: And by coincidence it was a protocol testing app using packet sniffer
perl, sed, awk, lex, antler, ... These are languages I'm somewhat familiar with that I'd like to write something like this in. It depends on the data flow, though.
It's really hard to say what the correct thing to use is. Something that I don't think anyone else has mentioned is to keep in mind that the scripts will have bugs. It's very easy to design something like this in such a way that bugs in the scripts (especially run time errors) either just show up a "error in script" or kill the whole system.
You should keep that the scripts should be unit testable and that failures should be reproducible.
I don't think it matters what you do as long as you do one thing, drop the in-house language. It sounds like you choose to make C into a scripting language. One issue I see with this is it will look familiar to C programmers, but not be the same. I can't imagine you have mimicked the semantics of C that would make existing C programmers comfortable. And as you have mentioned, others will find it hard to learn.
The company I am working at have developed their own language. It uses XML for structure so parsing is easy. The language grows "as needed." Meaning if a feature is missing then it will be added. I'm pretty sure it went from an XML database to something that needed control flow. But my point is that if you aren't thinking about building it as a language, then you'll be limiting what users can do with it unintentionally.
Personally I've been looking at how I can get the company to start taking advantage of Lua. And specifically Lua for several reasons. Lua was developed as an extension language that was general purpose. It easily interfaces with the language, including Python and Ruby. It is small and simple for use by non-programmers (not really needed in your case). It is simple enough to replace XML, INI... for configuration settings and powerful enough to replace the need for another programming language.
http://www.lua.org/spe.html
After five years of professional Java (and to a lesser extent, Python) programming and slowly feeling my computer science education slip away, I decided I wanted to broaden my horizons / general usefulness to the world and do something that feels more (to me) like I really have an influence over the machine. I chose to learn C and Unix programming since I feel like that is where many of the most interesting problems are.
My end goal is to be able to do this professionally, if for no other reason than the fact that I have to spend 40-50 hours per week on work that pays the bills, so it may as well also be the type of coding I want to get better at. Of course, you don't get hired to do things you haven't done before, so for now I am ramping up on my own.
To this end, I started with K&R, which was a great resource in part due to the exercises spread throughout each chapter. After that I moved on to Computer Systems: A Programmer's Perspective, followed by ten chapters of Advanced Programming in the Unix Environment. When I am done with this book, I will read Unix Network Programming.
What I'm missing in the Stevens books is the lack of programming problems; they mainly document functionality and provide examples, with a few end-of-chapter questions following. I feel that I would benefit much more from being challenged to use the knowledge in each chapter a la K&R. I could write some test program for each function, but this is a less desirable method as (1) I would probably be less motivated than if I were rising to some external challenge, and (2) I will naturally only think to use the function in the ways that have already occurred to me.
So, I'd like to get some recommendations on how to practice. Obviously, my first choice would be to find some resource that has Unix programming challenges. I have also considered finding and attempting to contribute to some open source C project, but this is a bit daunting as there would be some overhead in learning to use the software, then learning the codebase. The only open-source C project I can think of that I use regularly is Python, and I'm not sure how easy that would be to get started on.
That said, I'm open to all kinds of suggestions as there are likely things I haven't even thought of.
Reinvent a lot of the core Unix utilities. Most of these were (and still are) written in C, so they are a good way to start off learning. Depending on your skill, pick harder or easier utilities to copy.
Try writing your own malloc. You'll learn a lot about Unix and a lot of C programming as well.
Google for computer science operating system courses and do the projects there. Many schools have these projects on public websites so you could get everything you need. Here is a link to Purdue's site. Give the shell project a shot; it was difficult, but really educational.
Here are a few stackoverflow postings discussing C/Unix programming books. Their main claim to fame is extensive fan-out to other resources.
Practice some of the idioms (understand the ins and outs of pointers etc) and pick projects that help with that. The third item in the list has an answer (disclaimer, I wrote it) with a bunch of high-level C idioms where an idiomatic C program would differ from Java.
Learning XLib or Win32 GUI programming with C is probably less useful as almost anything is better than C for GUI programming and wrapping a core C engine with another language is generally much easier - unless you're really concerned with speed. As a starting point, concentrate on 'systems' programming applications, which is where you are most likely to get real mileage from C. Learn how the C interface of one of the scripting languages like Tcl or Python works.
Writing some bit-twiddly and pointer heavy code (e.g. picking apart network packets or interpreting a protocol), or server-side items using sockets will take you right into C's core competencies. If you've got some of the WRS books, try making pthreads work (IIRC UNP2 has a big section about pThreads). For a contrast, write something that uses non-blocking I/O. This stuff is C's home turf and you'll get a good working knowledge of C by doing this. Troll through the RFCs for some ideas of network protocols to implement.
What are good Linux/Unix books for an advancing user?
What are some good resources for learning C beyond K&R
Resources for learning C program design
What’s a good way to start learning about Data Structures & Algorithms?
Algorithms in C
Are you open to book suggestions? Although it is a bit dated (where "a bit" perhaps is a huge understatement), Maurice Bach's "The Design of the Unix Operating System" is a great book. It belongs next to K&R on your bookshelf.
You might try working your way through all the examples in the book Software Tools (Amazon). Much of it is pretty pedestrian (right-justify text, de-tabify, etc.), but it's a great introduction to the Unix philosophy and basic C programming.
This is pretty interesting. As we know that Unix is designed by using the C language, it may not difficult to learn. If you are using the Windows OS, you can use a "Windows services for Unix" application to practice your programs. If you are using Unix, then we can use editors like vi.
I would recommend one thing highly.
Try and re-write all usual Linux command lines tools like ls, mkdir, cd etc.
You'll gain a lot of knowlege about programming and Linux both at the same time.
Pick the commands from the easiest, say "time" and work all the way up to the more complicated ones.
The best way to consolidate your learnings it to practise. So just choose a kind of application that interest you and start developing it (for example, a network client/server simple application).
Try to test most of the Unix APIs (files, sockets, etc.) to see how they work. You could for example get an Unix programming book, follow its chapters and test on your application everything you read, by creating your own functions. On that way, you can start to develop your own function library to be used in further projects.
Write a webserver.
Make it multi-threaded.
Have it support a new scripting language you develop (a la PHP, etc.)
Allow uploads from authenticated users.
Write a plugin for your favorite tool (i.e. integrate with SVN to give a webview).
I know there is a way to call Perl routines from C. As shown here
http://search.cpan.org/dist/perl/pod/perlcall.pod#NAME
But, still I do not see a widespread use of this by C programmers. Has any one used this ...ever? or any idea what are the reasons that it is not used so much?
Well, the most well known example of a C program embedding perl is Apache and mod_perl, which lets people access the Apache API through Perl.
You probably don't see many programs embedding perl because they don't need it as a feature. Why don't more C programs send email? :)
Its usually done in reverse. That is, the scripting language coordinates things, and the C program does the processing.
The idea is that the code that coordinates is easy to modify, and the code that does the processing runs fast.
The thing to do would probably be to spawn the perl interpreter via the "system" call from C to do something. Probably not quite what you have in mind though.
There are two reasons why one may call a perl function from C: extending and embedding.
In the former case, it isn't all that uncommon actually, but it's rather invisible for outsiders.
What I think your question really is though is "why don't people embed perl more often?" There are a number of reasons for that, it being far more difficult than it should be is the most important one IMHO (see perlcall, perlembed, perlguts and perlapi).
Why would I slow down my C program by calling Perl? And I say that as a confirmed Perler. Now, calling into C from Perl, that makes sense (if you need the extra speed).
When one develops in C, it is because they want to trade development time for execution speed.
When one develops in Perl, it is because they want to trade execution speed for development time.
One situation where this is used is in a plug-in or scripting architecture. For example, the irssi IRC client is written in C but supports scripting with Perl.
In my experience however, building irssi with Perl enabled is troublesome and prone to failure. I have had instances where the program wouldn't build at all with Perl bindings enabled (some compile or link failure), and also where the program would compile but would immediately fail on startup with a runtime error. These sorts of problems might explain why it's not terribly popular—calling one language from another is almost never as easy as it is in .Net.
Many programmers know how to obfuscate C very effectively, without needing to resort to Perl.
I've been working on a C program which does quite a lot of string manipulation, and very often needs to be tweaked and recompiled for some sort of special case processing. I've been thinking that embedding some scripting language with good string manipulation support might make sense for the project.
What language would provide the best string manipulation support while being easy to embed in a C program?
For some extra background...
Performance is pretty important (especially startup time)
Needs easily be compiled on multiple platforms (Linux, Solaris, Win32 (ideally with MinGW), Darwin)
Needs to be a language which will still be around in 5 years time
I've looked a little at Python (perhaps too heavy weight?) and Lua (perhaps not focused on string manipulation?) but don't really know enough about them or what other choices might be out there.
I've never regretted using Lua.
It's very easy to embed in your application. In fact, now I usually don't write C applications, i just write C libraries and control them from Lua.
Text manipulation isn't its best feature, but it's certainly far better than C alone. And the LPEG library makes building parsers almost trivially easy, putting any regex to shame (but still has a couple of regex-like syntaxes if you prefer them).
Lua stands head and shoulders above other choices.
... best string manipulation support while being easy to embed?
Lua is designed to be embedded in C; the API is clear and easy to use; the documentation is terrific.
Some other responses have denigrated Lua's string capabilities. I think they're underestimating Lua.
Lua's string capabilities actually find a sweet spot between "just concatenation" and the full complexity of regular expressions. String formatting capability is very strong, and accumulating strings through "buffers" or tables is simple and efficient.
String scanning is, in my opinion, one of the best parts of the design. It doesn't have "or" patterns but otherwise gives you a large fraction of what you get from regular expressions, including a very powerful and elegant "capture" function. For example, I can convert a string to hex by capturing every single character and applying a function to it:
s:gsub('.', function(c) return string.format("%02x", string.byte(c)) end)
Or I can escape non-alphanumeric, non-space characters into octal:
s:gsub('[^%w%s]', function(c) return string.format([[\%03o]], string.byte(c)) end)
Some of the features on display here:
The escape character for string scanning is %, which is different from the escape character for string quoting, which is \. This decision is brilliant and should win an award by itself :-)
There are multiple mechanisms for quoting literal strings, including [[...]] in which no characters have to be escaped. If you want to generate or match strings with backslashes in them (like LaTeX for example), this is a godsend.
If you want the full power of a context-free parser, you can always use LPEG, a library written by one of Lua's designers.
Performance is pretty important (especially startup time)
Lua consistently wins performance awards. Startup is lightning fast: the whole system (including compiler, library, garbage collector, and runtime system) fits in 150KB. To avoid pause times, Lua provides incremental garbage collection. See also SO question Why is Lua faster than other scripting languages?
You can make startup even faster by precompiling your scripts, but I've never found it necessary to do this—and because compiled code (as opposed to source code) is not portable, precompilation usually creates more headache than it solves.
Needs easily be compiled on multiple platforms
Lua compiles using pure ANSI C and does not even require POSIX. I have a version running on my PalmOS PDA.
Needs to be a language which will still be around in 5 years time.
Lua has been around since 1993. Moreover, the two members of the team who provide the most support are tenured professors at PUC-Rio. Lua is their livelihood. Finally, the whole system is only 17,000 lines of code. If Rio fell off the map tomorrow, anybody with a good undergraduate compiler course could pick the system up and maintain it. There would be plenty of volunteers.
I've looked a little at Python and Lua but don't really know enough about them
See SO question Which game scripting language is better to use: Lua or Python?.
People have been embedding tcl in larger projects for what seems like ages. It's been a while since I've had to use tcl for anything...
One of the things that sets tcl apart from other programming languages is that everything is a string.
And for your reference, here's the tcl documentation on string functions.
tcl might be easier to embed than perl, but I do have to agree #Matthew Scharley's reasoning. Also, tcl isn't exactly known for it's performance, but maybe that's changed in recent years.
Anyway, here is the tcl wiki link on embedding tcl in C applications, and a relevant quote from the page:
"How do I embed a Tcl interpreter in my existing C (or C++) application?" is a very frequently-asked question. It's straightforward, certainly far easier than doing the same with Perl or, in general, Python; moreover, this sort of "embeddability" was one of the original goals for Tcl, and many, many projects do it. There are no complete discussions of the topic available, but we can give an overview here. (RWT 14-Oct-2002)
Another alternative might be to go with Lua, as you mentioned, while extending it with another C string library of your choice (Google turns up The Better String Library, for instance).
Once you've compiled Lua into your application, you can "extend" C functions to Lua's interpreter. Or maybe the built-in string functions are adequate for you.
You certainly have a few options.
We looked at both Python and Lua for scripting for a .NET product. The goal was to provide some scripability for end users. The decision came down to Python because the powers-that-be preferred anything with Microsoft support to everything else. My choice was for Lua.
There's a good survey paper on the relative merits of the embedding APIs of various scripting languages:
H. Muhammad and R. Ierusalimschy. C APIs in extension
and extensible languages. Journal of Universal Computer
Science, 13(6):839–853, 2007.
Looking at combining both excellent string manipulation and an excellent embed API, I would suggest, in order:
Ruby: Excellent string support, including syntax support for regex. Well-designed embed API, very easy to use.
Lua: I'm not sure how its string support is, but its supposed to be a great language for embedding.
Python: Less easy to embed, slightly harder to use string features than Ruby. But it has Pyrex, so that might be an easier way to embed it.
PHP: Nasty API, nasty language. The embed SAPI is really a second-class citizen, but it does work. There are a lot of string manipulation functions. Still, I wouldn't recommend it.
Perl: Nasty to embed (so far as I've heard), string support could be better.
I can't comment about TCL, but I hear its designed for embedding.
Python is not heavyweight at all! It's quite simple to embed (here's the official guide, but you can find many tutorials as well), very powerful, great for string processing, and a pleasant and easy language to use overall. It has a huge user community and support base, which is a bonus.
Python has also been embedded into a large number of real-life applications. One cool example I can think of immediately is the Civilization IV game, most of which runs on Python scripts on top of a C++ API.
Some people may disagree but Sara Goleman has published a great book on extending and embedding PHP. Which is becoming one of the most widely used languages around... :)
PHP String support isn't as great as say Perl, but it's very usable.
Did I mention it's written in C?
</my2cents>
Perl. Its (original) reason for being is string manipulation.