I am looking for good academic references on how to benchmark programs. There seems to be a lot of lore in benchmarking, but I haven't seen many references that explain what a good benchmark is, what a bad one is, and how to write one.
Thanks.
Academically speaking, a relevant article is "Statistically rigorous Java performance evaluation" from OOPSLA 2007 (which you can find from Google Scholar); while focused on Java, it contains general lessons on benchmarking, and the content about Java generalizes nicely to most languages running on some virtual machine and simply using garbage collection. Finally, they summarize the statistics knowledge needed for analyzing the results.
Additionally, here is a framework from Google:
http://code.google.com/p/caliper/
And here their Wiki discusses some criteria for a good benchmark:
http://code.google.com/p/caliper/wiki/JavaMicrobenchmarkReviewCriteria
Related
I want to do some automatic story generation demonstration and the approach I am taking is using AI planning. I have been reading several relevant papers and have figured out that PDDL is perhaps the most widely used language to form the planning problem. I have been looking at the syntax and several example codes to learn how to use it.
The part where I am stuck is how to get the planner to work. I have found out some popular planners (fast-forward, MBP, IPP) but am not being able to make them work, using the instructions even from the sources itself.
I am using Gnome Terminal on Ubuntu 13.04.
I am very new to planning and this may be a very naive question but I assure that I have been searching for more than 3-4 days without any luck. Also, suggestions on using some other planning system are welcome.
If you are using Linux then I strongly suggest to use Fast Downward (it has its own web page - just google it). First of all, it is currently one of the best-known planning systems in the AI planning community and, further, it is really easy to get it to run. Well, you still need half an hour or so, but there is an easy-to-follow step-by-step description telling you where to check out the code and which commands you need to run.
It has also implemented most of the known planning heuristics that are required to solve problems fast or even optimal (planning requires search and heuristics make the search "goal-oriented" rather than blind and, if the heuristic is admissible and/or monotone (depending on the kind of search algorithm that is chosen -- see fast forward and pddl: is the computed solution the best?), it guarantees to find optimal solutions).
Concerning literature, I suggest to read/skip through the following two journal articles:
Porteous, J.; Cavazza, M.; and Charles, F. 2010. Applying planning to interactive storytelling: Narrative control using state constraints. ACM Trans. Intell. Syst. Tech. 10:1-10:21.
http://dl.acm.org/citation.cfm?id=1869399
Patrik Haslum. "Narrative Planning: Compilations to Classical Planning". Journal of AI Research, vol. 44, p. 383-395, 2012
http://www.jair.org/papers/paper3602.html
Well, both MBP and IPP are really, really old systems. If you're just looking for a ready-made planner to use in an off-the-shelf manner, I'd suggest you to follow the pointers leading to the authors (and software) that took part in the last International Planning Competition (2011):
http://www.plg.inf.uc3m.es/ipc2011-deterministic/ParticipatingPlanners.html
I'm considering doing something with Domain Specific Languages for my undergraduate project. My one problem is I can't really find any interesting examples that I can root around in. Does anyone have any good examples of DSELs (preferably open source)?
Also, one area I would love to look at is solving/addressing concurrency problems (coroutines etc) with DSEL's. Are there any good examples that anyone uses of this in DSELs? If this is a stupid application of DSELs please explain why...
Another potential area to explore would database programming. Again is this a stupid area to explore with DSEL's. For example, would adding some crazy database manipulation syntax to C# say be a good project to undertake?
EDIT: General languages I would be looking at implementing in would be Java, Python, Scala, C# etc. Probably not C++ or C.
Linda implementations can be considered as eDSLs. STM implementations like CL-STM are certainly eDSLs.
Unrelated to concurrency, but extremely useful are embedded Prolog implementations, there are plenty of them for Scheme, Lisp and Clojure. Parsing eDSLs had been mentioned already - and their patriarch Parsec definitely worth digging into.
EDIT: with your list of implementation languages you're missing the most interesting eDSL opportunities. The most powerful and flexible eDSLs are made with metaprogramming. Scala-style (or even Haskell-style) eDSLs are based on high order functions, i.e., on mini-interpreters. They're more complicated in design, much less flexible and limited to the syntax of your host language.
boost::spirit if you're after C++ is an interesting example. Quote:
Spirit is a set of C++ libraries for
parsing and output generation
implemented as Domain Specific
Embedded Languages (DSEL)...
(I have no idea what you mean by "solving concurrency" though. I don't see how you can solve "concurrency problems" in general, or how a DSEL could help.)
I'm starting to study machine learning and bayesian inference applied to computer vision and affective computing.
If I understand right, there is a big discussion between
classical IA, ontology, semantic web researchers
and machine learning and bayesian guys
I think it is usually referred as strong AI vs weak AI related also to philosophical issues like functional psychology (brain as black box set) and cognitive psychology (theory of mind, mirror neuron), but this is not the point in a programming forum like this.
I'd like to understand the differences between the two points of view. Ideally, answers will reference examples and academic papers where one approach get good results and the other fails. I am also interested in the historical trends: why approaches fell out of favour and a newer approaches began to rise up. For example, I know that Bayesian inference is computationally intractable, problem in NP, and that's why for a long time probabilistic models was not favoured in information technology world. However, they've began to rise up in econometrics.
I think you have got several ideas mixed up together. It's true that there is a distinction that gets drawn between rule-based and probabilistic approaches to 'AI' tasks, however it has nothing to do with strong or weak AI, very little to do with psychology and it's not nearly as clear cut as being a battle between two opposing sides. Also, I think saying Bayesian inference was not used in computer science because inference is NP complete in general is a bit misleading. That result often doesn't matter that much in practice and most machine learning algorithms don't do real Bayesian inference anyway.
Having said all that, the history of Natural Language Processing went from rule-based systems in the 80s and early 90s to machine learning systems up to the present day. Look at the history of the MUC conferences to see the early approaches to information extraction task. Compare that with the current state-of-the-art in named entity recognition and parsing (the ACL wiki is a good source for this) which are all based on machine learning methods.
As far as specific references, I doubt you'll find anyone writing an academic paper that says 'statistical systems are better than rule-based systems' because it's often very hard to make a definite statement like that. A quick Google for 'statistical vs. rule based' yields papers like this which looks at machine translation and recommends using both approaches, according to their strengths and weaknesses. I think you'll find that this is pretty typical of academic papers. The only thing I've read that really makes a stand on the issue is 'The Unreasonable Effectiveness of Data' which is a good read.
As for the "rule-based" vs. " probabilistic" thing you can go for the classic book by Judea Pearl - "Probabilistic Reasoning in Intelligent Systems. Pearl writes very biased towards what he calls "intensional systems" which is basically the counter-part to rule-based stuff. I think this book is what set off the whole probabilistic thing in AI (you can also argue the time was due, but then it was THE book of that time).
I think machine-learning is a different story (though it's nearer to probabilistic AI than to logics).
Has anyone worked with the programming language Church? Can anyone recommend practical applications? I just discovered it, and while it sounds like it addresses some long-standing problems in AI and machine-learning, I'm skeptical. I had never heard of it, and was surprised to find it's actually been around for a few years, having been announced in the paper Church: a language for generative models.
I'm not sure what to say about the matter of practical applications. Does modeling cognitive abilities with generative models constitute a "practical application" in your mind?
The key importance of Church (at least right now) is that it allows those of us working with probabilistic inference solutions to AI problems a simpler way to model. It's essentially a subset of Lisp.
I disagree with Chris S that it is at all a toy language. While some of these inference problems can be replicated in other languages (I've built several in Matlab) they generally aren't very reusable and you really have to love working in 4 and 5 for loops deep (I hate it).
Instead of tackling the problem that way, Church uses the recursive advantages of lamda calaculus and also allows for something called memoization which is really useful for generative models since your generative model is often not the same one trial after trial--though for testing you really need this.
I would say that if what you're doing has anything to do with Bayesian Networks, Hierarchical Bayesian Models, probabilistic solutions to POMDPs or Dynamic Bayesian Networks then I think Church is a great help. For what it's worth, I've worked with both Noah and Josh (two of Church's authors) and no one has a better handle on probabilistic inference right now (IMHO).
Church is part of the family of probabilistic programming languages that allows the separation of the estimation of a model from its definition. This makes probabilistic modeling and inference a lot more accessible to people that want to apply machine learning but who are not themselves hardcore machine learning researchers.
For a long time, probabilistic programming meant you'd have to come up with a model for your data and derive the estimation of the model yourself: you have some observed values, and you want to learn the parameters. The structure of the model is closely related to how you estimate the parameters, and you'd have to be pretty advanced knowledge of machine learning to do the computations correctly. The recent probabilistic programming languages are an attempt to address that and make things more accessible for data scientists or people doing work that applies machine learning.
As an analogy, consider the following:
You are a programmer and you want to run some code on a computer. Back in the 1970s, you had to write assembly language on punch cards and feed them into a mainframe (for which you had to book time on) in order to run your program. It is now 2014, and there are high-level, simple to learn languages that you can write code in even with no knowledge of how computer architecture works. It's still helpful to understand how computers work to write in those languages, but you don't have to, and many more people write code than if you had to program with punch cards.
Probabilistic programming languages do the same for machine learning with statistical models. Also, Church isn't the only choice for this. If you aren't a functional programming devotee, you can also check out the following frameworks for Bayesian inference in graphical models:
Infer.NET, written in C# by the Microsoft Research lab in Cambridge, UK
stan, written in C++ by the Statistics department at Columbia
You know what does a better job of describing Church than what I said? This MIT article: http://web.mit.edu/newsoffice/2010/ai-unification.html
It's slightly more hyperbolic, but then, I'm not immune to the optimism present in this article.
Likely, the article was intended to be published on April Fool's Day.
Here's another article dated late march of last year. http://dspace.mit.edu/handle/1721.1/44963
I've been working on a project, which is a combination of an application server and an object database, and is currently running on a single machine only. Some time ago I read a paper which describes a distributed relational database, and got some ideas on how to apply the ideas in that paper to my project, so that I could make a high-availability version of it running on a cluster using a shared-nothing architecture.
My problem is, that I don't have experience on designing distributed systems and their protocols - I did not take the advanced CS courses about distributed systems at university. So I'm worried about being able to design a protocol, which does not cause deadlock, starvation, split brain and other problems.
Question: Where can I find good material about designing distributed systems? What methods there are for verifying that a distributed protocol works right? Recommendations of books, academic articles and others are welcome.
I learned a lot by looking at what is published about really huge web-based plattforms, and especially how their systems evolved over time to meet their growth.
Here a some examples I found enlightening:
eBay Architecture: Nice history of their architecture and the issues they had. Obviously they can't use a lot of caching for the auctions and bids, so their story is different in that point from many others. As of 2006, they deployed 100,000 new lines of code every two weeks - and are able to roll back an ongoing deployment if issues arise.
Paper on Google File System: Nice analysis of what they needed, how they implemented it and how it performs in production use. After reading this, I found it less scary to build parts of the infrastructure myself to meet exactly my needs, if necessary, and that such a solution can and probably should be quite simple and straight-forward. There is also a lot of interesting stuff on the net (including YouTube videos) on BigTable and MapReduce, other important parts of Google's architecture.
Inside MySpace: One of the few really huge sites build on the Microsoft stack. You can learn a lot of what not to do with your data layer.
A great start for finding much more resources on this topic is the Real Life Architectures section on the "High Scalability" web site. For example they a good summary on Amazons architecture.
Learning distributed computing isn't easy. Its really a very vast field covering areas on communication, security, reliability, concurrency etc., each of which would take years to master. Understanding will eventually come through a lot of reading and practical experience. You seem to have a challenging project to start with, so heres your chance :)
The two most popular books on distributed computing are, I believe:
1) Distributed Systems: Concepts and Design - George Coulouris et al.
2) Distributed Systems: Principles and Paradigms - A. S. Tanenbaum and M. Van Steen
Both these books give a very good introduction to current approaches (including communication protocols) that are being used to build successful distributed systems. I've personally used the latter mostly and I've found it to be an excellent text. If you think the reviews on Amazon aren't very good, its because most readers compare this book to other books written by A.S. Tanenbaum (who IMO is one of the best authors in the field of Computer Science) which are quite frankly better written.
PS: I really question your need to design and verify a new protocol. If you are working with application servers and databases, what you need is probably already available.
I liked the book Distributed Systems: Principles and Paradigms by Andrew S. Tanenbaum and Maarten van Steen.
At a more abstract and formal level, Communicating and Mobile Systems: The Pi-Calculus by Robin Milner gives a calculus for verifying systems. There are variants of pi-calculus for verifying protocols, such as SPI-calculus (the wikipedia page for which has disappeared since I last looked), and implementations, some of which are also verification tools.
Where can I find good material about designing distributed systems?
I have never been able to finish the famous book from Nancy Lynch. However, I find that the book from Sukumar Ghosh Distributed Systems: An Algorithmic Approach is much easier to read, and it points to the original papers if needed.
It is nevertheless true that I didn't read the books from Gerard Tel and Nicola Santoro. Perhaps they are still easier to read...
What methods there are for verifying that a distributed protocol works right?
In order to survey the possibilities (and also in order to understand the question), I think that it is useful to get an overview of the possible tools from the book Software Specification Methods.
My final decision was to learn TLA+. Why? Even if the language and tools seem better, I really decided to try TLA+ because the guy behind it is Leslie Lamport. That is, not just a prominent figure on distributed systems, but also the author of Latex!
You can get the TLA+ book and several examples for free.
There are many classic papers written by Leslie Lamport :
(http://research.microsoft.com/en-us/um/people/lamport/pubs/pubs.html) and Edsger Dijkstra
(http://www.cs.utexas.edu/users/EWD/)
for the database side.
A main stream is NoSQL movement,many project are appearing in the market including CouchDb( couchdb.apache.org) , MongoDB ,Cassandra. These all have the promise of scalability and managability (replication, fault tolerance, high-availability).
One good book is Birman's Reliable Distributed Systems, although it has its detractors.
If you want to formally verify your protocol you could look at some of the techniques in Lynch's Distributed Algorithms.
It is likely that whatever protocol you are trying to implement has been designed and analysed before. I'll just plug my own blog, which covers e.g. consensus algorithms.