I've been studying hierachial reinforcement learning problems, and while a lot of papers propose interesting ways for learning a policy, they all seem to assume they know in advance a graph structure describing the actions in the domain. For example, The MAXQ Method for Hierarchial Reinforcement Learning by Dietterich describes a complex graph of actions and sub-tasks for a simple Taxi domain, but not how this graph was discovered. How would you learn the hierarchy of this graph, and not just the policy?
In Dietterich's MAXQ, the graph is constructed manually. It's considered to be a task for the system designer, in the same way that coming up with a representation space and reward functions are.
Depending on what you're trying to achieve, you might want to automatically decompose the state space, learn relevant features, or transfer experience from simple tasks to more complex ones.
I'd suggest you just start reading papers that refer to the MAXQ one you linked to. Without knowing what exactly what you want to achieve, I can't be very prescriptive (and I'm not really on top of all the current RL research), but you might find relevant ideas in the work of Luo, Bell & McCollum or the papers by Madden & Howley.
This paper describes one approach that is a good starting point:
N. Mehta, S. Ray, P. Tadepalli, and T. Dietterich. Automatic Discovery and Transfer of MAXQ Hierarchies. In International Conference on Machine Learning, 2008.
http://web.engr.oregonstate.edu/~mehtane/papers/hi-mat.pdf
Say there is this agent out there moving about doing things. You don't know its internal goals (task graph). How do you infer its goals?
In way way, this is impossible. Just as it is impossible for me to know what goal you had mind when you put that box down: maybe you were tired, maybe you saw a killer bee, maybe you had to pee....
You are trying to model an agent's internal goal structure. In order to do that you need some sort of guidance as to what are the set of possible goals and how these are represented by actions. In the research literature this problem has been studied under the terms "plan recognition" and also with the use of POMDP (partially observable markov decision process), but both of these techniques assume you do know something about the other agent's goals.
If you don't know anything about its goals, all you can do is either infer one of the above models (This is what we humans do. I assume others have the same goals I do. I never think, "Oh, he dropped his laptop, he must be ready to lay an egg" cse, he's a human.) or model it as a black box: a simple state-to-actions function then add internal states as needed (hmmmm, someone must have written a paper on this, but I don't know who).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
I know that it's still an open problem so I don't expect to see complete answers here. I just want to find some approaches to solve the next problem:
I have a model (assume that is's bot's memory), and different words are associated with different objects in the model. Speaking with the bot is like executing sql-queries with a DB. Language is a very hard formalizable protocol. And we can't just write a million lines of code to implement some real language. But I believe that it's absolutely possible to implement some self-learning mechanism. How can it be implemented? Is it possible to implement learning "from scratch" or "from few basic words"? Just want to hear your ideas.
Actually, English is a very strict language and it's one of the easiest languages for experimenting with AI. Many other languages allow you to change the order of words (for example). And in some cases changed order can change the whole meaning or just add some intonation. I really don't have any ideas how to teach a bot for these things.
The first step, in taking this game to the next level, is ...
...to have a very clear view of prior art!
(and pardon me to say, the question doesn't suggest that you have such an extensive insight into the matter [and you're not alone, count me in ;-)])
Even, and maybe in particular, if your intention is to apply completely novel techniques and models, it seems important to review the literature on current and past practices. Aside from possibly identifying elements that may be adapted or reused in a new implementation, a survey of the domain will provide an keen understanding of the nature of the problem[s].
I've personally tried -on various and multiple occasions!- either the naive approach or the sophomoric approach to tackling broadly-defined problems. With the naive approach, one has but a very slight idea of the true nature and scope of the problem. The sophomoric sees us better equipped with domain knowledge and also with related tools, but this can also be misleading because without a deeper understanding, we tend to mis-read/mis-understand new material offered to us and also misuse some of the tools (a bit like the the fellow who's "good with a hammer" for whom many things look like a nail...)
It is particularly easy to make these mistakes in the field of NLP. That's because
Common sense seems to be all is required: after all a child, who's native tongue is English understands subtleties like
"He's not really an expert"
"He's really not an expert" (small wink at the OP's reference to the ordering of word in the English language)
We live in such exciting times, technology and knowledge wise: Processing power, programming language and tools, mathematical techniques, availability of affordable corpora... to name a few of these things that make this moment in time so special.
Far from me the idea of discouraging you in your chat-bot endeavor, I just hope that this long and generic exposé will encourage to look-before-you-leap, as this will truly save you time in the long run, I think in two ways:
provide you some frames of references (again, even if your intention is to "think outside these boxes")
maybe entice you to redefine the problem, for example by limiting it to particular domains of conversation (sports, or health, or life at a particular university campus...) or by focusing on a particular aspect of the problem (semantic awareness, smooth, natural sounding grammar, use of colloquial forms...)
Good luck ;-)
Check out MegaHAL's implementation for some ideas. We've used a variant of this bot for ages in an IRC channel of ours, and he does on occasion appear to be the intelligent mixture of many of our dominant personalities.
You "train" the bot -
each time the bot answer, you rank (or the tester) the answer - if the answer is good/logical - give high rank, if the answer is bad... low/negative rank.
use the ranking in the future to choose the answer, and this is how the bot learns...
There's a great description of Eliza in Paradigms of AI Programming. You should be able to implement a simple Eliza bot in a few days of work.
This isn't a learning algorithm, but it's surprising how realistic answers can be from something so simple.
You can create your own chat bot on BOT libre, http://www.botlibre.com.
The bots learns, can be trained, can be scripted, and your can program them, or let them program themselves.
Thew site supports embedding your bot on your own site, has REST API access, Android, IRC, Twitter. Free hosting, even for commercial bots.
AIML from the AliceBot project may help you out. It's a whole XML schema (if that doesn't put you off) for the branch of AI its concerned with.
An example from Wikipedia:
<category>
<pattern>WHAT IS YOUR NAME</pattern>
<template>My name is <bot name="name"/>.</template>
</category>
RebbeccaAIML is one quite well documented implementation.
Thinking in regards to Sliverlight, I would like to know where would be good places to go to get a refresher on 3d space, transforms, matrix manipulation, and all that good stuff.
There's always The Bible
It is expensive and very heavy on the theory, so there's also the cheaper Bible Lite
As pointed out in some comments and additional answers, it is definitely worth noting that this book is now quite dated. However, in the context of the original question, there's not really been any change in the low-level principles of linear algebra in a seriously long time.
If you are looking to learn about high-level graphics programming this may well not be the first book for you. But if you like to know about "the guts-of-the-machine" and the underlying maths -- perhaps you are the kind of person that thinks folk should learn C :-) -- then go nuts.
It's not a place, but I've found 3D Programming for Windows by Charles Petzold excellent. It covers everything you ask about and is focused specifically on WPF/silverlight.
Of course Petzold (as usual) is able to communicate the important concepts beautifully.
Think I may have found it myself. Was looking at:
http://msdn.microsoft.com/en-us/library/cc189037(VS.95).aspx
and
http://www.c-sharpcorner.com/UploadFile/mgold/TransformswithGDIplus09142005064919AM/TransformswithGDIplus.aspx
Free graphics algorithms can be found in the
comp.graphics.algorithms faq
As previously mentioned you should really learn linear algebra, here are some great video lectures about it, MIT Linear Alebgra Video Lectures.
Any linear algebra textbook should provide the math refresher; there's a fairly good one available online at Linear Algebra textbook home page.
Personally I think that although the bible (by Foley & Van Damn that is) was the greatest book for its time, but it is somewhat outdated.
I would suggest 'Advanced animation & Rendering techniques' by Alan and Mark Watt.
The only problem with this book is that it gives you a good understanding almost about every broad aspect in CG but it assumes you have some familiarity with it, and does not explain it all the way.
You can always have a look in the Bib and find enhanced articles and books about each subject you are interested in depth.
If you want further on once you have more understanding, or if you want to dive into the world of computer graphics and the use of GPU I suggest to have a look at the three 'GPU Gems'.
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
Locked. This question and its answers are locked because the question is off-topic but has historical significance. It is not currently accepting new answers or interactions.
Back when I was at school, I remember tinkering with a Mac game where you programmed little robots in a sort of pseudo-assembler language which could then battle each other. They could move themselves around the arena, look for opponents in different directions, and fire some sort of weapon. Pretty basic stuff, but I remember it quite fondly, even if I can't remember the name.
Are there any good modern day equivalents?
I used to have a lot of fun coding my own robot with Robocode in college.
It is Java based, the API is detailled and it's pretty easy to get a challenging robot up and running.
Here is an example :
public class MyFirstRobot extends Robot {
public void run() {
while (true) {
ahead(100);
turnGunRight(360);
back(100);
turnGunRight(360);
}
}
public void onScannedRobot(ScannedRobotEvent e) {
fire(1);
}
}
Just found Light Bot. Program your robot to move around and perform tasks to complete a puzzle. Even includes subroutines. Program the bot by dragging tiles into slots. The game is very polished.
Update Lightbot is now the most recent version of the game, and has versions specifically designed for kids ages 4-8 or ages 9+ (with no upper limit) and also features kind of an if
screen of lightbot 1 http://www.lostateminor.com/wp-content/uploads/2008/10/light-bot.jpg
I think the original game was called Core Wars (this Wikipedia article contains a lot of interesting links); there still seem to be programs and competitions around, for example at corewars.org. I never had the time to look into these games, but they seem like great fun.
In the flash game Manufactoria you "program" a factory by laying out the conveyor belts and switches in a way that's very similar to a FSM, but more powerful. This game is really great. Give it a try, especially if you're into formal languages and automata!
Manufactoria screen shot http://www.tomdalling.com/wp-content/uploads/manufactoria-bubble-sort.png
A game in which you have to graphically construct and train artificial neural networks in order to control a bug is Bug Brain.
Bug Brain screen shot http://www.infionline.net/~wtnewton/oldcomp/bugbrain.jpg
The game in question was definitely Robowar for the Mac. My son had a lot of fun with it and went on to program real robots.
As mentioned earlier by Proud, there is a wiki page for it:
http://en.wikipedia.org/wiki/RoboWar
Although there has not been a lot of activity surrounding the game over the last few years, there was a tournament held recently, and there is a yahoo email group.
If you want to step away from your keyboard, Wizards of the Coast relased a game called RoboRally that is a combative programming board game.
http://www.wizards.com/roborally/
http://www.pythonchallenge.com/
highly addictive, and a great way to learn python
I'm surprised that Space Chem isn't mentioned yet. Programming with symbols, but programming nevertheless.
http://spacechemthegame.com/
Another good one is CEEBot. It teaches C / Java style programming in a fun, robot-programming kind of game. It is aimed at 10-15 year olds, but it is a good one.
Colobot
It's usually easy for new programmers to pick up on languages like C++ when you have a strong understanding of Java basics. Colobot allows you to program automated craft using a language almost identical to Java and to move, sense, and manipulate their environment in order to accomplish missions on a dangerous planet.
I was also keen on these kind of games. One modern example which I have used is http://www.robotbattle.com/. There are various others - for example the ones listed at http://www.google.com/Top/Games/Video_Games/Simulation/Programming_Games/Robotics/
Core Wars is the classic, of course. But Rocky's Boots is another one. Imagine! There was a time (1982) when you could sell a commercial game based on logic gates!
If you are willing to look at single player games like Light Bot and Manufactoria then I highly recommend RoboZZle. It has conditional commands which include function calls. This allows for complex stack manipulation. There are thousands of user created puzzles from pathetically obvious to mind blowing enigmas. They have recently added support for smartphones.
I also think The Codex of Alchemical Engineering is worth a look.
I think .NET Terrarium is one of the best 'learn-to-program' games for the .NET platform.
I like Ruby Warrior. It is still somewhat under development, but it is a great game with a clever interface.
Core Wars
There's also mySQLgame, I found it pretty amusing (shortly after finding out I suck).
Here's what Casual Gameplay has to say about it.
Kara is about programming a bug(!) coming up in various versions, e.g. Finite State Machine, Java, Turing Machine, Multithreading
Kara http://www.swisseduc.ch/compscience/karatojava/kara/icons/kara-worldeditor.gif
Planetwars is a game specifically written for Google Ai Contest, bots are controlling fleets for conquering planets, they support many languages
I'd say the most famous programming game there has been is the core wars. I don't know if you can still find active "rings" although there was a lot when I tried it some time ago (4 or 5 years).
I've never heard or Core Wars before, but it looks interesting. I do have to vouch for RoboCode, though. That's fun and challenging, especially if you have a group of people competing against either other.
http://en.wikipedia.org/wiki/Hacker_(computer_game)
http://en.wikipedia.org/wiki/Hacker_2
There is also a great hacking game the name of which I simply cannot remember. Hrm.
Matt, I think the game you're referring to is CRobots (or one of its clones, perhaps -- my first contact was with PRobots, in Pascal, if I remember correctly). It was a lot of fun.
While it was more logic than programming per se, one I really enjoyed back in elementary school was Rocky's Boots. It had sensors, AND gates, OR gates, NOT gates, wires, timers, and all sorts of other stuff. Fantastic program for teaching a kid logic.
Go to the link and you can still play it!
Carnage Heart for PlayStation was fun. It would let you program little mechs to do battle using a flow diagram.
I have to give a shout out to RobotWar which was the first programming "game" that I played way back in the Apple II days. It was written by Silas Warner of Castle Wolfenstein fame.
I got myself addicted to uplink a few months ago. It's not really coding based, more hacking. It's still fun and super geeky.
Although not strictly programming-based, I enjoyed a lot Robot Odyssey, a game where you wired logic gates to sensors and motors in a robot, to make it move and react to environment, to get out of a city, escaping obstacles. I played in on Apple //e, it was one of the best games on this computer (with Lode Runner! :-)).
You must be thinking of RoboWar. Oh how lovely it is.
Still exists, though the community is slowly dying.
http://robowar.sourceforge.net/RoboWar5/index.html
http://tech.groups.yahoo.com/group/robowar/