I have only little background knowledge about Neural Networks (NN).
However, up to know I learnt, that training the network is the actual expensive part. Processing data by an already trained network is much cheaper/faster, ultimately.
Still, I'm not entirely sure what the expensive parts are within the processing chain. As far as I know, it's mostly Matrix-Multiplication for standard layers. Not the cheapest operation, but definitly doable. On top, there are are other layers, like max-pooling, or activation-functions at each node, which might have higher complexities. Are those the bottle-necks?
Now, I wonder if "simple" Hardware provided by Smartphones or even cheap stand-alone Hardware like Raspberry PIs are capable of utilizing a (convolutional-) Neuronal Networks to do, for example, Image Processing, like Object Detection. Of course, I mean doing the calculations on the device itself, not by transmitting the data to a second, powerful machine or even a cloud, which does the calculations, before sending back the results to the smartphone.
If so, what are the maximum Neurons such a Network should have (e.g. how many layers and how many neurons per layer), roughly estimated. And last, are there any good either projects, or librarys, using NNs for reduced simpler Hardware?
Current neural networks use convolutional layers, which perform a convolution on the input image. Also the high amount of parameters and dimensions is a real problem for low budget hardware. But anyways there are approaches that work on android for newer smartphones, like the SqueezeNet. Much of the work is actually done on gpus nowadays and so I am not sure if it works on a rasperry.
A better description than I could ever write on the topic can be found here: https://hackernoon.com/how-hbos-silicon-valley-built-not-hotdog-with-mobile-tensorflow-keras-react-native-ef03260747f3?gi=adb83ae18a85, where they actually built a neural network for a mobile phone. You can actually download the app and try it on your mobile phone if you have android or ios.
There are a lot of research going on in this area. There are roughly two lines of areas that deal with this:
Efficient network architectures
Some post-processing for an already trained model
You can have a look at ICNet Figure 1, where some architectures for fast inference for semantic segmentation are shown. Many of these models can be tweaked to do classification or other image processing tasks in real time. These models all have a low number of parameters compared to other networks and can be evaluated on embedded platforms.
For "post-hoc" optimizations you can look at TensorFlows Graph Transform Tool that do many of such for you: Graph Transform Tool
or maybe look into the paper by Song Han Deep Compression where many of these ideas are described. Song Han has also given many great lectures in this area and you can e.g. find one in the CS231n class at Stanford.
The speed of the inference phase depend on a lot of other things than the number of parameters or neurons. So I don't think there is a rule of thumb for saying how many neurons are the maximum.
Related
I will be in my final year (Electrical and Computer Engineering )the next semester and I am searching for a graduation project in embedded systems or hardware design . My professor advised me to search for a current system and try to improve it using hardware/software codesign and he gave me an example of the "Automated License Plate Recognition system" where I can use dedicated hardware by VHDL or verilog to make the system perform better .
I have searched a bit and found some youtube videos that are showing the system working ok .
So I don't know if there is any room of improvement . How to know if certain algorithms or systems are slow and can benefit from codesign ?
How to know if certain algorithms or systems are slow and can benefit
from codesign ?
In many cases, this is an architectural question that is only answered with large amounts of experience or even larger amounts of system modeling and analysis. In other cases, 5 minutes on the back of an envelop could show you a specialized co-processor adds weeks of work but no performance improvement.
An example of a hard case is any modern mobile phone processor. Take a look at the TI OMAP5430. Notice it has a least 10 processors, of varying types(the PowerVR block alone has multiple execution units) and dozens of full-custom peripherals. Anytime you wish to offload something from the 'main' CPUs, there is a potential bus bandwidth/silicon area/time-to-market cost that has to be considered.
An easy case would be something like what your professor mentioned. A DSP/GPU/FPGA will perform image processing tasks, like 2D convolution, orders of magnitude faster than a CPU. But 'housekeeping' tasks like file-management are not something one would tackle with an FPGA.
In your case, I don't think that your professor expects you to do something 'real'. I think what he's looking for is your understanding of what CPUs/GPUs/DSPs are good at, and what custom hardware is good at. You may wish to look for an interesting niche problem, such as those in bioinformatics.
I don't know what codesign is, but I did some verilog before; I think simple image (or signal) processing tasks are good candidates for such embedded systems, because many times they involve real time processing of massive loads of data (preferably SIMD operations).
Image processing tasks often look easy, because our brain does mind-bogglingly complex processing for us, but actually they are very challenging. I think this challenge is what's important, not if such a system were implemented before. I would go with implementing Hough transform (first for lines and circles, than the generalized one - it's considered a slow algorithm in image processing) and do some realtime segmentation. I'm sure it will be a challenging task as it evolves.
First thing to do when partitioning is to look at the dataflows. Draw a block diagram of where each of the "subalgorithms" fits, along with the data going in and out. Anytime you have to move large amounts of data from one domain to another, start looking to move part of the problem to the other side of the split.
For example, consider an image processing pipeline which does an edge-detect followed by a compare with threshold, then some more processing. The output of the edge-detect will be (say) 16-bit signed values, one for each pixel. The final output is a binary image (a bit set indicates where the "significant" edges are).
One (obviously naive, but it makes the point) implementation might be to do the edge detect in hardware, ship the edge image to software and then threshold it. That involves shipping a whole image of 16-bit values "across the divide".
Better, do the threshold in hardware also. Then you can shift 8 "1-bit-pixels"/byte. (Or even run length encode it).
Once you have a sensible bandwidth partition, you have to find out if the blocks that fit in each domain are a good fit for that domain, or maybe consider a different partition.
I would add that in general, HW/SW codesign is useful when it reduces cost.
There are 2 major cost factors in embedded systems:
development cost
production cost
The higher is your production volume, the more important is the production cost, and development cost becomes less important.
Today it is harder to develop hardware than software. That means that development cost of codesign-solution will be higher today. That means that it is useful mostly for high-volume production. However, you need FPGAs (or similar) to do codesign today, and they cost a lot.
That means that codesign is useful when cost of necessary FPGA will be lower than an existing solution for your type of problem (CPU, GPU, DSP, etc), assuming both solutions meet your other requirements. And that will be the case (mostly) for high-performance systems, because FPGAs are costly today.
So, basically you will want to codesign your system if it will be produced in high volumes and it is a high-performance device.
This is a bit simplified and might become false in a decade or so. There is an ongoing research on HW/SW synthesis from high-level specifications + FPGA prices are falling. That means that in a decade or so codesign might become useful for most of embedded systems.
Any project you end up doing, my suggestion would be to make a software version and a hardware version of the algorithm to do performance comparison. You can also do a comparison on development time etc. This will make your project a lot more scientific and helpful for everyone else, should you choose to publish anything. Blindly thinking hardware is faster than software is not a good idea, so profiling is important.
I need to implement a multi-agent system for an assignment. I have been brainstorming for ideas as to what I should implement, but I did not come up with anything great yet. I do not want it to be a traffic simulation application, but I need something just as useful.
I once saw an application of multiagent systems for studying/simulating fire evacuation plans in large buildings. Imagine a large building with thousands of people; in case of fire, you want these people to follow some rules for evacuating the building. To evaluate the effectiveness of your evacuation plan rules, you may want to simulate various scenarios using a multiagent system. I think it's a useful and interesting application. If you search the Web, you will find papers and works in this area, from which you might get further inspiration.
A few come to mind:
Exploration and mapping: send a team of agents out into an environment to explore, then assimilate all of their observations into consistent maps (not an easy task!)
Elevator scheduling: how to service call requests during peak capacities considering the number and location of pending requests, car locations, and their capacities (not too far removed from traffic-light scheduling, though)
Air traffic control: consider landing priorities (i.e. fuel. number of passengers, emergency conditions,etc.), airplane position and speed, and landing conditions (ie. number of runways, etc). Then develop a set of rules so that each "agent" (i.e. airplane) assumes its place in a landing sequence. Note that this is a harder version of the flocking problem mentioned in another reply.
Not sure what you mean by "useful" but... you can always have a look at swarmbased AI (school of fish, flock of birds etc.) Each agent (boid) is very very simple in this case. Make the individual agents follow each other, stay away from a predator etc.
Its not quite multi-agent but have you considered a variation on ant colony optimisation ?
http://en.wikipedia.org/wiki/Ant_colony_optimization_algorithms
I am developing a number of different mobile applications for a number of different mobile devices and I want to quickly test in a local development environment. I was wondering if it is possible (with some sort of hardware) to set up a local desktop CDMA / GSM base station for testing devices over a local personal cellular network. The range does not have to be very far. The alternative is purchasing a SIM card and plans for various carriers but not all carriers/network types are available in our area.
I'm sure I had seen some sort of desktop device that would let you setup local networks for development/testing purposes but can't seem to find it.
Thanks.
About the least expensive such device I know is the Agilent 8960. It's difficult to give an accurate price, since it depends on the options you choose, but you are likely to need to drop about $30,000 or more for a new one.
GSM, GPRS, EDGE, WCDMA, HSPA, CDMA 2000, 1x, EV-DO are all supported, although a box with all of those options in it will be well over the figure I quoted above!
The device has been around for a while though, so you may find something on eBay or via surplus sales and the like.
The upside is that it gives you an enormous amount of control over the cellular environment, and will let you do repeatable throughput tests (something that is really impossible on 'live' networks unless you use statistical techniques and many, many test runs) but the obvious downside is the price!
In a standard network setup you would need an antenna, a BTS, a BSC, a MSC, a GGSN and a SGSN to do data traffic, all horribly expensive and requiring expert knowledge to get the stuff up and running.
If you are interested in experiments then try OpenBSC altough it might be difficult to find BTS hardware.
If you want to buy actual products then have a look at IPaccess. They offer picocell hardware. I am not sure though if their BSC can work without an MSC and SGSN. But still expect a 5-digit price. Tecore also might be worth a visit.
Test and measurement equipment manufacturers might be an alternative as well. There you should check if you actually can branch out the data traffic into the internet or some test server if you need that.
If you want to do this for a living and not for fun, I would assume that simply buying SIMs plus data plans is the cheapest alternative.
You can roll your own cell network! with http://openbts.org/ but stills you need a development kit(Hardware) which is a little expensive. Or you can try to hack your own phone to use it as a radio which is really difficult but cheap.
The answer to this question may be of interest to you also, depending on what your application does:
Testing Mobile Sites
Essentially there are companies that offer a sort of virtual testing service, allowing you test phones with different location and operator combinations.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
Our brain consists of billions of neurons which basically work with all the incoming data from our senses, handle our consciousness, emotions and creativity as well as our hormone system, etc.
So I'm completely new to this topic but doesn't each neuron have a fixed function? E.g.: If a signal of strength x enters, if the last signal was x ms ago, redirect it.
From what I've learned in biology about our nerves system which includes our brain because both consist of simple neurons, it seems to me as our brain is one big, complicated computer.
Maybe so complicated that things such as intelligence and cognition become possible?
As the most complicated things about a neuron pretty much are the chemical aspects on generating an electric singal, keeping itself alive, and eventually segmenting itself, it should be pretty easy emulating some on a computer, or?
You won't have to worry about keeping your virtual neuron alive, or?
If you can emulate a single neuron on a computer, which shouldn't be too hard, could you theoretically emulate more than 1000 billions of them, recreating intelligence, cognition and maybe even creativity?
In my question I'm leaving out the following aspects:
Speed of our current (super) computers
Actually writing a program for emulating neurons
I don't know much about this topic, please tell me if I got anything wrong :)
(My secret goal: Make a copy of my brain and store it on some 10 million TB HDD and make someone start it up in the future)
A neuron-like circuit can be built with a handful of transistors. Let's say it takes about a dozen transistors on average. (See http://diwww.epfl.ch/lami/team/vschaik/eap/neurons.html for an example.)
A brain-sized circuit would require 100 billion such neurons (more or less).
That's 1.2 trillion transistors.
A quad-core Itanium has 2 billion transistors.
You'd need a server rack with 600 quad-core processors to be brain-sized. Think $15M US to purchase the servers. You'll need power management and cooling plus real-estate to support this mess.
One significant issue in simulating the brain is scale. The actual brain only dissipates a few watts. Power consumption is 3 square meals per day. A pint of gin. Maintenance is 8 hours of downtime. Real estate is a 42-foot sailboat (22 Net Tons of volume as ships are measured) and a place to drop the hook.
A server cage with 600 quad-core processors uses a lot more energy, cooling and maintenance. It would require two full-time people to keep this "brain-sized" server farm running.
It seems simpler to just teach the two people what you know and skip the hardware investment.
Roger Penrose presents the argument that human consciousness is non-algorithmic, and thus is not capable of being modeled by a conventional Turing machine-type of digital computer. If it's like that you can forget about building a brain with a computer...
Simulating a neuron is possible and therefore theoretically simulating a brain is possible.
The two things that always stump me as an issue is input and output though.
We have a very large number of nerve endings that all provide input to the brain. Without them the brain is useless. How can we simulate something as complicated as the human brain without also simulating the entire human body!?!
Output, once the brain has "dealt" with all of the inputs that it gets, what is then the output from it? How could you say that the "copy" of your brain was actually you without again hooking it up to a real human body that could speak and tell you?
All in all, a fascinating subject!!!!
The key problem with simulating neural networks (and human brain is a neural network) is that they function continuously, while digital computers function in cycles. So in a neural network different neurons function independently in parallel while in a computer you only simulate discrete system states.
That's why adequately simulating real neural networks is very problematic at the moment and we're very far from it.
Yes, the Blue Brain Project is getting close, and I believe Moore's Law has a $1000 computer getting there by 2049.
The main issue is that our brains are based largely on controlling a human body, which means that our language comprehension and production, the basis of our high-level reasoning and semantic object recognition, is strongly tied to its potential and practiced outputs to a larynx, tongue, and face muscles. Further, our reward systems are tied to signals that indicate sustenance and social approval, which are not the goals we generally want a brain-based AI to have.
An exact simulation of the human brain will be useful in studying the effects of drugs and other chemicals, but I think that the next steps will be in isolating pathways that let us do things that are hard for computers (e.g. visual system, fusiform gyrus, face recognition), and developing new or modifying known structures for representing concepts.
Short: yes we will surely be able to reproduce artificial brains, but no it maybe won't be with our current computers models (Turing machines), because we simply don't know yet enough about the brain to know if we need new computers (super-Turing or biologically engineered brains) or if current computers (with more power/storage space) are enough to simulate a whole brain.
Long:
Disclaimer: I am working in computational neuroscience research and I am interested both by the neurobiological side and the computational (artificial intelligence) side.
Most of the answers assume as true OP's postulate that simulating neurons is enough to save the whole brain state and thus simulate a whole brain.
That's not true.
The brain is more than just neurons.
First, there is the connectivity, the synapses, that is of paramount importance, even maybe more than neurons.
Secondly, there are glial cells such as astrocytes and oligodendrocytes that also possess their own connectivity and communication system.
Thirdly, neurons are heterogenous, which means that there is not just one template model of a neuron that we could just scale up to the required amount to simulate a brain, we also have to define multiple types of neurons and place them pertinently at the right places. Plus, the types can be continuous, so in fact you can have neurons that are half way between 3 different types...
Fourthly, we don't know much about the rules of brain's information processing and management. Sure, we discovered that the cerebellum works pretty much like an artificial neural network using stochastic gradient descent, and that the dopaminergic system works like TD-learning, but then we have no clue about the rest of the brain, even memory is out of reach (although we guess it's something close to a Hopfield network, but there's no precise model yet).
Fifthly, there are so many other examples from current research in neurobiology and computational neuroscience showing the complexity of brain's objects and networks dynamics that this list can go on and on.
So in the end, your question cannot be answered, because we simply do not know yet enough about the brain to know if our current computers (Turing machines) are enough to reproduce the complexity of biological brains to give rise to the full spectrum of cognitive functions.
However, biology field is getting closer and closer to computer science field, as you can see with biologically engineered viruses and cells that are programmed pretty much like you develop a computer program, and genetical therapies that basically re-engineer a living system based on its "class" template (the genome). So I dare to say that once we know enough about the brain's architecture and dynamics, the in-silico reproduction won't be an issue: if our current computers cannot reproduce the brain because of theoretical constraints, we will devise new computers. And if only biological systems can reproduce the brain, we will be able to program an artificial biological brain (we can already 3D-print functional bladders and skin and veins and hearts etc.).
So I would dare say (even if it can be controversial, this is here my own claim) that yes, artificial brains will surely be possible someday, but whether it will be as a Turing machine computer, a super-Turing computer or a biologically engineered brain remain to be seen depending on our progress in the knowledge of brain's mechanisms.
I don't think they are remotely close enough to understanding the human brain to even begin thinking about replicating it.
Scientists would have you think we are nearly there, but with regards to the brain we're not much further along than Dr. Frankenstein.
What is your goal? Do you want a program that can make intelligent decisions or a program that provides a realistic model of how the human brain actually works? Artificial intelligence can be approached from the perspective of psychology, where the goal is to simulate the brain and thereby get a better understanding of how humans think, or from the perspective of mathematics, optimization theory, decision theory, information theory, and computer science, in which case the goal is to create a program that is capable of making intelligent decisions in a computationally efficient manner. The latter, I would say is pretty much solved, although advances are definitely still being made. When it comes to a realistic simulation of the brain, I think we were only recently able to simulate a brain of cat semi-realistically; when it comes to humans, it would not be very computationally feasible at present.
Researchers far smarter than most recon so, see Blue Brain from IBM and others.
The Blue Brain Project is the first
comprehensive attempt to
reverse-engineer the mammalian brain,
in order to understand brain function
and dysfunction through detailed
simulations.
Theoretically the brain can be modeled using a computer (as software and hard/wetware are compatible or mutually expressible). The question isn't a theoretical one as far as computer science goes, but a philosophical one:
Can we model the (chaotic) way in which a brain develops. Is a brains power it's hardware or the environment that shapes the development and emergent properties of that hardware as it learns
Even more mental:
If I, with 100% accuracy modeled my own brain, then started the simulation. And that brain had my memories (as it has my brain's physical form) ... is it me? If not, what do I have that it doesn't?
I think that if we are ever in a position to emulate the brain, we should have been working on logical system based on biological principles with better applications than the brain itself.
We all have a brain, and we all have access to it's amazing power already ;)
A word of caution. Current projects on brain simulation work on a model of a human brain. Your idea about storing your mind on a hard-disk is crazy: if you want a replica of your mind you'll need two things. First, another "blank" brain. Second, devise a method to perfectly transfer all the information contained in your brain: down to the quantum states of every atom in it.
Good luck with that :)
EDIT: The dog ate part of my text.
What do you use?
Fast Artificial Neural Network Library (FANN) is a free open source neural network library, which implements multilayer artificial neural networks in C with support for both fully connected and sparsely connected networks. Cross-platform execution in both fixed and floating point are supported. It includes a framework for easy handling of training data sets. It is easy to use, versatile, well documented, and fast. PHP, C++, .NET, Ada, Python, Delphi, Octave, Ruby, Prolog Pure Data and Mathematica bindings are available.
FannTool A graphical user interface is also available for the library.
There are a lot of different network simulators dependant on how detailed you want to do your sim, and what kind of network you want to simulate.
NEURON and GENESIS are good if you want to simulate full biological networks (Which I'm geussing you probably don't) even down to the behaviour of dendrites etc.
NEST and SPLIT and some others are good for doing population simulations where you create the population on a node-by-node basis and see what the whole population does. This is pretty much the 'industry' standard approach, and is used a lot in research and commercial applications, so there are worth looking into. I know that IBM use SPLIT for some of their research.
MIIND is good if you want to use differential equations to model what a population would do, but this approach is relatively new and computationally expensive (if very cool).
Not sure if that is exactly what you wanted!
(N.B. if you google any of the names in caps along with the word "simulator" you will end up at the relevant web page =)
Whenever I've wanted to play around with any data mining algorithm quickly, I just load up Weka. It's pretty complex but it implements a lot of algorithms (including neural networks) with a lot of customizability. Plus, it has some visualizations for NNs.
It is old, but I have always used NeuroShell 2 when not using my own code. Unfortunately, it is not free. I think The newer NeuroShells are designed only for predicting stocks.
If you're looking to experiment with deep learning, you should look into
Theano
Pylearn2 (which is based on Theano)