I was recently studying Uninformed Search from here. In the case of Depth First Search, it is given that the space taken by the fringe is O(b.m), but I am unable to figure out how(I could not find proof of this anywhere online). Any help or pointers to specific material would be much appreciated.
A depth-first tree search needs to store only a single path from the root to a leaf node, along with the remaining unexpanded sibling nodes for each node on the path.
Once a node has been expanded, it can be removed from memory as soon as all its descendants has been fully expanded.
So that, for a state space with branching factor band maximum depth m, depth-first search requires storage of only O(b*m).
ref. Russel & Norvig, Artificial Intelligence, Figure 3.16, p87
The Depth First Search (DFS) algorithm has to store few nodes in the fridge because it processes the lastly added node first (Last In First Out), which results in a space complexity of O(bd). Thus, for a depth of d it has to store at most the b children of the d nodes above.
The Breadth First Search (BFS) algorithm, however, gets the firstly inserted node first (First In First Out). Because of this, it has to keep track of all the children nodes it encounters, which results in a space complexity of O(b^d).
Thus, for a depth of d it has to store the children and the children's children, etc., resulting in the exponential growth.
Related
I had a question regarding BFS. After expanding nodes whether it be a graph or a tree what path will BFS take as the solution from the starting point to the goal? Does it take into account the cost of moving from one node to the other and takes the lowest cost route or does it take the path with the least amount of nodes needed to get to the goal?
The classical breadth-first search algorithm does not take the weight of the edges into account. In each iteration, you simply put the direct reachable neighbors of the current node in the queue without any checks. You can find the shortest path between two nodes A and B in the sense of a minimal amount of "steps" that are necessary to reach node B from node A.
PREMISE
So lately i have been thinking of a problem that is common to databases: trying to optimize insertion, search, deletion and update of data.
Usually i have seen that most of the databases nowadays use the BTree or B+Tree to solve such a problem, but they are usually used to store data inside the disk and i wanted to work with in-memory data, so i thought about using the AVLTree (the difference should be minimal because the purpose of the BTrees is kind of the same of the AVLTree but the implementation is different and so are the effects).
Before continuing with the reasoning behind this i would like to get in a deeper level of what i am trying to solve.
So in a modern database data stored in a table with a PRIMARY KEY which tends to be INDEXED (i am not very experienced in indexing so what i will say is basic reasoning i put into this problem), usually the PRIMARY KEY is an increasing number (even though nowadays is a bad practice) starting from 1.
Using normally an AVLTree should be more then enough to solve the problem cause this particular tree is always balanced and offers O(log2(n)) operations, BUT i wanted to reach this on a deeper level trying to optimize it even more then needed.
THEORY
So as the title of the question suggests i am trying to optimize the AVLTree merging it with a Btree.
Basically every node of this new Tree is lets say an array of ten elements every node as also the corresponding height in the tree and every element of the array is ordered ascending.
INSERTION
The insertion initally fills the array of the root node when the root node is full it generates the left and right children which also contains an array of 10 elements.
Whenever a new node is added the Tree autorebalances the nodes based on the first key of the vectors of the left and right child using also their height (note that this is actually how the AVLTree behaves but the AVLTree only has 2 nodes and no vector just the values).
SEARCH
Searching an element works this way: staring from the root we compare the value we are searching K with the first and last key of the array of the current node if the value is in between, we know that it surely will be in the array of the current node so we can start using a binarySearch with O(log2(n)) complexity into this array of ten elements, otherise we go on the left if the key we are searcing is smaller then the first key or we go to the right if it is bigger.
DELETION
The same of the searching but we delete the value.
UPDATE
The same of the searching but we update the value.
CONCLUSION
If i am not wrong this should have a complexity of O(log10(log2(10))) which is always logarithmic so we shouldn't care about this optimization, but in my opinion this could make the height of the tree so much smaller while providing also quick time on the search.
B tree and B+ tree are indeed used for disk storage because of the block design. But there is no reason why they could not be used also as in-memory data structure.
The advantages of a B tree include its use of arrays inside a single node. Look-up in a limited vector of maybe 10 entries can be very fast.
Your idea of a compromise between B tree and AVL would certainly work, but be aware that:
You need to perform tree rotations like in AVL in order to keep the tree balanced. In B trees you work with redistributions, merges and splits, but no rotations.
Like with AVL, the tree will not always be perfectly balanced.
You need to describe what will be done when a vector is full and a value needs to be added to it: the node will have to split, and one half will have to be reinjected as a leaf.
You need to describe what will be done when a vector gets a very low fill-factor (due to deletions). If you leave it like that, the tree could degenerate into an AVL tree where every vector only has 1 value, and then the additional vector overhead will make it less efficient than a genuine AVL tree. To keep the fill-factor of a vector above a minimum you cannot easily apply the redistribution mechanism with a sibling node, as would be done in B-trees. It would work with leaf nodes, but not with internal nodes. So this needs to be clarified...
You need to describe what will be done when a value in a vector is updated. Of course, you would insert it in its sorted position: but if it becomes the first or last value in that vector, this may violate the order with regards to left and right children, and so also there you may need to define more precisely the algorithm.
Binary search in a vector of 10 may be overkill: a simple left-to-right scan may be faster, as CPUs are optimised to read consecutive memory. This does not impact the time complexity, since we set that the vector size is limited to 10. So we are talking about doing either at most 4 comparisons (3-4 on average depending on binary search implementation) or at most 10 comparisons (5 on average).
If I am not wrong this should have a complexity of O(log10(log2(n))) which is always logarithmic
Actually, if that were true, it would be sub-logarithmic, i.e. O(loglogn). But there is a mistake here. The binary search in a vector is not related to n, but to 10. Also, this work comes in addition to finding the node with that vector. So it is not a logarithm of a logarithm, but a sum of logarithms:
O(log10n + log210) = O(log n)
Therefore the time complexity is no different than the one for AVL or B-tree -- provided that the algorithm is completed with the missing details, keeping within the logarithmic complexity.
You should maybe also consider to implement a pure B tree or B+ tree: that way you also benefit from some of the advantages that neither the AVL, nor the in-between structure has:
The leaves of the tree are all at the same level
No rotations are needed
The tree height only changes at one spot: the root.
B+ trees provide a very fast mean for iterating all values in their order.
I'm trying to solve the following problem: "Given a sorted (increasing order) array with unique integer elements, write an algorithm to create a BST with minimal height."
The given answer takes the root node to be the middle of the array. While doing this makes sense to me intuitively, I'm trying to prove, rigorously, that it's always best to make the root node the middle of the array.
The justification given in the book is: "To create a tree of minimal height, we need to match the number of nodes in the left subtree to the number of nodes in the right subtree as much as possible. This means that we want the root node to be the middle of the array, since this would mean that half the elements would be less than the root and half would be greater."
I'd like to ask:
Why would any tree of minimal height be one where the number of nodes in the left subtree be as equal as possible to the number of nodes in the right subtree? (Or, do you have any other way to prove that it's best to make the root node the middle of the array?)
Is a tree with minimal height the same as a tree that's balanced? From a previous question on SO, that's the impression I got, (Visualizing a balanced tree) but I'm confused because the book specifically states "BST with minimal height" and never "balanced BST".
Thanks.
Source: Cracking the Coding Interview
The way I like to think about it, if you balance a tree using tree rotations (zig-zig and zig-zag rotations) you will eventually reach a state in which the left and right subtree differ by at most height of one. It is not always the case that a balanced tree must have the same number of children on the right and the left; however, if you have that invariant(same # of children on each side), you can reach a tree that is balanced using tree rotations)
Balance is defined arbitrarily. AVL trees define it in such as way that no subtree of the tree has a children whose heights differ by more than one. Other trees define balance in different ways, so they are not the same distinction. They are inherently related yet not exactly the same. That being said, a tree of minimal height will always be balanced under any definition since balancing exists to maintain a O(log(n)) lookup time of the BST.
If I missed anything or said anything wrong, feel free to edit/correct me.
Hope this helps
Why would any tree of minimal height be one where the number of nodes
in the left subtree be as equal as possible to the number of nodes in
the right subtree?
There can be a scenario where in minimal height tree which are ofcourse balanced can have different number of node count on left and right hand side. BST worst case traversal is O(n) in case if it is sorted and in minimal height trees the complexity for worst case is O(log n).
*
/ \
* *
/
*
Here you can clearly see that left node count and right nodes are not equal though it is a minimal height tree.
Is a tree with minimal height the same as a tree that's balanced? From a previous question on SO, that's the impression I got, (Visualizing a balanced tree) but I'm confused because the book specifically states "BST with minimal height" and never "balanced BST".
Minimal height tree is balanced one, for more details you can take a look on AVL trees which are also known as height-balanced trees. While making BST a height balanced tree you have to perform rotations (LR, RR, LL, RL).
For an assignment I need to implement Dijkstra's algorithm using a priority queue implemented as a min heap. I already have an implementation in which I use a distance table and an unordered array of the unprocessed vertexes. (As described towards the bottom of the page here).
An example input file is given below, where the first line is the number of vertexes and each line after is: source, destination, weight.
3
1 2 3
3 2 1
1 3 1
My initial thought was to treat each edge in the graph as a node in the heap, but that isn't right because one of the test files has 15677372 edges and I can't even make an array that big without an immediate segfault. After implementing the method with the array of unprocessed vertexes it seems that I need to somehow replace that array with the heap, but I'm not sure how to go about that.
Could anyone point me in the right direction?
Typically, in Dijkstra's algorithm, your priority queue will hold the nodes in the graph and the best estimated distance to that node so far. The standard technique is to enqueue all nodes in the graph into the priority queue initially at distance ∞, then to use the queue's decrease-key operation to lower them as needed.
This is often completely infeasible from a memory standpoint, so an alternate interpretation is to keep the queue empty initially and then seed it with the start node at distance 0. Every time you process a node, you then update the distance estimate for each of its adjacent nodes. You do this as follows:
If the node is already in the priority queue, you can use decrease-key to lower the node's distance.
If the node is not already in the priority queue, then you insert it at the required priority.
This keeps the priority queue small for graphs that aren't absurdly dense.
Hope this helps!
When searching in a tree, my understanding of uniform cost search is that for a given node A, having child nodes B,C,D with associated costs of (10, 5, 7), my algorithm will choose C, as it has a lower cost. After expanding C, I see nodes E, F, G with costs of (40, 50, 60). It will choose 40, as it has the minimum value from both 3.
Now, isn't it just the same as doing a Greedy-Search, where you always choose what seems to be the best action?
Also, when defining costs from going from certain nodes to others, should we consider the whole cost from the beginning of the tree to the current node, or just the cost itself from going from node n to node n'?
Thanks
Nope. Your understanding isn't quite right.
The next node to be visited in case of uniform-cost-search would be D, as that has the lowest total cost from the root (7, as opposed to 40+5=45).
Greedy Search doesn't go back up the tree - it picks the lowest value and commits to that. Uniform-Cost will pick the lowest total cost from the entire tree.
In a uniform cost search you always consider all unvisited nodes you have seen so far, not just those that are connected to the node you looked at. So in your example, after choosing C, you would find that visiting G has a total cost of 40 + 5 = 45 which is higher than the cost of starting again from the root and visiting D, which has cost 7. So you would visit D next.
The difference between them is that the Greedy picks the node with the lowest heuristic value while the UCS picks the node with the lowest action cost. Consider the following graph:
If you run both algorithms, you'll get:
UCS
Picks: S (cost 0), B (cost 1), A (cost 2), D (cost 3), C (cost 5), G (cost 7)
Answer: S->A->D->G
Greedy:
*supposing it chooses the A instead of B; A and B have the same heuristic value
Picks: S , A (h = 3), C (h = 1), G (h = 0)
Answer: S->A->C->G
So, it's important to differentiate the action cost to get to the node from the heuristic value, which is a piece of information that is added to the node, based on the understanding of the problem definition.
Greedy search (for most of this answer, think of greedy best-first search when I say greedy search) is an informed search algorithm, which means the function that is evaluated to choose which node to expand has the form of f(n) = h(n), where h is the heuristic function for a given node n that returns the estimated value from this node n to a goal state. If you're trying to travel to a place, one example of a heuristic function is one that returns the estimated distance from node n to your destination.
Uniform-cost search, on the other hand, is an uninformed search algorithm, also known as a blind search strategy. This means that the value of the function f for a given node n, f(n), for uninformed search algorithms, takes into consideration g(n), the total action cost from the root node to the node n, that is, the path cost. It doesn't have any information about the problem apart from the problem description, so that's all it can know. You don't have any information that can help you decide how close one node is to a goal state, only to the root node. You can watch the nodes expanding here (Animation of the Uniform Cost Algorithm) and see how the cost from node n to the root is used to choose which nodes to expand.
Greedy search, just like any greedy algorithm, takes locally optimal solutions and uses a function that returns an estimated value from a given node n to the goal state. You can watch the nodes expanding here (Greedy Best First Search | Quick Explanation with Visualization) and see how the return of the heuristic function from node n to the goal state is used to choose which nodes to expand.
By the way, sometimes, the path chosen by greedy search is not a global optimum. In the example in the video, for example, node A is never expanded because there are always nodes with smaller values of h(n). But what if A has such a high value, and the values for the next nodes are very small and therefore a global optimum? That can happen. A bad heuristic function can cause this. Getting stuck in a loop is also possible. A*, which is also a greedy search algorithm, fixes this by making use of both the path cost (which implies knowing nodes already visited) and a heuristic function, that is, f(n) = g(n) + h(n).
It's possible that to this point, it's still not clear to you HOW uniform-cost knows there is another path that looks better locally but not globally. It should become clear after telling you that if all paths have the same cost, uniform cost search is the same thing as the breadth-first search (BFS). It would expand all nodes just like BFS.
UCS cares about history,
Greedy does not.
In your example, after expanding C, the next node would be D according to the UCS. Because, it's our history. UCS can't forget the past and remember that the total cost of D is much lower than E.
Don't be Greedy. Be UCS and if going back is really a better choice, don't afraid of going back!