So I have a multi-layered neural network that succeeded in learning AND, OR, NOT and XOR. I have a doubt with back propagation.
I'm using the sigmoid function, so to determine the gradient of the error it goes something like this:
(Target - Output) * Output * (1 - Output)
But my question is, if I have a target of 1, and my output is 0, it would result in something like
(1 - 0) * 0 * (1 - 0), so it will tell me my error is 0, even though it's (I think) 1. Is it really supposed to be 0 or is it something I should account for when evaluating the gradient? Can somebody explain me what's the logic behind that 0? is it a local minimum of the function or something like that?
If you think about it, it's gonna be that way even if your target is 1 and the output is 1.
The reason this doesn't happen is because you won't usually get a real 1 or 0 from a properly functioning backpropagation network, because you are using a sigmoid activation function at each node, so it's more likely that you get values that are close to 0 or 1.
If you get 0 or 1 from your activations, it means the sigmoid saturated.
You can see how the sigmoid function behaves here.
EDIT: I think I should focus on the saturation.
So suppose you have a 1 at the output layer. This means that your sigmoid function returned 1, which means the value on the input was approaching to 6. If you look at the sigmoid plot, you'll see that when x is close to 6, the output is close to 1 and the derivative of the output would be close to 0 as well. This is a situation when we say the sigmoid "saturated". You do want to avoid situations like that. Hope it's clearer now.
Did you see this question?
Back propagation Error Function
It says you need to work with the derivative of the sigmoid function for the error.
Related
The N2 diagram for my full problem is below.
The N2 diagram for the coupled portion of the problem is below.
I have a DirectSolver handling the coupling between LLTForces and ImplicitLiftingLine, and an LNBGS solver handling the coupling between LiftingLineGroup and TestCL.
The gist for the problem is here: https://gist.github.com/eufren/31c0e569ed703b2aea3e2ef5360610f7
I have implemented guess_nonlinear() on ImplicitLiftingLine, which should use various outputs from LLTGeometry to give a good initial guess for the vortex strengths based on a linearised form of the governing equations.
def guess_nonlinear(self, inputs, outputs, resids):
freestream_unit_vector = inputs['freestream_unit_vector']
freestream_velocity = inputs['freestream_velocity']
n = inputs['normal_vectors']
A = inputs['surface_areas']
l = inputs['bound_vortices']
ic_tot = inputs['influence_coefficients_total']
v_inf = freestream_velocity
v_inf_vec = v_inf*freestream_unit_vector
lin_numerator = np.pi * v_inf * A * np.sum(n * v_inf_vec, axis=1)
lin_denominator = (np.linalg.norm(np.cross(v_inf_vec, l), axis=1) - np.pi * v_inf * A * np.sum(np.sum(n * ic_tot, axis=2), axis=1))
lin_vtx_str = lin_numerator / lin_denominator
outputs['vortex_strengths'] = lin_vtx_str
However, when the problem is run for the first time, any inputs not explicitly set with p.set_val() are all 1s. This causes guess_nonlinear() to give a bad output and so the system fails to converge:
As far as I can tell, the execution order for the LLT group is correct, and the geometry components should be being executed before the implicit component. I'm confused as to why this doesn't seem to actually be happening when the code is run, and instead these inputs are taking their default values.
What do I need to change to get this to work properly? Additionally, I've found difficulty in getting LNBGS to converge (hence adding guess_nonlinear()) during optimisation - only DirectSolver gets all the way through the optimisation without issues, but it's very slow for large numbers of LLT nodes). How can I improve the linear and nonlinear solver selection, and improve the reliability of the iterative solver?
Note: Thanks for providing a testable example. It made figuring out the answer to your question a lot simpler. Your problem was a bit subtle and I would not have been able to give a good answer without runnable code
Your first question: "Why are all the inputs 1"
"Short" Answer
You have put the nonlinear solver to high in the model hierarchy, which then included a key precurser component that computed your input values. By moving the solver down to a lower level of the model, I was able to ensure that the precurser component (LTTGeometry) ran and had valid outputs before you got to the guess_nonlinear of implicit component.
Here is what you had (Notice the implicit solver included LTTGeometry even though the data cycle does not require that component:
I moved both the nonlinear solver and the linear solver down into the LTTCycle group, which then allows the LTTGeometry component to execute before getting to the nonlinear solver and guess_nonlinear step:
My fix is only partially correct, since there is a secondary cycle from the TestCL component that also needs a solver and does not have one. However, that cycle still does not involve the LTTGeometry group. So the fully correct fix is to restructure you model top run geometry first, and then put the LTTCycle and TestCL groups together so you can run a solver over just them. That was a bit more hacking than I wanted to do on your test problem, but you can see the general idea from the adjusted N2 above.
Long Answer
The guess_nonlinear sequence in OpenMDAO does NOT run the compute method of explicit components or of groups. It follows the execution hierarchy, and calls any guess_nonlinear that it finds. So that means that any explicit components you have in your model will NOT get executed, their outputs will not get updated with computed values, and those computed values will not get passed to the inputs of downstream components.
Things get a little tricky when you have deep model hierarchies. The guess_nonlinear method is called as the first step in the nonlinear solver process. If you have a NonLinearRunOnce solver at the top level, it will follow the compute chain down the line calling compute or solve_nonlinear on each child and doing a data transfer after each one. If one of those children happens to be a group with a nonlinear solver, then that solver will call guess_nonlinear on its children (grandchildren of the top group with the NonLinearRunOnce solver) as the first step. So any outputs that were computed by the siblings of this group will be valid, but none of the outputs from the grandchild level will have been computed yet.
You may be wondering why not just have the guess_nonlinear method call the compute for any explicit components? There is a difficult to balance trade off here. If you assume that all explicit components are very cheap to run, then it might make sense to run the compute methods --- or it might not. A lot depends on the cyclic data structure. If some early component in the group needs guesses from the later one, then running its compute isn't going to help you much at all. Perhaps more importantly though, not all explicit components are cheap to run. You might have a very expensive computation, and calling compute as part of the guess process would be way too costly.
The compromise here, if you need some kind of top level guess process, is that you can implement guess_nonlinear at the group level. It's less common to do, but it gives you total control over what happens. You can call whatever you need to call in whatever sequence.
So the absolute key thing to remember is that the only data you have available to you when a guess_nonlinear is called is any data that was computed before your containing solver was executed. That means any thing that was computed before you got to the model scope of the containing solver (not the scope of the component with the guess_method itself).
Your second question: "How can I speed this up when the number of nodes gets large?"
This one not possible to give a generic answer to at all. I noticed that you have already specified sparse partial derivatives. That is a great start, but if its still not fast enough for you then it means you're reaching the limits of what you can do with a DirectSolver. You note that this solver is the only one that gets you through the optimization without issues, which I will take to mean that ScipyKryloventer link description here and PetscKrylov are not converging the linear system well for you --- at least not by themselves. Thats not surprising, as krylov solvers almost always require some kind of preconditioner... and this is why I can't offer a generic answer. Setting up efficient linear solvers for larger-scale compute is a tricky subject. If you look into the literature, you'll find some good suggestions. You can also study open source implementations like VSPAero for some tips.
effectively, you've reached the limit of what simple linear solvers can offer you. From this point forward, OpenMDAO can help a bit by making it easier to implement some preconditioning, but you'll have to suffer the math side yourself.
I'm trying to code a XOR neural network for some weeks, but I always face the same problem. First of all you have to know that I spent hours and hours trying all I found on the net but nothing worked.
After trying to do it using 3Blue1Brown videos on the subject without success, I am now using this http://neuralnetworksanddeeplearning.com/chap2.html. I coded a Matrix library
with all the necessary functions.
My network does have 3 layers with: 2 input neurons, 2 hidden neurons, 1 output neuron.
Moreover I have 2 biases pointing the hidden neurons, and one pointing the output neuron. I use the sigmoid function to have values between 0 and 1, and the quadratic cost function. Everytime I train the network (ie everytime I use backpropagation) I choose a random input with its corresponding output.
The problem is, whatever how many times I train it, the output is never even close to 0 or 1 but always messing around 0.5, and my cost function is stuck around 0.14.
ANY HINT OR HELP IS APPRECIATED -- I really don't get where the problem is, I feel like I've tried everything. PS: Did not show any code here, if needed, don't hesitate to say it.
I've managed to resolve my problem by adding layers in my network. Moreover, when I improved it in order to code an OCR, I added a learning rate to escape from local miminas which were partly the problem everytime my network was getting stuck.
I want to know if there is a way in Tensorflow's seq2seq framework where I can know if a reply to an input can be given with x% of confidence.
An example below:
I have hi as reply to hello. It works fine. I also have bunch of other trained sentences. However, let's say I enter some junk like this - sdjshj sdjk oiqwe qw. Seq2seq still tries to give a response. I understand it designed that way, but I want to know if there is a way which says the framework cannot answer this with confidence. Or no such words were trained.
This would be of great help.
Use logistic function (or sigmoid) on the output logits:
Because logit function is basically the inverse of sigmoid function:
Logit Function:
Sigmoid Function:
You can see that it is similar. In tensorflow. there is the sigmoid function, but I find the program is faster when you just code the sigmoid function:
If you use sigmoid function. You will get a value from 0 to 1 which is the confidence you are looking for. More information can be found here:
https://en.wikipedia.org/wiki/Sigmoid_function
https://en.wikipedia.org/wiki/Logit
I think average perplexityreturned by seq2seq_model.model.stop is the confidence, the smaller, the better. But it could be hard for one to tell a proper threshhold.
I want to implement the following pseudo-code in SIMULINK:
q^0 = q_init, i = 0
IF q^i ! = q_final
q^(i+1) = q^i + alpha * F(q^i)
i = i + 1
ELSE return <q^0,q^1 ... q^i>
The main problem is, that F(q^i) is a function of the current q and is calculated in every iteration. When trying to impement this in SIMULINK, I run into an algebraic loop that cannot be solved.
What is the proper way of solving the problem (in SIMULINK) ?
Thank you !
Miklos
Understand what to partition inside of a Simulink simulation and what should be the output.
You'd usually cache the outputs q^0 to q^i as outputs from the model, rather than trying to get the whole model to output that.
That means the Simulink model would just be dealing with calculating q^(i+1) from q^(i) (which you store with a unit-delay block).
Regarding the if condition: It is best not to run a simulation forever unless you know it will definitely tend towards q_final. If you're not using integers that might get very hard for the == to exactly evaluate. In that case you should use (abs(q^i - final) > threshold) then, and check it will end.
Ending a simulation exactly at that point might be possible with a breakpoint, but a simpler option would to allowing the simulation to run past it and then trim your output (q^0 to q^i, then some more values) in Matlab.
I am currently trying to write a code to solve a non linear system of equations. I am using the functions of the gsl library, more specifically the multiroot_fdf_solver.
My problem is that it currently doesn't want to converge. More specifically, I have the following behavior:
-if my initial conditions are close to the result, the gsl_multiroot_fdf_solver_iterate does not update the parameters at all. I tried to display the results on the different steps, and I have for all the parameters dx = NaN (I think this quite srange), the status of gsl_multiroot_fdf_solver_iterate is "success" and the status of gsl_multiroot_test_residual is "the iteration has not converged yet"
-the parameters are only updated if my initial conditions are really far from the expected result. Obvisously in this case it does not converge to the right values.
I have already checked multiple times the expression of my function and my Jacobian, and they seem good.
I have to precise that my Jacobian (and my system as well) are quite complicated expression with many trigonometric function.
Would you have any idea of what it could be? Is it possible that if the expression of the Jacobian is too complicated, it has troubles to compute it?
Thank you in advance for your answers, I am really stucked at this point.