I've configured a RunnableGraph using GraphDSL.create(). I've also specified a ClosedShape and connected all outlets/inlets. When I try to execute the program I get the following runtime exception:
requirement failed: The inlets [] and outlets [] must correspond to the inlets [filter.in] and outlets [out]
Any idea where I've not correctly connected inlets and outlets?
Here's the graph code:
val g = RunnableGraph.fromGraph(GraphDSL.create() {
implicit builder =>
import GraphDSL.Implicits._
// Source
val A: Outlet[String] = builder.add(Source.fromIterator(() => flightDelayLines)).out
// Flows
val B: FlowShape[String, FlightEvent] = builder.add(csvToFlightEvent)
val C: FlowShape[FlightEvent, DelayRecord] = builder.add(flightEventToDelayRecord)
val D: UniformFanOutShape[DelayRecord, DelayRecord] = builder.add(Broadcast[DelayRecord](2))
val F: FlowShape[DelayRecord, (Int, Int)] = builder.add(countByCarrier)
// Sinks
val E: Inlet[Any] = builder.add(Sink.ignore).in
val G: Inlet[Any] = builder.add(Sink.ignore).in
// Graph
A ~> B ~> flightEventToDelayRecord ~> D ~> E
D ~> F ~> G
ClosedShape
}).run()
I solved my own problem. It was a very simple oversight. Rather than use C that I added to the builder, I was using the flightEventToDelayRecord function in the graph. The solution was to use C in the graph instead.
// Graph
A ~> B ~> C ~> D ~> E
D ~> F ~> G
This made me realize how important it is to break large graphs into small ones. The runtime exception does not pinpoint root cause (e.g, "C is unused"), so it will likely be easier to debug these runtime exceptions if working with smaller graphs. Hope this helps anyone else who becomes stuck.
Related
I want to copy one array to another using functions. But the following code doesnot work. Please correct me and guide me how to do this simplest task. I am a beginner in programming.
#-----------------------------------------
def copy():
a=b
# ========================================
import numpy as np
a = np.ones(5)
b = np.zeros(5)
copy()
print(a)
# ========================================
The above code displays [1 1 1 1 1] instead of [0 0 0 0 0]
I am getting correct answer while doing the same without using functions, as follows :
# ============================
import numpy as np
a = np.ones(5)
b = np.zeros(5)
a=b
print(a)
# ============================
def copy():
a=b
This method is merely declaring a new local variable a and assigning it to b. Now if you had defined a global variable b earlier then a will take the value of that b. Otherwise, it'll throw an error saying b was not defined earlier.
In either case, this method does not touch the global variable a.
If you wanted to modify the global variable a, you need to tell python about it:
def copy():
global a
a = b
But this solution is not very clean (and overly complicated) for something as simple as assignment. I'd suggest sticking to the second version you have in your question.
You are using wrong syntax :-
try this one-
def copy(a, b):
a = b.copy()
return a
a = np.ones(5)
b = np.zeros(5)
print(copy(a, b))
ML function that will accept a boolean function and a list of values and return the last value in the list that meets the given condition or NONE if no values in the list do
my current function looks like this:
fun last func nil = NONE
| last func L =
let val f =
fun getlast(x) = SOME x
| getlast(x::xs) = getlast xs
in List.filter func L
end;
Can anyone help me debug my code and also help me understand local environment in ML?
You're overcomplicating this a bit, and it's unclear what the purposes of f and getlast could be as you're never using them (and the "definition" of f is a syntax error).
If you test your getlast outside of this function (this is usually a good idea) you'll notice that getlast [] is SOME [] and getlast [1,2,3] is SOME [1,2,3]; getlast y is SOME y no matter what y you pass to it.
Also, the result of List.filter func L is an 'a list, not an 'a option, so it's not very useful as the definition of last.
One way of finding such an element in the list xs is using explicit recursion:
If xs is empty, the result is NONE.
If xs is not empty, first see if there is a "last element" in the tail of xs.
If there is, that's the answer.
If there isn't, then
If func holds for the head of xs, that's your answer.
Otherwise, the result is NONE.
Translating this to ML, it might look something like this:
fun last _ [] = NONE
| last f (x::xs) = case last f xs of
NONE => if f x then SOME x else NONE
| result => result
If you want to use List.filter and avoid manual recursion, then note that the last element of a list is the first element of the reverse of that list:
fun last f xs = case List.rev (List.filter f xs) of
[] => NONE
| y::ys => SOME y
Create your own environment for ML or any new project
$ virtualenv -p python3 env (if you have 2.7,3.5)
$ virtualenv env(only you have 3.5+)
$ source env/bin/activate
$ pip install <packagename>
Given:
let weights = [0.5;0.4;0.3]
let X = [[2;3;4];[7;3;2];[5;3;6]]
what I want is: wX = [(0.5)*[2;3;4];(0.4)*[7;3;2];(0.3)*[5;3;6]]
would like to know an elegant way to do this with lists as well as with arrays. Additional optimization information is welcome
You write about a list of lists, but your code shows a list of tuples. Taking the liberty to adjust for that, a solution would be
let weights = [0.5;0.4;0.3]
let X = [[2;3;4];[7;3;2];[5;3;6]]
X
|> List.map2 (fun w x ->
x
|> List.map (fun xi ->
(float xi) * w
)
) weights
Depending on how comfortable you are with the syntax, you may prefer a oneliner like
List.map2 (fun w x -> List.map (float >> (*) w) x) weights X
The same library functions exist for sequences (Seq.map2, Seq.map) and arrays (in the Array module).
This is much more than an answer to the specific question but after a chat in the comments and learning that the question was specifically a part of a neural network in F# I am posting this which covers the question and implements the feedforward part of a neural network. It makes use of MathNet Numerics
This code is an F# translation of part of the Python code from Neural Networks and Deep Learning.
Python
def backprop(self, x, y):
"""Return a tuple ``(nabla_b, nabla_w)`` representing the
gradient for the cost function C_x. ``nabla_b`` and
``nabla_w`` are layer-by-layer lists of numpy arrays, similar
to ``self.biases`` and ``self.weights``."""
nabla_b = [np.zeros(b.shape) for b in self.biases]
nabla_w = [np.zeros(w.shape) for w in self.weights]
# feedforward
activation = x
activations = [x] # list to store all the activations, layer by layer
zs = [] # list to store all the z vectors, layer by layer
for b, w in zip(self.biases, self.weights):
z = np.dot(w, activation)+b
zs.append(z)
activation = sigmoid(z)
activations.append(activation)
F#
module NeuralNetwork1 =
//# Third-party libraries
open MathNet.Numerics.Distributions // Normal.Sample
open MathNet.Numerics.LinearAlgebra // Matrix
type Network(sizes : int array) =
let mutable (_biases : Matrix<double> list) = []
let mutable (_weights : Matrix<double> list) = []
member __.Biases
with get() = _biases
and set value =
_biases <- value
member __.Weights
with get() = _weights
and set value =
_weights <- value
member __.Backprop (x : Matrix<double>) (y : Matrix<double>) =
// Note: There is a separate member for feedforward. This one is only used within Backprop
// Note: In the text layers are numbered from 1 to n with 1 being the input and n being the output
// In the code layers are numbered from 0 to n-1 with 0 being the input and n-1 being the output
// Layers
// 1 2 3 Text
// 0 1 2 Code
// 784 -> 30 -> 10
let feedforward () : (Matrix<double> list * Matrix<double> list) =
let (bw : (Matrix<double> * Matrix<double>) list) = List.zip __.Biases __.Weights
let rec feedfowardInner layer activation zs activations =
match layer with
| x when x < (__.NumLayers - 1) ->
let (bias, weight) = bw.[layer]
let z = weight * activation + bias
let activation = __.Sigmoid z
feedfowardInner (layer + 1) activation (z :: zs) (activation :: activations)
| _ ->
// Normally with recursive functions that build list for returning
// the final list(s) would be reversed before returning.
// However since the returned list will be accessed in reverse order
// for the backpropagation step, we leave them in the reverse order.
(zs, activations)
feedfowardInner 0 x [] [x]
In weight * activation * is an overloaded operator operating on Matrix<double>
Related back to your example data and using MathNet Numerics Arithmetics
let weights = [0.5;0.4;0.3]
let X = [[2;3;4];[7;3;2];[5;3;6]]
first the values for X need to be converted to float
let x1 = [[2.0;3.0;4.0];[7.0;3.0;2.0];[5.0;3;0;6;0]]
Now notice that x1 is a matrix and weights is a vector
so we can just multiply
let wx1 = weights * x1
Since the way I validated the code was a bit more than most I will explain it so that you don't have doubts to its validity.
When working with Neural Networks and in particular mini-batches, the starting numbers for the weights and biases are random and the generation of the mini-batches is also done randomly.
I know the original Python code was valid and I was able to run it successfully and get the same results as indicated in the book, meaning that the initial successes were within a couple of percent of the book and the graphs of the success were the same. I did this for several runs and several configurations of the neural network as discussed in the book. Then I ran the F# code and achieved the same graphs.
I also copied the starting random number sets from the Python code into the F# code so that while the data generated was random, both the Python and F# code used the same starting numbers, of which there are thousands. I then single stepped both the Python and F# code to verify that each individual function was returning a comparable float value, e.g. I put a break point on each line and made sure I checked each one. This actually took a few days because I had to write export and import code and massage the data from Python to F#.
See: How to determine type of nested data structures in Python?
I also tried a variation where I replaced the F# list with Linked list, but found no increase in speed, e.g. LinkedList<Matrix<double>>. Was an interesting exercise.
If I understand correctly,
let wX = weights |> List.map (fun w ->
X |> List.map (fun (a, b, c) ->
w * float a,
w * float b,
w * float c))
This is an alternate way to achieve this using Math.Net: https://numerics.mathdotnet.com/Matrix.html#Arithmetics
I'm working on a project involving reading and writing from an array in Haskell. I want to store the array in an immutable array type (because I want to avoid mutability as much as possible), so I'm working with the freeze and thaw functions for MArrays. I am confused about how I am supposed to use these functions though. This is what I have (that does not compile):
modifyBoard bd = freeze mbd
where
t = view falling st
ps = extractLocs t
mbd = runSTArray $ do
a <- thaw bd
mapM_ (\xy -> writeArray mbd xy (Filled t)) ps
return a
Should I not be using arrays this way or how can I fix this?
It should work with a couple of changes:
1.) Modify your thawed a instead of the unthawed bd.
2.) No no need to re-freeze after runSTArray. The result you return is already frozen.
modifyBoard bd = mbd
where
t = view falling st
ps = extractLocs t
mbd = runSTArray $ do
a <- thaw bd
mapM_ (\xy -> writeArray a xy (Filled t)) ps
return a
jhickner's answer is correct, but it also seems like you've simply reinvented (//) (which under the hood is implemented very much like how you've written it):
modifyBoard bd = bd // map (\xy -> (xy, Filled t)) ps
where t = ...
ps = ...
I have an array like this,
[|{Name = "000016.SZ";
turnover = 3191591006.0;
MV = 34462194.8;};
{Name = "000019.SZ";
turnover = 2316868899.0;
MV = 18438461.48;};
{Name = "000020.SZ";
turnover = 1268882399.0;
MV = 7392964.366;};
.......
|]
How do I sort this array according to "turnover"? Thanks
(does not have much context to explain the code section? how much context should I write)
Assuming that the array is in arr you can just do
arr |> Array.sortBy (fun t -> t.turnover)
I know this has already been answered beautifully; however, I am finding that, like Haskell, F# matches the way I think and thought I'd add this for other novices :)
let rec sortData =
function
| [] -> []
| x :: xs ->
let smaller = List.filter (fun e -> e <= x) >> sortData
let larger = List.filter (fun e -> e > x) >> sortData
smaller xs # [ x ] # larger xs
Note 1: "a >> b" is function composition and means "create a function, f, such that f x = b(a(x))" as in "apply a then apply b" and so on if it continues: a >> b >> c >>...
Note 2: "#" is list concatenation, as in [1..100] = [1..12] # [13..50] # [51..89] # [90..100]. This is more powerful but less efficient than cons, "::", which can only add one element at a time and only to the head of a list, a::[b;c;d] = [a;b;c;d]
Note 3: the List.filter (fun e ->...) expressions produces a "curried function" version holding the provided filtering lambda.
Note 4: I could have made "smaller" and "larger" lists instead of functions (as in "xs |> filter |> sort"). My choice to make them functions was arbitrary.
Note 5: The type signature of the sortData function states that it requires and returns a list whose elements support comparison:
_arg1:'a list -> 'a list when 'a : comparison
Note 6: There is clarity in brevity (despite this particular post :) )
As a testament to the algorithmic clarity of functional languages, the following optimization of the above filter sort is three times faster (as reported by VS Test Explorer). In this case, the list is traversed only once per pivot (the first element) to produce the sub-lists of smaller and larger items. Also, an equivalence list is introduced which collects matching elements away from further comparisons.
let rec sort3 =
function
| [] -> []
| x::xs ->
let accum tot y =
match tot with
| (a,b,c) when y < x -> (y::a,b,c)
| (a,b,c) when y = x -> (a,y::b,c)
| (a,b,c) -> (a,b,y::c)
let (a,b,c) = List.fold accum ([],[x],[]) xs
(sort3 a) # b # (sort3 c)