(Dependantly?) typing containers - generic-programming

Given a type constructor/container F[_] I want to make a combinator which can compose polymorphic types and hlist types into a new container by the following ruleset:
F[HNil] and F[HNil] into a F[HNil]
F[A] and F[HNil] into a F[A :: HNil]
F[HNil] and F[A] into a F[A :: HNil]
F[A] and F[B] into a F[A :: B :: HNil]
F[L <: HList] and F[R <: HList] into a F[L :: R] (hlist concat)
F[A] and F[R <: HList] into a F[A :: R] (hlist prepend)
F[L <: HList] and F[B] into a F[L :: B :: HNil] (hlist append)
Is there a nice way to do this in shapeless? Or is there a easier way to solve this particular problem :) The thing is that I have combinators which yield a F[Int], F[HNil] and F[HNil]. Combining these should not yield a F[Int :: HNil :: HNil] (but F[Int :: HNil]) for example. For this purpose now I've made typeclass:
trait CodecAppender[F, A, B] {
type Out
final type CodecOut = Codec[F, Out]
def apply(a: Codec[F, A], b: Codec[F, B]): CodecOut
}
object CodecAppender {
type Aux[F, A, B, C] = CodecAppender[F, A, B] { type Out = C }
private def make[F, A, B, C](decompose: C => (A, B))(implicit S: Monoid[F], D: DecoderAppender.Aux[F, A, B, C]): Aux[F, A, B, C] = new CodecAppender[F, A, B] {
type Out = C
def apply(a: Codec[F, A], b: Codec[F, B]): CodecOut = new Codec[F, C] {
def encode(in: C) = {
val (head, tail) = decompose(in)
a.encode(head) |+| b.encode(tail)
}
val decoder = D.append(a.decoder, b.decoder)
}
}
implicit def HNilAndHNil[F, O](implicit M: Monoid[F],
D: DecoderAppender.Aux[F, HNil, HNil, HNil]) =
make[F, HNil, HNil, HNil](r => HNil -> HNil)
//other cases omitted, but there are 6 more (which I described above)
}

There is a nice way of doing just that using a series of type classes. You can do something if your F[_] has an instance of an Monad:
trait Conv[A]{
type Out <: HList
def apply(a: A): Out
}
object Conv extends LowPrior{
implicit def hlist[L <: HList] = new Conv[L]{
type Out = L
def apply(hl: L) = hl
}
}
trait LowPrior{
implicit def other[A] = new Conv[A]{
type Out = H :: HNil
def apply(a: A) = a :: HNil
}
}
def combineF[F[_], A, B](fa: F[A], fb: F[B])(implicit ca: Conv[A], cb: Conv[B], m: Monad[F]) = m.flatMap(fa){ a: A =>
m.map(fb){ b: B => ca(a) ::: c(b) }
}
If you want the real return types you could define an Aux type in Conv companion object and then use the Prepend type class of an HList to get the final realized result type. (I'm actually fairly positive you could achieve the same thing with an Applicative or even a Zip type class. Will leave that up to you to discover.)

Scodec is doing something like that :
def prepend[A, L <: HList](a: Codec[A], l: Codec[L]): Codec[A :: L] = new Codec[A :: L] {
override def sizeBound = a.sizeBound + l.sizeBound
override def encode(xs: A :: L) = Codec.encodeBoth(a, l)(xs.head, xs.tail)
override def decode(buffer: BitVector) = Codec.decodeBothCombine(a, l)(buffer) { _ :: _ }
override def toString = s"$a :: $l"
}
https://github.com/scodec/scodec/blob/ac71016dcea61cc6aaabe4f4dff4ab5bf13ac239/shared/src/main/scala/scodec/codecs/HListCodec.scala#L19-L25

Related

Scala: Parsing Array of String to a case class

I have created a case class like this:
def case_class(): Unit = {
case class StockPrice(quarter : Byte,
stock : String,
date : String,
open : Double,
high : Double,
low : Double,
close : Double,
volume : Double,
percent_change_price : Double,
percent_change_volume_over_last_wk : Double,
previous_weeks_volume : Double,
next_weeks_open : Double,
next_weeks_close : Double,
percent_change_next_weeks_price : Double,
days_to_next_dividend : Double,
percent_return_next_dividend : Double
)
And I have thousands of line as Array of String like this:
1,AA,1/7/2011,$15.82,$16.72,$15.78,$16.42,239655616,3.79267,,,$16.71,$15.97,-4.42849,26,0.182704
1,AA,1/14/2011,$16.71,$16.71,$15.64,$15.97,242963398,-4.42849,1.380223028,239655616,$16.19,$15.79,-2.47066,19,0.187852
1,AA,1/21/2011,$16.19,$16.38,$15.60,$15.79,138428495,-2.47066,-43.02495926,242963398,$15.87,$16.13,1.63831,12,0.189994
1,AA,1/28/2011,$15.87,$16.63,$15.82,$16.13,151379173,1.63831,9.355500109,138428495,$16.18,$17.14,5.93325,5,0.185989
How Can I parse data from Array into that case class?
Thank you for your help!
You can proceed as below (I've taken simplified example)
Given your case class and data (lines)
// Your case-class
case class MyCaseClass(
fieldByte: Byte,
fieldString: String,
fieldDouble: Double
)
// input data
val lines: List[String] = List(
"1,AA,$1.1",
"2,BB,$2.2",
"3,CC,$3.3"
)
Note: you can read lines from a text file as
val lines = Source.fromFile("my_file.txt").getLines.toList
You can have some utility methods for mapping (cleaning & parsing)
// remove '$' symbols from string
def removeDollars(line: String): String = line.replaceAll("\\$", "")
// split string into tokens and
// convert into MyCaseClass object
def parseLine(line: String): MyCaseClass = {
val tokens: Seq[String] = line.split(",")
MyCaseClass(
fieldByte = tokens(0).toByte,
fieldString = tokens(1),
fieldDouble = tokens(2).toDouble
)
}
And then use them to convert strings into case-class objects
// conversion
val myCaseClassObjects: Seq[MyCaseClass] = lines.map(removeDollars).map(parseLine)
As a more advanced (and generalized) approach, you can generate the mapping (parsing) function for converting tokens into fields of your case-class using something like reflection, as told here
Here's one way of doing it. I'd recommend splitting everything you do up into lots of small, easy-to-manage functions, otherwise you will get lost trying to figure out where something is going wrong if it all starts throwing exceptions. Data setup:
val array = Array("1,AA,1/7/2011,$15.82,$16.72,$15.78,$16.42,239655616,3.79267,,,$16.71,$15.97,-4.42849,26,0.182704",
"1,AA,1/14/2011,$16.71,$16.71,$15.64,$15.97,242963398,-4.42849,1.380223028,239655616,$16.19,$15.79,-2.47066,19,0.187852",
"1,AA,1/21/2011,$16.19,$16.38,$15.60,$15.79,138428495,-2.47066,-43.02495926,242963398,$15.87,$16.13,1.63831,12,0.189994",
"1,AA,1/28/2011,$15.87,$16.63,$15.82,$16.13,151379173,1.63831,9.355500109,138428495,$16.18,$17.14,5.93325,5,0.185989")
case class StockPrice(quarter: Byte, stock: String, date: String, open: Double,
high: Double, low: Double, close: Double, volume: Double, percent_change_price: Double,
percent_change_volume_over_last_wk: Double, previous_weeks_volume: Double,
next_weeks_open: Double, next_weeks_close: Double, percent_change_next_weeks_price: Double,
days_to_next_dividend: Double, percent_return_next_dividend: Double
)
Function to turn Array[String] into Array[List[String]] and handle any empty fields (I've made an assumption here that you want empty fields to be 0. Change this as necessary):
def splitArray(arr: Array[String]): Array[List[String]] = {
arr.map(
_.replaceAll("\\$", "") // Remove $
.split(",") // Split by ,
.map {
case x if x.isEmpty => "0" // If empty
case y => y // If not empty
}
.toList
)
}
Function to turn a List[String] into a StockPrice. Note that this will fall over if the List isn't exactly 16 items long. I'll leave you to handle any of that. Also, the names are pretty non-descriptive so you can change that too. It will also fall over if your data doesn't map to the relevant .toDouble or toByte or whatever - you can handle this yourself too:
def toStockPrice: List[String] => StockPrice = {
case a :: b :: c :: d :: e :: f :: g :: h :: i :: j :: k :: l :: m :: n :: o :: p :: Nil =>
StockPrice(a.toByte, b, c, d.toDouble, e.toDouble, f.toDouble, g.toDouble, h.toDouble, i.toDouble, j.toDouble,
k.toDouble, l.toDouble, m.toDouble, n.toDouble, o.toDouble, p.toDouble)
}
A nice function to bring this all together:
def makeCaseClass(arr: Array[String]): Seq[StockPrice] = {
val splitArr: Array[List[String]] = splitArray(arr)
splitArr.map(toStockPrice)
}
Output:
println(makeCaseClass(array))
//ArraySeq(
// StockPrice(1,AA,1/7/2011,15.82,16.72,15.78,16.42,2.39655616E8,3.79267,0.0,0.0,16.71,15.97,-4.42849,26.0,0.182704),
// StockPrice(1,AA,1/14/2011,16.71,16.71,15.64,15.97,2.42963398E8,-4.42849,1.380223028,2.39655616E8,16.19,15.79,-2.47066,19.0,0.187852),
// StockPrice(1,AA,1/21/2011,16.19,16.38,15.6,15.79,1.38428495E8,-2.47066,-43.02495926,2.42963398E8,15.87,16.13,1.63831,12.0,0.189994),
// StockPrice(1,AA,1/28/2011,15.87,16.63,15.82,16.13,1.51379173E8,1.63831,9.355500109,1.38428495E8,16.18,17.14,5.93325,5.0,0.185989)
//)
Edit:
To explain the a :: b :: c ..... bit - this is a way of assigning names to items in a List or Seq, given you know the List's size.
val ls = List(1, 2, 3)
val a :: b :: c :: Nil = List(1, 2, 3)
println(a == ls.head) // true
println(b == ls(1)) // true
println(c == ls(2)) // true
Note that the Nil is important because it signifies the last element of the List being Nil. Without it, c would be equal to List(3) as the rest of any List is assigned to the last value in your definition.
You can use this in pattern matching as I have in order to do something with the results:
val ls = List(1, "b", true)
ls match {
case a :: b :: c if c == true => println("this will not be printed")
case a :: b :: c :: Nil if c == true => println(s"this will get printed because c == $c")
} // not exhaustive but you get the point
You can also use it if you know what you want each element in the List to be, like this:
val personCharacteristics = List("James", 26, "blue", 6, 85.4, "brown")
val name :: age :: eyeColour :: otherCharacteristics = personCharacteristics
println(s"Name: $name; Age: $age; Eye colour: $eyeColour")
// Name: James; Age: 26; Eye colour: blue
Obviously these examples are pretty trivial and not exactly what you'd see as a professional Scala developer (at least I don't), but it's a handy thing to be aware of as I do still use this :: syntax at work sometimes.

Haskell fetchName from X11 library does not return the name of the window

Having the following Haskell code:
import Control.Concurrent
import Data.Time.Clock
import Debug.Trace
import Graphics.X11
import Graphics.X11.Xlib.Extras
main :: IO ()
main = do
d <- openDisplay ""
loop d
loop :: Display -> IO ()
loop d = do
time <- getCurrentTime
(w, _) <- getInputFocus d
maybeName <- fetchName d w
windowAttrs <- getWindowAttributes d w
print $ show time ++ " Name: " ++ show maybeName ++ " Width: " ++ show (wa_width windowAttrs)
threadDelay 1000000
loop d
The window title returned by fetchName is always Nothing.
Haskell X11 library is a wrapper around Xlib
Possibly related issues:
The width of the window is either correct or it has the value 1.
XFetchName always returns 0
It looks like fetchName isn't always filled in. Instead you need to use the _NET_WM_NAME property:
import Control.Concurrent
import Data.Time.Clock
import Debug.Trace
import Graphics.X11
import Graphics.X11.Xlib.Extras
main :: IO ()
main = do
d <- openDisplay ""
loop d
loop :: Display -> IO ()
loop d = do
time <- getCurrentTime
(w, _) <- getInputFocus d
a <- internAtom d "_NET_WM_NAME" False
p <- getTextProperty d w a
ps <- wcTextPropertyToTextList d p
windowAttrs <- getWindowAttributes d w
print $ show time ++ " Name: " ++ show ps ++ " Width: " ++ show (wa_width windowAttrs)
threadDelay 1000000
loop d
This is what XMonad does:
https://github.com/xmonad/xmonad/blob/8b055621e92e7ade127043e968f50713c15a00a0/src/XMonad/ManageHook.hs#L71-L80
In the end I had to adapt Brian's answer to the more complete definition from XMonad (because of exceptions being thrown by some windows):
import Control.Exception.Extensible (SomeException (..),
bracket)
import qualified Control.Exception.Extensible as E
import Graphics.X11
import Graphics.X11.Xlib.Extras
getWindowTitle :: Display -> IO String
getWindowTitle d = do
(w, _) <- getInputFocus d
let getProp =
(internAtom d "_NET_WM_NAME" False >>= getTextProperty d w)
`E.catch` \(SomeException _) -> getTextProperty d w wM_NAME
extract prop = do l <- wcTextPropertyToTextList d prop
return $ if null l then "" else head l
bracket getProp (xFree . tp_value) extract `E.catch` \(SomeException _) -> return ""

Q2: AttributeError: 'builtin_function_or_method' object has no attribute 'size'

Could anyone tell me why I'm getting the error type: AttributeError: 'builtin_function_or_method' object has no attribute
'size' in like 57?
for this synthax: out=np.zeros((x.size,y.size))
import numpy as np
import sympy as sp
from numpy import exp,sqrt,pi
from sympy import Integral, log, exp, sqrt, pi
import math
from numpy import array
import matplotlib.pyplot as plt
import scipy.integrate
from scipy.special import erf
from scipy.stats import norm, gaussian_kde
from quantecon import LAE
from sympy.abc import q
#from sympy import symbols
#var('q')
#q= symbols('q')
## == Define parameters == #
mu=80
sigma=20
b=0.2
Q=80
Q1=Q*(1-b)
Q2=Q*(1+b)
d = (sigma*np.sqrt(2*np.pi))
phi = norm()
n = 500
#Phi(z) = 1/2[1 + erf(z/sqrt(2))].
def p_k_positive(x, y):
# x, y = np.array(x, dtype=float), np.array(y, dtype=float)
Positive_RG = norm.pdf(x[:, None] - y[None, :]+Q1, mu, sigma)
print('Positive_R = ', Positive_RG)
return Positive_RG
def p_k_negative(x, y):
# x, y = np.array(x, dtype=float), np.array(y, dtype=float)
Negative_RG = norm.pdf(x[:, None] - y[None, :]+Q2, mu, sigma)
print('Negative_RG = ', Negative_RG)
return Negative_RG
def p_k_zero(x, y):
# x, y = np.array(x, dtype=float), np.array(y, dtype=float)
Zero_RG = (1/(2*math.sqrt(2*math.pi)))*(erf((x[:, None]+Q2-mu)/(sigma*math.sqrt(2)))-erf((x[:, None]+Q1-mu)/(sigma*math.sqrt(2))))
#Zero_RG =norm.pdf
print('Zero_RG',Zero_RG)
return Zero_RG
def myFilter(x,y):
x, y = x.squeeze, y.squeeze
out=np.zeros((x.size,y.size))
xyDiff = x[:, None] - y[None, :]
out=np.where(np.bitwise_and(y[None, :] > 0.0, xyDiff >= -Q1), p_k_positive(x, y), out) # unless the sum functions are different
out=np.where(np.bitwise_and(y[None, :] < 0.0, x[:, None] >= -Q1), p_k_negative(x, y), out)
out=np.where(np.bitwise_and(y[None, :] ==0.0, xyDiff >= -Q1), p_k_zero(x, y), out)
return out
Z = phi.rvs(n)
X = np.empty(n)
for t in range(n-1):
X[t+1] = X[t] + Z[t]
#X[t+1] = np.abs(X[t]) + Z[t]
psi_est = LAE(myFilter, X)
k_est = gaussian_kde(X)
fig, ax = plt.subplots(figsize=(10,7))
ys = np.linspace(-200.0, 200.0, 200)
ax.plot(ys, psi_est(ys), 'g-', lw=2, alpha=0.6, label='look ahead estimate')
ax.plot(ys, k_est(ys), 'k-', lw=2, alpha=0.6, label='kernel based estimate')
ax.legend(loc='upper left')
plt.show()
x, y = x.squeeze, y.squeeze
Should be
x, y = x.squeeze(), y.squeeze()
or you're trying to take the size of a function.

Node.js effects in Thermite for Electron

I am trying to expand https://github.com/kRITZCREEK/electron-apps-with-purescript to add an input for the directory whose filenames I would display.
However, I cannot get the following function to typecheck (the last two lines are problematic, the last is line 53):
performAction :: T.PerformAction _ State _ Action
performAction (SetEditText s) _ _ = void do
T.cotransform $ _ { dir = s }
performAction (UpdateFiles s) _ _ = void do
filenames <- either (const []) id <$> try (readdir s)
T.cotransform $ _ { names = filenames }
Here readdir is from Node.FS.Sync and has the signature
forall eff. String -> Eff (fs :: FS, err :: EXCEPTION | eff) (Array String)
and performAction is of type
forall t39 t40. Action -> t40 -> {dir :: String, names :: Array String} -> FreeT (CoTransform (Maybe {dir :: String, names :: Array String}) ({dir :: String, names :: Array String} -> {dir :: String, names :: Array String}))
The actual error is
Could not match type at line 53 col 4
FreeT
(CoTransform t2
({ files :: t3
| t4
}
-> { files :: Array String
| t4
}
)
)
with type
Eff
while trying to match type FreeT
(CoTransform t2
({ files :: t3
| t4
}
-> { files :: Array String
| t4
}
)
)
t5
with type Eff
( fs :: FS
| t0
)
while checking that expression (apply cotransform) (\$1 ->
$1 { files = f
}
)
has type Eff
( fs :: FS
| t0
)
t1
in value declaration performAction
where t1 is an unknown type
t0 is an unknown type
t2 is an unknown type
t5 is an unknown type
t4 is an unknown type
t3 is an unknown type
(The whole project can be found under https://github.com/MarkusBarthlen/electron-apps-with-purescript/blob/master/src/Main.purs)
I suspect I have to use any of lift/liftEff/liftEff'/liftAff. However,
performAction :: T.PerformAction _ State _ Action
performAction (UpdateFiles s) _ _ = void do
filenames <- ( lift (either (const []) id <$> try (readdir s)))
T.cotransform $ _ { names = filenames }
results in
Could not match type at line 55 col 47
Eff
with type
Aff
while trying to match type Eff
( fs :: FS
| t1
)
with type Aff t0
You can make it work with:
performAction :: forall e. T.PerformAction (fs :: FS | e) State _ Action
performAction (SetEditText s) _ _ = void do
T.cotransform $ _ { dir = s }
performAction (UpdateFiles s) _ _ = void do
filenames <- lift (liftEff (either (const []) id <$> try (readdir s)))
T.cotransform $ _ { names = filenames }
-- T.cotransform $ _ { dir = ""}
liftEff takes your Eff to an Aff and lift then raises that into the FreeT ... that Thermite uses. The extra lift shouldn't be necessary, but I think the problem is type inference around rows and typeclasses here and the situation should get better with the next release where we'll most likely have functional dependencies.

Auto-generated static field in F# class/module prevents creation of SQL CLR function

I'm trying to write a CLR user-defined function in F#, but CREATE ASSEMBLY gives the error:
CREATE ASSEMBLY failed because type 'StringMetrics' in safe assembly 'MyNamespace.SqlServer.Text' has a static field 'field1776#'. Attributes of static fields in safe assemblies must be marked readonly in Visual C#, ReadOnly in Visual Basic, or initonly in Visual C++ and intermediate language.
Here's how it looks in Reflector. This is not a field I've explicitly created.
[DebuggerBrowsable(DebuggerBrowsableState.Never)]
internal static <PrivateImplementationDetails$MyNamespace-SqlServer-Text>.T1775_18Bytes# field1776#; // data size: 18 bytes
I've tried using a module and a class. Both generate the field, just in different places. What is this field for? Is there a way to avoid its creation? Is there another approach I should be using to create a CLR function in F#? Is it even possible?
Complete Code
namespace MyNamespace.SqlServer.Text
module StringMetrics =
open System
open System.Collections.Generic
open System.Data
open System.Data.SqlTypes
[<CompiledName("FuzzyMatch")>]
let fuzzyMatch (strA:SqlString) (strB:SqlString) =
if strA.IsNull || strB.IsNull then SqlInt32.Zero
else
let comparer = StringComparer.OrdinalIgnoreCase
let wordBoundaries = [|' '; '\t'; '\n'; '\r'; ','; ':'; ';'; '('; ')'|]
let stringEquals a b = comparer.Equals(a, b)
let isSubstring (search:string) (find:string) = find.Length >= search.Length / 2 && search.IndexOf(find, StringComparison.OrdinalIgnoreCase) >= 0
let split (str:string) = str.Split(wordBoundaries)
let score (strA:string) (strB:string) =
if stringEquals strA strB then strA.Length * 3
else
let lenA, lenB = strA.Length, strB.Length
if strA |> isSubstring strB then lenA * 2
elif strB |> isSubstring strA then lenB * 2
else 0
let arrA, arrB = split strA.Value, split strB.Value
let dictA, dictB = Dictionary(), Dictionary()
arrA |> Seq.iteri (fun i a ->
arrB |> Seq.iteri (fun j b ->
match score a b with
| 0 -> ()
| s ->
match dictB.TryGetValue(j) with
| true, (s', i') -> //'
if s > s' then //'
dictA.Add(i, j)
dictB.[j] <- (s, i)
| _ ->
dictA.Add(i, j)
dictB.Add(j, (s, i))))
let matchScore = dictB |> Seq.sumBy (function (KeyValue(_, (s, _))) -> s)
let nonMatchA =
arrA
|> Seq.mapi (fun i a -> i, a)
|> Seq.fold (fun s (i, a) ->
if dictA.ContainsKey(i) then s
else s + a.Length) 0
let wordsB = HashSet(seq { for (KeyValue(i, _)) in dictB -> arrB.[i] }, comparer)
let nonMatchB =
arrB |> Seq.fold (fun s b ->
if wordsB.Add(b) then s + b.Length
else s) 0
SqlInt32(matchScore - nonMatchA - nonMatchB)
It seems it's generated by the wordBoundaries array. If you express it as a list instead, and convert it to an array at runtime, this internal static field is not generated:
let wordBoundaries = Array.ofList [' '; '\t'; '\n'; '\r'; ','; ':'; ';'; '('; ')']
However, it seems that also the function itself is represented as a static, non-readonly
field:
public static SqlInt32 fuzzyMatch(SqlString strA, SqlString strB);
Maybe using a class instead of a module cures this.

Resources