TF-IDF algorithm in gremlin - graph-databases

Question

I am stuck trying to calculate TF_IDF in my rexster graph database. Here is what I got:
Say I have a graph consisting of a set of vertices representing terms, T, and a set of vertices representing documents, D.
There are edges, E, between terms in T and documents in D. Each edge has a term frequency, tf.
Eg. (pseudocode):
#x, y, and z are arbitrary IDs.
T(x) - E(y) -> D(z)
E(y).tf = 20
T(x).outE()
=> A set of edges.
T(x).outE().inV()
=> A list of Documents, a subset of D
How could I write a germlin script that calculates the TF_IDF when I am trying to do the following?
A: Given one term t, calculate TF_IDF of each Document directly related to t.
B: Given a set of terms Ts, calculate sum of the TF_IDF of each document in Ts.outE().inV() in relation to each applicable term in Ts.
What I have thus far:
#I know this does not work
term = g.v(404)
term.outE().inV().as('docs').path().
groupBy{it.last()}{
it.findAll{it instanceof Edge}.
collect{it.getProperty('frequency')} #I would actually like to use augmented frequency (aka frequency_of_t_in_document / max_frequency_of_any_t_in_document)
}.collect{d,tf-> [d,
tf * ??log(??g.V.has('isDocument') / docs.count() ?? ) ??
]}
#I feel I am close, but I can't quite make this work.

Answer 1

I probably haven't covered the part
B: ...in relation to each applicable term in Ts.
...but the rest should work as expected. I wrote a little helper function that accepts single terms as well as multiple terms:
tfidf = { g, terms, N ->
def closure = {
def paths = it.outE("occursIn").inV().path().toList()
def numPaths = paths.size()
[it.getProperty("term"), paths.collectEntries({
def title = it[2].getProperty("title")
def tf = it[1].getProperty("frequency")
def idf = Math.log10(N / numPaths)
[title, tf * idf]
})]
}
def single = terms instanceof String
def pipe = single ? g.V("term", terms) : g.V().has("term", T.in, terms)
def result = pipe.collect(closure).collectEntries()
single ? result[terms] : result
}
Then I took the Wikipedia example to test it:
g = new TinkerGraph()
g.createKeyIndex("type", Vertex.class)
g.createKeyIndex("term", Vertex.class)
t1 = g.addVertex(["type":"term","term":"this"])
t2 = g.addVertex(["type":"term","term":"is"])
t3 = g.addVertex(["type":"term","term":"a"])
t4 = g.addVertex(["type":"term","term":"sample"])
t5 = g.addVertex(["type":"term","term":"another"])
t6 = g.addVertex(["type":"term","term":"example"])
d1 = g.addVertex(["type":"document","title":"Document 1"])
d2 = g.addVertex(["type":"document","title":"Document 2"])
t1.addEdge("occursIn", d1, ["frequency":1])
t1.addEdge("occursIn", d2, ["frequency":1])
t2.addEdge("occursIn", d1, ["frequency":1])
t2.addEdge("occursIn", d2, ["frequency":1])
t3.addEdge("occursIn", d1, ["frequency":2])
t4.addEdge("occursIn", d1, ["frequency":1])
t5.addEdge("occursIn", d2, ["frequency":2])
t6.addEdge("occursIn", d2, ["frequency":3])
N = g.V("type","document").count()
tfidf(g, "this", N)
tfidf(g, "example", N)
tfidf(g, ["this", "example"], N)
Output:
gremlin> tfidf(g, "this", N)
==>Document 1=0.0
==>Document 2=0.0
gremlin> tfidf(g, "example", N)
==>Document 2=0.9030899869919435
gremlin> tfidf(g, ["this", "example"], N)
==>this={Document 1=0.0, Document 2=0.0}
==>example={Document 2=0.9030899869919435}
I hope this already helps.
Cheers,
Daniel