Neo4j Create new label based on other labels with certain properties - database

Question

I need to create Trajectories based on Points.
A Trajectory can contain any number of Points that meet certain criteria.
Criteria are: cameraSid, trajectoryId, classType and classQual should be equal.
The difference in time (at) of each point must be less or equal to 1 hour.
In order to create a Trajectory we need at least one point.
In order to associate a new point to an existing Trajectory, the latest associated Point of the trajectory must be no older and 1 hour compared to the new point.
If the new Point has the exact same properties but is older than 1 hour, then a new Trajectory need to be created.
I've been reading a lot but, I cannot make this work as it should.
This is what I have tried so far:
MATCH (p:Point)
WHERE NOT (:Trajectory)-[:CONTAINS]->(p)
WITH p.cameraSid AS cameraSid, p.trajectoryId AS trajectoryId, p.classType AS classType, p.classQual AS classQual, COLLECT(p) AS points
UNWIND points AS point
MERGE (trajectory:Trajectory{trajectoryId:point.trajectoryId, cameraSid: point.cameraSid, classType: point.classType, classQual: point.classQual, date: date(datetime(point.at))})
MERGE (trajectory)-[:CONTAINS{at:point.at}]->(point)
I have no idea how to create this sort of condition (1hr or less) in the MERGE clause.
Here are the neo4j queries to create some data
// Create points
LOAD CSV FROM 'https://uca54485eb4c5d2a6869053af475.dl.dropboxusercontent.com/cd/0/get/AmR2pn0hC0c-CQW_mSS-TDqHQyi7MNVjPvqffQHhSIyMP37D7UMtfODdHDkNWi6-HqzQdp4ob2Q3326g6imEd26F3sdNJyJuAeNa8wJA2o_E6A/file?dl=1#' AS line
CREATE (:Point{trajectoryId: line[0],at: line[1],cameraSid: line[2],activity: line[3],x: line[4],atEpochMilli: line[5],y: line[6],control: line[7],classQual: line[8],classType: line[9],uniqueIdentifier: line[10]})
// Create Trajectory based on Points
MATCH (p:Point)
WHERE NOT (:Trajectory)-[:CONTAINS]->(p)
WITH p.cameraSid AS cameraSid, p.trajectoryId AS trajectoryId, p.classType AS classType, p.classQual AS classQual, COLLECT(p) AS points
UNWIND points AS point
MERGE (trajectory:Trajectory{trajectoryId:point.trajectoryId, cameraSid: point.cameraSid, classType: point.classType, classQual: point.classQual, date: date(datetime(point.at))})
MERGE (trajectory)-[:CONTAINS{at:point.at}]->(point)
If the link to the CSV file does not work, here is an alternative, in this case, you will have to download the file and then import it locally from your neo4j instance.

Answer 1

I think this is one of those just because you can do it in one Cypher statement doesn't mean you should situations, and you will almost certainly find this easier to do in application code.
Regardless, it can be done using APOC and by introducing an instanceId unique property on your Trajectory nodes.
Possible solution
This almost certainly won't scale, and you'll want indexes (discussed later based on educated guesswork).
First we need to change your import script to make sure that the at property is a datetime and not just a string (otherwise we end up peppering the queries with datetime() calls:
LOAD CSV FROM 'file:///export.csv' AS line
CREATE (:Point{trajectoryId: line[0], at: datetime(line[1]), cameraSid: line[2], activity: line[3],x: line[4], atEpochMilli: line[5], y: line[6], control: line[7], classQual: line[8], classType: line[9], uniqueIdentifier: line[10]})
The following then appears to take your sample data set and add Trajectories per your requirements (and can be run whenever new Points are added).
CALL apoc.periodic.iterate(
'
MATCH (p: Point)
WHERE NOT (:Trajectory)-[:CONTAINS]->(p)
RETURN p
ORDER BY p.at
',
'
OPTIONAL MATCH (t: Trajectory { trajectoryId: p.trajectoryId, cameraSid: p.cameraSid, classQual: p.classQual, classType: p.classType })-[:CONTAINS]-(trajPoint:Point)
WITH p, t, max(trajPoint.at) as maxAt, min(trajPoint.at) as minAt
WITH p, max(case when t is not null AND (
(p.at <= datetime(maxAt) + duration({ hours: 1 }))
AND
(p.at >= datetime(minAt) - duration({ hours: 1 }))
)
THEN t.instanceId ELSE NULL END) as instanceId
MERGE (tActual: Trajectory { trajectoryId: p.trajectoryId, cameraSid: p.cameraSid, classQual: p.classQual, classType: p.classType, instanceId: COALESCE(instanceId, randomUUID()) })
ON CREATE SET tActual.date = date(datetime(p.at))
MERGE (tActual)-[:CONTAINS]->(p)
RETURN instanceId
',
{ parallel: false, batchSize: 1 })
Explanation
The problem as posed is tricky because the decision on whether or not to create a new Trajectory or add the point to an existing one depends entirely on how we handled all prior Points. That means two things:
We need to process the Points in order to make sure we create reliable Trajectories - we start with the earliest, and work up
We need each creation or amend of a Trajectory to be immediately visible for the processing of the next Point - that is to say that we need to handle each Point in isolation, as though it were a mini-transaction
We'll use apoc.periodic.iterate with a batchSize of 1 to give us the behaviour we need.
The first parameter builds the set of nodes to be processed - all those Points which aren't currently part of a Trajectory, sorted by their timestamp.
The second parameter to apoc.periodic.iterate is where the magic's happening so let's break that down - given a point p that isn't part of a Trajectory so far:
OPTIONAL MATCH (t: Trajectory { trajectoryId: p.trajectoryId, cameraSid: p.cameraSid, classQual: p.classQual, classType: p.classType })-[:CONTAINS]-(trajPoint:Point)
WITH p, t, max(trajPoint.at) as maxAt, min(trajPoint.at) as minAt
WITH p, max(case when t is not null AND (
(p.at <= datetime(maxAt) + duration({ hours: 1 }))
AND
(p.at >= datetime(minAt) - duration({ hours: 1 }))
)
THEN t.instanceId ELSE NULL END) as instanceId
Finds any Trajectory that matches the key fields and that contain a Point that's within an hour of the incoming point p and pick out its instanceId property if we find a suitable match (or the biggest one we found if there are multiple matches - we just want to ensure there's zero or one rows by this point)
We'll see what instanceId is all about in a minute, but consider it a unique identifier for a given Trajectory
MERGE (tActual: Trajectory { trajectoryId: p.trajectoryId, cameraSid: p.cameraSid, classQual: p.classQual, classType: p.classType, instanceId: COALESCE(instanceId, randomUUID()) })
ON CREATE SET tActual.date = date(datetime(p.at))
Ensure that there's a Trajectory that matches the incoming point's key fields - if the previous code found a matching Trajectory then the MERGE has no work to do. Otherwise create the new Trajectory - we'll add a property instanceId to a new random UUID if we didn't match a Trajectory earlier to compel the MERGE to create a new node (since no other will exist with that UUID, even if one matches all the other key fields)
MERGE (tActual)-[:CONTAINS]->(p)
RETURN instanceId
tActual is now the Trajectory that the incoming point p should belong to - create the :CONTAINS relationship
The third parameter is vital:
{ parallel: false, batchSize: 1 })
Important: For this to work, each iteration of the 'inner' Cypher statement has to happen exactly in order, so we force a batchSize of 1 and disable parallelism to prevent APOC from scheduling the batches in any way other than one-at-a-time.
Indexing
I think the performance of the above is going to degrade quickly as the size of the import grows and the number of Trajectories increases. At a minimum I think you'll want a composite index on
:Trajectory(trajectoryId, cameraSid, classQual, classType) - so that the initial match to find candidate trajectories for a given Point is quick
:Trajectory(trajectoryId, cameraSid, classQual, classType, instanceId) - so that the MERGE at the end finds the existing Trajectory to add to quickly, if one exists
However - that's guesswork from eyeballing the query, and unfortunately you can't see into the query properly to tell what the execution plan is because we're using apoc.periodic.iterate - an EXPLAIN or PROFILE will just tell you that there's one procedure call costing 1 db hit, which is true but not helpful.