Why is this SQLite query slow and getting slower? - database

Question

I have a long term data logging service which produces files containing one day worth of data each. I'm loading the files into an SQLite DB in a Windows forms application. The procedure to insert the data from the file into the DB includes two queries, the result of which is used in the subsequent insert.
Using SQLconnect As New SQLite.SQLiteConnection("Data Source=" & fn & ";")
SQLconnect.Open()
Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand
Dim SqlTrans As System.Data.SQLite.SQLiteTransaction = SQLconnect.BeginTransaction
For Each Path As String In paths
fs = System.IO.File.Open(Path, IO.FileMode.Open, IO.FileAccess.Read, IO.FileShare.Read) 'Open file
Do While ReadFromStoreFile(fs, dt, Sent) = True 'Read a Timestamp/sentence pair
'Create a positions table for this MMSI if one doesn't already exist
SQLcommand.CommandText = "CREATE TABLE IF NOT EXISTS MMSI" & msg.MMSI & " (PosID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Lat REAL, Lon REAL, Status INTEGER, SOG REAL, COG INTEGER, HDG INTEGER, VoyageID INTEGER);"
SQLcommand.ExecuteNonQuery()
Select Case msg.Type 'Dynamic position report
Case AIS.MsgType.PosRptClsA
'###THIS QUERY TAKES 20 secs per file (day) and increases 3 seconds per day!
SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1" 'still the same
SQLreader = SQLcommand.ExecuteReader()
SQLreader.Read()
VID = SQLreader.Item(0)
SQLreader.Close()
SQLcommand.CommandText = "INSERT INTO MMSI" & msg.MMSI & " (Date, Lat, Lon, Status, SOG, COG, HDG, VoyageID) VALUES (" & ts & ", " & msg.Latitude & ", " & msg.Longitude & ", " & msg.NavStatus & ", " & SOG & ", " & COG & ", " & HDG & ", " & VID & ")"
SQLcommand.ExecuteNonQuery()
SQLreader.Close()
Case AIS.MsgType.StatAndVge
'Find the latest entry for this MMSI in the Voyages table
'###THIS QUERY takes 3 secs for same number of queries and does NOT increase
SQLcommand.CommandText = "SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1"
SQLreader = SQLcommand.ExecuteReader()
SQLreader.Read()
Dim NoVoyage As Boolean = Not (SQLreader.HasRows)
If Not NoVoyage Then
'If the data has changed, add a new entry
If Not (SQLreader.Item(2) = msg.Length) Then Changed = True
If Not (SQLreader.Item(3) = msg.Breadth) Then Changed = True
If Not (SQLreader.Item(4) = msg.Draught) Then Changed = True
If Not (SQLreader.Item(5) = msg.Destination) Then Changed = True
If Not (SQLreader.Item(6) = msg.ETA.Ticks) Then Changed = True
VoyageID = SQLreader.Item(0)
End If
SQLreader.Close()
If Changed Or NoVoyage Then
Changed = False 'reset flag
SQLcommand.CommandText = "INSERT INTO Voyages (Date, Length, Breadth, Draught, Destination, ETA, MMSI) VALUES (" & ts & ", " & msg.Length & ", " & msg.Breadth & ", " & msg.Draught & ", '" & msg.Destination.Replace("'", "''") & "', " & msg.ETA.Ticks & ", " & msg.MMSI_int & ")"
SQLcommand.ExecuteNonQuery()
SQLcommand.CommandText = "SELECT last_insert_rowid() FROM Voyages"
SQLreader = SQLcommand.ExecuteReader()
SQLreader.Read()
VoyageID = SQLreader.Item(0)
SQLreader.Close()
End If
End Select 'message type
Loop 'Read next entry from file
fs.Close() 'Close the file
'Write this file into the files table, so we know it has been written to the DB
fileinf = New System.IO.FileInfo(Path)
SQLcommand.CommandText = "INSERT OR REPLACE INTO Files (Name, Size, Date) VALUES ('" & fileinf.Name & "', '" & fileinf.Length & "', '" & fileinf.LastWriteTimeUtc.Ticks & "')"
SQLcommand.ExecuteNonQuery()
Next 'The next path in the list of paths to decode
SqlTrans.Commit() 'End of all files reached, commit all the changes to the DB
End Using 'SQLcommand
End Using 'SQLconnect
As indicated in the code, the first query is taking a very long time and (more importantly) is increasing in duration as data is loaded into the DB. When adding to 21 days of data in the DB, this query alone takes a cumulative time of around 20 secs per day and increases by about 3 secs per day for each day added. The really strange thing is that the second query (which seems identical to me) is fast (around 3 secs for the same number of queries) and is NOT increasing as more data is added.
Here is the function that creates the empty database:
Public Function CreateDB(fn As String, Force As Boolean) As Boolean
If System.IO.File.Exists(fn) Then
If Force Then
System.IO.File.Delete(fn) 'Delete the old DB and create a new one
Else
Return True 'DB alrewady exists so just return true
End If
End If
Using SQLconnect As New SQLite.SQLiteConnection
SQLconnect.ConnectionString = "Data Source=" & fn & ";"
SQLconnect.Open()
'Create Tables
Using SQLcommand As SQLite.SQLiteCommand = SQLconnect.CreateCommand
'Set page size
SQLcommand.CommandText = "PRAGMA Page_size = 4096;"
SQLcommand.ExecuteNonQuery()
'Set journalling mode to off
SQLcommand.CommandText = "PRAGMA journal_mode = OFF;"
SQLcommand.ExecuteNonQuery()
'Set auto indexing off
SQLcommand.CommandText = "PRAGMA automatic_index = false;"
SQLcommand.ExecuteNonQuery()
'Create Vessels Table
SQLcommand.CommandText = "CREATE TABLE Vessels(MMSI TEXT PRIMARY KEY, Name TEXT, Type INTEGER, IMO TEXT, CallSign TEXT, MothershipMMSI INTEGER, LastVoyageID INTEGER);"
SQLcommand.ExecuteNonQuery()
'Create Voyages Table
SQLcommand.CommandText = "CREATE TABLE Voyages(VoyageID INTEGER PRIMARY KEY AUTOINCREMENT, Date NUMERIC, Length INTEGER, Breadth INTEGER, Draught INTEGER, Destination TEXT, ETA NUMERIC, MMSI INTEGER);"
SQLcommand.ExecuteNonQuery()
'Create Meta Table
SQLcommand.CommandText = "CREATE TABLE Files(Name TEXT PRIMARY KEY, Size NUMERIC, Date NUMERIC);"
SQLcommand.ExecuteNonQuery()
End Using 'SQLcommand
End Using ' SQLconnect
Return True
End Function
What could be causing the first query to be so slow, compared to the second query, and take longer as more data is added to the DB?
SQlite and System.Data.Sqlite are the latest versions.

Answer 1

Assuming msg is changed by ReadFromStoreFile, then the query
"SELECT * FROM Voyages WHERE MMSI = " & msg.MMSI & " ORDER BY VoyageID DESC LIMIT 1"
will be slower if there are more voyages for the given MMSI. So I assume the MMSI values that have AIS.MsgType.PosRptClsA are inserted more frequently than the other MMSIs.
It appears the query is getting the maximum voyage id for the MMSI. You could do this more directly with
"SELECT max(VoyageID) FROM Voyages WHERE MMSI = " & msg.MMSI
I don't know if this will run faster.
Alternately, you could keep a dictionary of MMSI and max voyage id and update it when you do an insert into voyage to eliminate the query.