How to reconfigure path to the data files in Clarion 5 IDE? - database

There is a problem, the system is written in Clarion 5 came from the past and now it needs to be rewrite in Java.
To do this I need to deal with its current state and how it works.
I'm generate the executable file via Application Generator (\*.APP-> \*.CLW -> \*.EXE, \*.DLL).
But when I run it I get the message:
File(\...\...\DAT.TPS) could not be opened. Error: Path Not Found(3). Press OK to end this application
And then - halt, File Access Error
In what may be the problem? Is it possible in the Clarion 5 IDE to reconfigure the path to the data files?

Generally, Clarion uses a data dictionary (DCT) as the center of persistent data (files) that will be used by the program. There are other ways you can define a table, but since you mentioned you compile from the APP, I'm concluding that your APP is linked to a DCT.
In the DCT you have the declarations for every file your application will use. In the file declaration you can inform both logic and disk file name. The error message says you have a problem in the definition of the disk file name.
The Clarion language separates the logic data structure definition from its disk file. A "file" for a Clarion programm, is a complex data structure, which conforms to the following:
structName FILE, DRIVER( 'driverType' ), NAME( 'diskFileName' )
key KEY( keyName )
index INDEX( indexName )
recordName RECORD
field DATATYPE
.
.
END
END
The above is the basic declaration syntax, and a real example would be like:
orders FILE, DRIVER( 'TopSpeed' ), NAME( 'sales.dat\orders' )
ordersPK KEY( id ), PRIMARY
customerK INDEX( customerID )
notes MEMO( 4096 )
RECORD RECORD
id LONG
customerID LONG
datePlaced DATE
status STRING( 1 )
END
END
orderItems FILE, DRIVER( 'TopSpeed' ), NAME( 'sales.dat\items' )
itemsPK KEY( orderID, id ), PRIMARY
RECORD RECORD
orderID LONG
id LONG
productID LONG
quantityOrdered DECIMAL( 10, 2 )
unitPrice DECIMAL( 10, 2 )
END
END
Now, with the above two declarations, I have two logic files that reside in the same disk file. This is a capability offered for some file drivers, like the TopSpeed file driver. It is up to the system designer to decide if, and which files will reside in the same disk file, and I can talk about that on another post, if it is the case.
For now, the problem may be arising from the fact that you probably didn't change the NAME property of the file declaration, and the driver you're using doesn't support multi-file storage.
Here's a revised file definition for the same case above, but targeting a SQL database.
szDBConn CSTRING( 1024 ) ! //Connection string to DB server
orders FILE, DRIVER( 'ODBC' ), NAME( 'orders' ), OWNER( szDBconn )
ordersPK KEY( id ), PRIMARY
customerK INDEX( customerID )
notes MEMO( 4096 ), NAME( 'notes' )
RECORD RECORD
id LONG, NAME( 'id | READONLY' )
customerID LONG
datePlaced DATE
status STRING( 1 )
END
END
orderItems FILE, DRIVER( 'ODBC' ), NAME( 'order_items' ), OWNER( szDBconn )
itemsPK KEY( orderID, id ), PRIMARY
RECORD RECORD
orderID LONG
id LONG
productID LONG
quantityOrdered DECIMAL( 10, 2 )
unitPrice DECIMAL( 10, 2 )
END
END
Now, if you pay attention, you'll notice the presence of a szDBconn variable declaration, which is referenced at the file declarations. This is necessary to inform the Clarion file driver system what to pass the ODBC manager in order to connect to the dabase. Check Connection Strings for plenty of connection strings examples.
Check the DCT definitions of your files to see if they reflect what the driver expects.
Also, be aware that Clarion does allow mixing different file drivers to be used by the same program. Thus, you can adapt an existing program to use an external data source if needed.
Here is a complete Clarion program to transfer information from an ISAM file to a DBMS.
PROGRAM
MAP
END
INCLUDE( 'equates.clw' ) ! //Include common definitions
szDBconn CSTRING( 1024 )
inputFile FILE, DRIVER( 'dBase3' )
RECORD RECORD
id LONG
name STRING( 50 )
END
END
outuputFile FILE, DRIVER( 'ODBC' ), NAME( 'import.newcustomers' ), |
OWNER( szDBconn )
RECORD RECORD
id LONG
name STRING( 50 )
backendImportedColumn STRING( 8 )
imported GROUP, OVER( backendImportedColumn )
date DATE
time TIME
END
processed CHAR( 1 )
END
END
CODE
IF NOT EXISTS( COMMAND( 1 ) )
MESSAGE( 'File ' & COMMAND( 1 ) & ' doesn''t exist' )
RETURN
END
imputFile{ PROP:Name } = COMMAND( 1 )
OPEN( inputFile, 42h )
IF ERRORCODE()
MESSAGE( 'Error openning file ' & inputFile{ PROP:Name } )
RETURN
END
szDBconn = 'Driver={{PostgreSQL ANSI};Server=192.168.0.1;Database=test;' & |
'Uid=me;Pwd=plaintextpassword'
OPEN( outputFile, 42h )
IF ERRORCODE()
MESSAGE( 'Error openning import table: ' & FILEERROR() )
RETURN
END
! // Lets stuff the information thatll be used for every record
outputFile.imported.date = TODAY()
outputFile.imported.time = CLOCK()
outputFile.processed = 'N'
! //arm sequential ISAM file scan
SET( inputFile, 1 )
LOOP UNTIL EOF( inputFile )
NEXT( inputFile )
outputFile.id = inputFile.id
outputFile.name = input.name
ADD( outputFile )
END
BEEP( BEEP:SystemExclamation )
MESSAGE( 'File importing completed' )
Well, this example program serves only the purpose of showing how the different elements of the program should be used. I didn't use a window to let the user track the progress, and used Clarion's primitives, like ADD(), which work for sure, but inside a loop can represent a drag in performance.
Much better would be to encapsulate the entire reading in a transaction opened with outputFile{ PROP:SQ } = 'BEGIN TRANSACTION' and at the end, issue a outputFile{ PROP:SQL } = 'COMMIT'.
Yes, trhough the PROP:SQL one can issue ANY command accepted by the server, including a DROP DATABASE, so it is very powerful. Use with care.
Gustavo

Related

The merge into command on Snowflake loads only null values when loading data from S3

I'm having issues when running the command MERGE INTO on Snowflake. The data is located in a bucket on S3. The files format are .snappy.parquet.
The command runs well, it identifies the files in S3, but it loads only NULL values to the table. The total row numbers are also good.
I confirmed that #myExternalStageToS3 points to the right location by running a query which returned the expected values:
SELECT
$1:DAY,
$1:CHANNEL_CATEGORY,
$1:SOURCE,
$1:PLATFORM,
$1:LOB
#myExternalStageToS3
(FILE_FORMAT => 'sf_parquet_format')
As it is a new table with no records, the condition uses INSERT.
MERGE INTO myTable as target USING
(
SELECT
$1:DAY,
$1:CHANNEL_CATEGORY,
$1:SOURCE,
$1:PLATFORM,
$1:LOB
FROM #myExternalStageToS3
(FILE_FORMAT => 'sf_parquet_format')
) as src
ON target.CHANNEL_CATEGORY = src.$1:CHANNEL_CATEGORY
AND target.SOURCE = src.$1:SOURCE
WHEN MATCHED THEN
UPDATE SET
DAY= src.$1:DAY
,CHANNEL_CATEGORY= src.$1:CHANNEL_CATEGORY
,SOURCE= src.$1:SOURCE
,PLATFORM= src.$1:PLATFORM
,LOB= src.$1:LOB
WHEN NOT MATCHED THEN
INSERT
(
DAY,
CHANNEL_CATEGORY,
SOURCE,
PLATFORM,
LOB
) VALUES
(
src.$1:DAY,
src.$1:CHANNEL_CATEGORY,
src.$1:SOURCE,
src.$1:PLATFORM,
src.$1:LOB
);
The sf_parque_format was created with these details:
create or replace file format sf_parquet_format
type = 'parquet'
compression = auto;
Do you have any idea what am I missing?
The query inside USING part was altered(data type casts and aliases):
MERGE INTO myTable as target USING (
SELECT
$1:DAY::TEXT AS DAY,
$1:CHANNEL_CATEGORY::TEXT AS CHANNEL_CATEGORY,
$1:SOURCE::TEXT AS SOURCE,
$1:PLATFORM::TEXT AS PLATFROM,
$1:LOB::TEXT AS LOB
FROM #myExternalStageToS3
(FILE_FORMAT => 'sf_parquet_format')
) as src
ON target.CHANNEL_CATEGORY = src.CHANNEL_CATEGORY
AND target.SOURCE = src.SOURCE
WHEN MATCHED THEN
UPDATE SET
DAY= src.DAY
,PLATFORM= src.PLATFORM
,LOB= src.LOB
WHEN NOT MATCHED THEN
INSERT (
DAY,
CHANNEL_CATEGORY,
SOURCE,
PLATFORM,
LOB
) VALUES (
src.DAY,
src.CHANNEL_CATEGORY,
src.SOURCE,
src.PLATFORM,
src.LOB
);
The UPDATE part does not require ,CHANNEL_CATEGORY= src.CHANNEL_CATEGORY ,SOURCE= src.SOURCE as condition is already met by ON clasue.

String delimiter present in string not permitted in Polybase?

I'm creating an external table using a CSV stored in an Azure Data Lake Storage and populating the table using Polybase in SQL Server.
However, I ran into this problem and figured it may be due to the fact that in one particular column there are double quotes present within the string, and the string delimiter has been specified as " in Polybase (STRING_DELIMITER = '"').
HdfsBridge::recordReaderFillBuffer - Unexpected error encountered filling record reader buffer: HadoopExecutionException: Could not find a delimiter after string delimiter
Example:
I have done quite an extensive research in this and found that this issue has been around for years but yet to see any solutions given.
Any help will be appreciated.
I think the easiest way to fix this up because you are in charge of the .csv creation is to use a delimiter which is not a comma and leave off the string delimiter. Use a separator which you know will not appear in the file. I've used a pipe in my example, and I clean up the string once it is imported in to the database.
A simple example:
IF EXISTS ( SELECT * FROM sys.external_tables WHERE name = 'delimiterWorking' )
DROP EXTERNAL TABLE delimiterWorking
GO
IF EXISTS ( SELECT * FROM sys.tables WHERE name = 'cleanedData' )
DROP TABLE cleanedData
GO
IF EXISTS ( SELECT * FROM sys.external_file_formats WHERE name = 'ff_delimiterWorking' )
DROP EXTERNAL FILE FORMAT ff_delimiterWorking
GO
CREATE EXTERNAL FILE FORMAT ff_delimiterWorking
WITH (
FORMAT_TYPE = DELIMITEDTEXT,
FORMAT_OPTIONS (
FIELD_TERMINATOR = '|',
--STRING_DELIMITER = '"',
FIRST_ROW = 2,
ENCODING = 'UTF8'
)
);
GO
CREATE EXTERNAL TABLE delimiterWorking (
id INT NOT NULL,
body VARCHAR(8000) NULL
)
WITH (
LOCATION = 'yourLake/someFolder/delimiterTest6.txt',
DATA_SOURCE = ds_azureDataLakeStore,
FILE_FORMAT = ff_delimiterWorking,
REJECT_TYPE = VALUE,
REJECT_VALUE = 0
);
GO
SELECT *
FROM delimiterWorking
GO
-- Fix up the data
CREATE TABLE cleanedData
WITH (
CLUSTERED COLUMNSTORE INDEX,
DISTRIBUTION = ROUND_ROBIN
)
AS
SELECT
id,
body AS originalCol,
SUBSTRING ( body, 2, LEN(body) - 2 ) cleanBody
FROM delimiterWorking
GO
SELECT *
FROM cleanedData
My results:
String Delimiter issue can be avoided if you have the Data lake flat file converted to Parquet format.
Input:
"ID"
"NAME"
"COMMENTS"
"1"
"DAVE"
"Hi "I am Dave" from"
"2"
"AARO"
"AARO"
Steps:
1 Convert Flat file to Parquet format [Using Azure Data factory]
2 Create External File format in Data Lake [Assuming Master key, Scope credentials available]
CREATE EXTERNAL FILE FORMAT PARQUET_CONV
WITH (FORMAT_TYPE = PARQUET,
DATA_COMPRESSION = 'org.apache.hadoop.io.compress.SnappyCodec'
);
3 Create External Table with FILE_FORMAT = PARQUET_CONV
Output:
I believe this is the best option as Microsoft don't have an solution currently to handle this string delimiter occurring with in the data for External table

SQL Threadsafe UPDATE TOP 1 for FIFO Queue

I have a table of invoices being prepared, and then ready for printing.
[STATUS] column is Draft, Print, Printing, Printed
I need to get the ID of the first (FIFO) record to be printed, and change the record status. The operation must be threadsafe so that another process does not select the same InvoiceID
Can I do this (looks atomic to me, but maybe not ...):
1:
WITH CTE AS
(
SELECT TOP(1) [InvoiceID], [Status]
FROM INVOICES
WHERE [Status] = 'Print'
ORDER BY [PrintRequestedDate], [InvoiceID]
)
UPDATE CTE
SET [Status] = 'Printing'
, #InvoiceID = [InvoiceID]
... perform operations using #InvoiceID ...
UPDATE INVOICES
SET [Status] = 'Printed'
WHERE [InvoiceID] = #InvoiceID
or must I use this (for the first statement)
2:
UPDATE INVOICES
SET [Status] = 'Printing'
, #InvoiceID = [InvoiceID]
WHERE [InvoiceID] =
(
SELECT TOP 1 [InvoiceID]
FROM INVOICES WITH (UPDLOCK)
WHERE [Status] = 'Print'
ORDER BY [PrintRequestedDate], [InvoiceID]
)
... perform operations using #InvoiceID ... etc.
(I cannot hold a transaction open from changing status to "Printing" until the end of the process, i.e. when status is finally changed to "Printed").
EDIT:
In case it matters the DB is READ_COMMITTED_SNAPSHOT
I can hold a transaction for both UPDATE STATUS to "Printing" AND get the ID. But I cannot continue to keep transaction open all the way through to changing the status to "Printed". This is an SSRS report, and it makes several different queries to SQL to get various bits of the invoice, and it might crash/whatever, leaving the transaction open.
#Gordon Linoff "If you want a queue" The FIFO sequence is not critical, I would just like invoices that are requested first to be printed first ... "more or less" (don't want any unnecessary complexity ...)
#Martin Smith "looks like a usual table as queue requirement" - yes, exactly that, thanks for the very useful link.
SOLUTION:
The solution I am adopting is from comments:
#lad2025 pointed me to SQL Server Process Queue Race Condition which uses WITH (ROWLOCK, READPAST, UPDLOCK) and #MartinSmith explained what the Isolation issue is and pointed me at Using tables as Queues - which talks about exactly what I am trying to do.
I have not grasped why UPDATE TOP 1 is safe, and UPDATE MyTable SET xxx = yyy WHERE MyColumn = (SELECT TOP 1 SomeColumn FROM SomeTable ORDER BY AnotherColumn) (without Isolation Hints) is not, and I ought to educate myself, but I'm happy just to put the isolation hints in my code and get on with something else :)
Thanks for all the help.
My concern would be duplicate [InvoiceID]
Multiple print requests for the same [InvoiceID]
On the first update ONE row gets set [Status] = 'Printing'
On the second update all [InvoiceID] rows get set [Status] = 'Printed'
This would even set rows with status = 'draft'
Maybe that is what you want
Another process could pick up the same [InvoiceID] before the set [Status] = 'Print'
So some duplicates will print and some will not
I go with comments on use the update lock
This is non-deterministic but you could just take top (1) and skip the order by. You will tend to get the most recent row but it is not guaranteed. If you clear the queue then you get em all.
This demonstrates you can lose 'draft' = 1
declare #invID int;
declare #T table (iden int identity primary key, invID int, status tinyint);
insert into #T values (1, 2), (5, 1), (3, 1), (4, 1), (4, 2), (2, 1), (1, 1), (5, 2), (5, 2);
declare #iden int;
select * from #t order by iden;
declare #rowcount int = 1;
while (#ROWCOUNT > 0)
begin
update top (1) t
set t.status = 3, #invID = t.invID, #iden = t.iden
from #t t
where t.status = '2';
set #rowcount = ##ROWCOUNT;
if(#rowcount > 0)
begin
select #invID, #iden;
-- do stuff
update t
set t.status = 4
from #t t
where t.invID = #invID; -- t.iden = #iden;
select * from #T order by iden;
end
end
ATOMicity of a Single Statement
I think your code's fine as it is. i.e. Because you have a single statement which updates the status to printing as soon as the statement runs the status is updated; so anything running before which searches for print would have updated the same record to printing before your process saw it; so your process would pick a subsequent record, or any process hitting it after your statement had run would see it as printing so would not pick it up`. There isn't really a scenario where a record could pick it up whilst the statement's running since as discussed a single SQL statement should be atomic.
Disclaimer
That said, I'm not enough of an expert to say whether explicit lock hints would help; in my opinion they're not needed as the above is atomic, but others in the comments are likely better informed than me. However, running a test (albeit where database and both threads are run on the same machine) I couldn't create a race condition... maybe if the clients were on different machines / if there was a lot more concurrency you'd be more likely to see issues.
My hope is that others have interpreted your question differently, hence the disagreement.
Attempt to Disprove Myself
Here's the code I used to try to cause a race condition; you can drop it into LINQPad 5, select language C# Program, tweak the connectionstring (and optionally any statements) as required, then run:
const long NoOfRecordsToTest = 1000000;
const string ConnectionString = "Server=.;Database=Play;Trusted_Connection=True;"; //assumes a database called "play"
const string DropFifoQueueTable = #"
if object_id('FIFOQueue') is not null
drop table FIFOQueue";
const string CreateFifoQueueTable = #"
create table FIFOQueue
(
Id bigint not null identity (1,1) primary key clustered
, Processed bit default (0) --0=queued, null=processing, 1=processed
)";
const string GenerateDummyData = #"
with cte as
(
select 1 x
union all
select x + 1
from cte
where x < #NoRowsToGenerate
)
insert FIFOQueue(processed)
select 0
from cte
option (maxrecursion 0)
";
const string GetNextFromQueue = #"
with singleRecord as
(
select top (1) Id, Processed
from FIFOQueue --with(updlock, rowlock, readpast) --optionally include this per comment discussions
where processed = 0
order by Id
)
update singleRecord
set processed = null
output inserted.Id";
//we don't really need this last bit for our demo; I've included in case the discussion turns to this..
const string MarkRecordProcessed = #"
update FIFOQueue
set Processed = 1
where Id = #Id";
void Main()
{
SetupTestDatabase();
var task1 = Task<IList<long>>.Factory.StartNew(() => ExampleTaskForced(1));
var task2 = Task<IList<long>>.Factory.StartNew(() => ExampleTaskForced(2));
Task.WaitAll(task1, task2);
foreach (var processedByBothThreads in task1.Result.Intersect(task2.Result))
{
Console.WriteLine("Both threads processed id: {0}", processedByBothThreads);
}
Console.WriteLine("done");
}
static void SetupTestDatabase()
{
RunSql<int>(new SqlCommand(DropFifoQueueTable), cmd => cmd.ExecuteNonQuery());
RunSql<int>(new SqlCommand(CreateFifoQueueTable), cmd => cmd.ExecuteNonQuery());
var generateData = new SqlCommand(GenerateDummyData);
var param = generateData.Parameters.Add("#NoRowsToGenerate",SqlDbType.BigInt);
param.Value = NoOfRecordsToTest;
RunSql<int>(generateData, cmd => cmd.ExecuteNonQuery());
}
static IList<long> ExampleTaskForced(int threadId) => new List<long>(ExampleTask(threadId)); //needed to ensure prevent lazy loadling from causing issues with our tests
static IEnumerable<long> ExampleTask(int threadId)
{
long? x;
while ((x = ProcessNextInQueue(threadId)).HasValue)
{
yield return x.Value;
}
//yield return 55; //optionally return a fake result just to prove that were there a duplicate we'd catch it
}
static long? ProcessNextInQueue(int threadId)
{
var id = RunSql<long?>(new SqlCommand(GetNextFromQueue), cmd => (long?)cmd.ExecuteScalar());
//Debug.WriteLine("Thread {0} is processing id {1}", threadId, id?.ToString() ?? "[null]"); //if you want to see how we're doing uncomment this line (commented out to improve performance / increase the likelihood of a collision
/* then if we wanted to do the second bit we could include this
if(id.HasValue) {
var markProcessed = new SqlCommand(MarkRecordProcessed);
var param = markProcessed.Parameters.Add("#Id",SqlDbType.BigInt);
param.Value = id.Value;
RunSql<int>(markProcessed, cmd => cmd.ExecuteNonQuery());
}
*/
return id;
}
static T RunSql<T>(SqlCommand command, Func<SqlCommand,T> callback)
{
try
{
using (var connection = new SqlConnection(ConnectionString))
{
command.Connection = connection;
command.Connection.Open();
return (T)callback(command);
}
}
catch (Exception e)
{
Debug.WriteLine(e.ToString());
throw;
}
}
Other comments
The above discussion's really only talking about multiple threads taking the next record from the queue whilst avoiding any single record being picked up by multiple threads. There are a few other points...
Race Condition outside of SQL
Per our discussion, if FIFO is mandatory there are other things to worry about. i.e. whilst your threads will pick up each record in order after that it's up to them. e.g. Thread 1 gets record 10 then Thread 2 gets record 11. Now Thread 2 sends record 11 to the printer before Thread 1 sends record 10. If they're going to the same printer, your prints would be out of order. If they're different printers, not a problem; all prints on any printer are sequential. I'm going to assume the latter.
Exception Handling
If any exception occurs in a thread that's processing something (i.e. so the thread's record is printing) then consideration should be made on how to handle this. One option is to keep that thread retrying; though that may be indefinite if it's some fundamental fault. Another is to put the record to some error status to be handled by another process / where it's accepted that this record won't appear in order. Finally, if the order of invoices in the queue is an ideal rather than a hard requirement you could have the owning thread put the status back to print so that it or another thread can pick up that record to retry (though again, if there's something fundamentally wrong with the record, this may block up the queue).
My recommendation here is the error status; as then you have more visibility on the issue / can dedicate another process to dealing with issues.
Crash Handling
Another issue is that because your update to printing is not held within a transaction, if the server crashes you leave the record with this status in the database, and when your system comes back online it's ignored. Ways to avoid that are to include a column saying which thread is processing it; so that when the system comes back up that thread can resume where it left off, or to include a date stamp so that after some period of time any records with status printing which "timeout" can be swept up / reset to Error or Print statuses as required.
WITH CTE AS
(
SELECT TOP(1) [InvoiceID], [Status], [ThreadId]
FROM INVOICES
WHERE [Status] = 'Print'
OR ([Status] = 'Printing' and [ThreadId] = #ThreadId) --handle previous crash
ORDER BY [PrintRequestedDate], [InvoiceID]
)
UPDATE CTE
SET [Status] = 'Printing'
, [ThreadId] = #ThreadId
OUTPUT Inserted.[InvoiceID]
Awareness of Other Processes
We've mostly been focussed on the printing element; but other processes may also be interacting with your Invoices table. We can probably assume that aside from creating the initial Draft record and updating it to Print once ready for printing, those processes won't touch the Status field. However, the same records could be locked by completely unrelated processes. If we want to ensure FIFO we must not use the ReadPast hint, as some records may have status Print but be locked, so we'd skip over them despite them having the earlier PrintRequestedDate. However, if we want things to print as quickly as possible, with them being in order when otherwise not inconvenient, including ReadPast will allow our print process to skip the locked records and carry on, coming back to handle them once they're released.
Similarly another process may lock our record when it's at Printing status, so we can't update it to mark it as complete. Again, if we want to avoid this from causing a holdup we can make use of the ThreadId column to allow our thread to leave the record at status Printing and come back to clean it up later when it's not locked. Obviously this assumes that the ThreadId column is only used by our print process.
Have a dedicated print queue table
To avoid some of the issues with unrelated processes locking your invoices, move the Status field into its own table; so you only need to read from the invoices table; not update it.
This will also have the advantage that (if you don't care about print history) you can delete the record once done, so you'll get better performance (as you won't have to search through the entire invoices table to find those which are ready to print). That said, there's another option for this one (if you're on SQL2008 or above).
Use Filtered Index
Since the Status column will be updated several times it's not a great candidate for an index; i.e. as the record's position in the index jumps from branch to branch as the status progresses.
However, since we're filtering on it it's also something that would really benefit from having an index.
To get around this contradiction, one option's to use a filtered index; i.e. only index those records we're interested in for our print process; so we maintain a small index for a big benefit.
create nonclustered index ixf_Invoices_PrintStatusAndDate
on dbo.Invoices ([Status], [PrintRequestedDate])
include ([InvoiceId]) --just so we don't have to go off to the main table for this, but can get all we need form the index
where [Status] in ('Print','Printing')
Use "Enums" / Reference Tables
I suspect your example uses strings to keep the demo code simple, however including this for completeness.
Using strings in databases can make things hard to support. Rather than having status be a string value, instead use an ID from a related Statuses table.
create table Statuses
(
ID smallint not null primary key clustered --not identity since we'll likely mirror this enum in code / may want to have defined ids
,Name
)
go
insert Statuses
values
(0, 'Draft')
, (1,'Print')
, (2,'Printing')
, (3,'Printed')
create table Invoices
(
--...
, StatusId smallint foreign key references Statuses(Id)
--...
)

Lua - SQLite3 isn't inserting rows into its database

I'm trying to build an expense app for Android phones, and I need a way to display the expenses. The plan (for my current step) is to allow the user to view their expenses. I want to show a calendar-like screen, and if there is at least one expense for a day, then use a different color for the button.
My problem is in inserting information to the sqlite3 table. Here is my code:
require "sqlite3"
--create path
local path = system.pathForFile("expenses.sqlite", system.DocumentsDirectory )
file = io.open( path, "r" )
if( file == nil )then
-- Doesn't Already Exist, So Copy it In From Resource Directory
pathSource = system.pathForFile( "expenses.sqlite", system.ResourceDirectory )
fileSource = io.open( pathSource, "r" )
contentsSource = fileSource:read( "*a" )
--Write Destination File in Documents Directory
pathDest = system.pathForFile( "expenses.sqlite", system.DocumentsDirectory )
fileDest = io.open( pathDest, "w" )
fileDest:write( contentsSource )
-- Done
io.close( fileSource )
io.close( fileDest )
end
db = sqlite3.open( path )
--setup the table if it doesn't exist
local tableSetup = [[CREATE TABLE IF NOT EXISTS expenses (id INTEGER PRIMARY KEY, amount, description, year, month, day);]]
db:exec(tableSetup)
local tableFill = [[INSERT INTO expenses VALUES (NULL,']] .. 15 .. [[',']] .. "Groceries" .. [[',']] .. 2013 .. [[',']] .. 4 .. [[',']] .. 8 ..[[');]]
db:exec(tableFill)
for row in db:nrows("SELECT * FROM expenses") do
print("hi")
if row.year == dateTable[i].year and row.month == dateTable[i].month and row.day == dateTable[i].day then
flag = dateTabel[i].day
end
end
I have looked everywhere to see if I've used the wrong sqlite3 commands wrong since I'm not very familiar to it, but I tried everything I found and nothing worked. The print("hi")
line doesn't execute, so that tells me that there are no rows in the table.
Also, if I say db:nrows("SELECT year, month, day FROM expenses"), sqlite3 gives me an error saying there is no year column. My overall guess is that I'm not inserting the information into the table properly, but I've tried everything I can think of. Can anyone help?
I figured out that there was an issue with the current version of sqlite3 and the one I have on my computer. Anyway, I changed a couple of lines and it works flawlessly. I changed the select statement and the for loop.
--sqlite statement
local check = "SELECT DISTINCT year, month, day FROM expenses WHERE year = '"..dateTable[i].year.."' AND month = '"..dateTable[i].month.."' AND day = '"..dateTable[i].day.."'"
--check if there is at least one expense for a given day
--if there isn't one, the for loop won't execute
for row in db:nrows(check) do
flag = row.day
end
Then I go on to create a button with a different color if the day number is equal to the flag variable.
This is all inside a another for loop which creates each dateTable[i].

auto increment ID in H2 database

Is there a way to have an auto_incrementing BIGINT ID for a table.
It can be defined like so
id bigint auto_increment
but that has no effect (it does not increment automatically).
I would like to insert all fields but the ID field - the ID field should be provided by the DBMS.
Or do I need to call something to increment the ID counter?
It works for me. JDBC URL: jdbc:h2:~/temp/test2
drop table test;
create table test(id bigint auto_increment, name varchar(255));
insert into test(name) values('hello');
insert into test(name) values('world');
select * from test;
result:
ID NAME
1 hello
2 world
IDENTITY
The modern approach uses the IDENTITY type, for automatically generating an incrementing 64-bit long integer.
This single-word syntax used in H2 is an abbreviated variation of GENERATED … AS IDENTITY defined in the SQL:2003 standard. See summary in PDF document SQL:2003 Has Been Published. Other databases are implementing this, such as Postgres.
CREATE TABLE event_
(
pkey_ IDENTITY NOT NULL PRIMARY KEY , -- ⬅ `identity` = auto-incrementing long integer.
name_ VARCHAR NOT NULL ,
start_ TIMESTAMP WITH TIME ZONE NOT NULL ,
duration_ VARCHAR NOT NULL
)
;
Example usage. No need to pass a value for our pkey column value as it is being automatically generated by H2.
INSERT INTO event_ ( name_ , start_ , stop_ )
VALUES ( ? , ? , ? )
;
And Java.
ZoneId z = ZoneId.of( "America/Montreal" ) ;
OffsetDateTime start = ZonedDateTime.of( 2021 , Month.JANUARY , 23 , 19 , 0 , 0 , 0 , z ).toOffsetDateTime() ;
Duration duration = Duration.ofHours( 2 ) ;
myPreparedStatement.setString( 1 , "Java User Group" ) ;
myPreparedStatement.setObject( 2 , start ) ;
myPreparedStatement.setString( 3 , duration.toString() ) ;
Returning generated keys
Statement.RETURN_GENERATED_KEYS
You can capture the value generated during that insert command execution. Two steps are needed. First, pass the flag Statement.RETURN_GENERATED_KEYS when getting your prepared statement.
PreparedStatement pstmt = conn.prepareStatement( sql , Statement.RETURN_GENERATED_KEYS ) ;
Statement::getGeneratedKeys
Second step is to call Statement::getGeneratedKeys after executing your prepared statement. You get a ResultSet whose rows are the identifiers generated for the created row(s).
Example app
Here is an entire example app. Running on Java 14 with Text Blocks preview feature enabled for fun. Using H2 version 1.4.200.
package work.basil.example;
import org.h2.jdbcx.JdbcDataSource;
import java.sql.*;
import java.time.*;
import java.util.Objects;
public class H2ExampleIdentity
{
public static void main ( String[] args )
{
H2ExampleIdentity app = new H2ExampleIdentity();
app.doIt();
}
private void doIt ( )
{
JdbcDataSource dataSource = Objects.requireNonNull( new JdbcDataSource() ); // Implementation of `DataSource` bundled with H2.
dataSource.setURL( "jdbc:h2:mem:h2_identity_example_db;DB_CLOSE_DELAY=-1" ); // Set `DB_CLOSE_DELAY` to `-1` to keep in-memory database in existence after connection closes.
dataSource.setUser( "scott" );
dataSource.setPassword( "tiger" );
String sql = null;
try (
Connection conn = dataSource.getConnection() ;
)
{
sql = """
CREATE TABLE event_
(
id_ IDENTITY NOT NULL PRIMARY KEY, -- ⬅ `identity` = auto-incrementing integer number.
title_ VARCHAR NOT NULL ,
start_ TIMESTAMP WITHOUT TIME ZONE NOT NULL ,
duration_ VARCHAR NOT NULL
)
;
""";
System.out.println( "sql: \n" + sql );
try ( Statement stmt = conn.createStatement() ; )
{
stmt.execute( sql );
}
// Insert row.
sql = """
INSERT INTO event_ ( title_ , start_ , duration_ )
VALUES ( ? , ? , ? )
;
""";
try (
PreparedStatement pstmt = conn.prepareStatement( sql , Statement.RETURN_GENERATED_KEYS ) ;
)
{
ZoneId z = ZoneId.of( "America/Montreal" );
ZonedDateTime start = ZonedDateTime.of( 2021 , 1 , 23 , 19 , 0 , 0 , 0 , z );
Duration duration = Duration.ofHours( 2 );
pstmt.setString( 1 , "Java User Group" );
pstmt.setObject( 2 , start.toOffsetDateTime() );
pstmt.setString( 3 , duration.toString() );
pstmt.executeUpdate();
try (
ResultSet rs = pstmt.getGeneratedKeys() ;
)
{
while ( rs.next() )
{
int id = rs.getInt( 1 );
System.out.println( "generated key: " + id );
}
}
}
// Query all.
sql = "SELECT * FROM event_ ;";
try (
Statement stmt = conn.createStatement() ;
ResultSet rs = stmt.executeQuery( sql ) ;
)
{
while ( rs.next() )
{
//Retrieve by column name
int id = rs.getInt( "id_" );
String title = rs.getString( "title_" );
OffsetDateTime odt = rs.getObject( "start_" , OffsetDateTime.class ); // Ditto, pass class for type-safety.
Instant instant = odt.toInstant(); // If you want to see the moment in UTC.
Duration duration = Duration.parse( rs.getString( "duration_" ) );
//Display values
ZoneId z = ZoneId.of( "America/Montreal" );
System.out.println( "id_" + id + " | start_: " + odt + " | duration: " + duration + " ➙ running from: " + odt.atZoneSameInstant( z ) + " to: " + odt.plus( duration ).atZoneSameInstant( z ) );
}
}
}
catch ( SQLException e )
{
e.printStackTrace();
}
}
}
Next, see results when run.
Instant, OffsetDateTime, & ZonedDateTime
At the time of this execution, my JVM’s current default time zone is America/Los_Angeles. At the point in time of the stored moment (January 23, 2021 at 7 PM in Québec), the zone America/Los_Angeles had an offset-from-UTC of eight hours behind. So the OffsetDateTime object returned by the H2 JDBC driver is set to an offset of -08:00. This is a distraction really, so in real work I would immediately convert that OffsetDateTime to either an Instant for UTC or ZonedDateTime for a specific time zone I had in mind. Be clear in understanding that the Instant, OffsetDateTime, and ZonedDateTime objects would all represent the same simultaneous moment, the same point on the timeline. Each views that same moment through a different wall-clock time. Imagine 3 people in California, Québec, and Iceland (whose zone is UTC, an offset of zero) all talking in a conference call end they each looked up at the clock on their respective wall at the same coincidental moment.
generated key: 1
id_1 | start_: 2021-01-23T16:00-08:00 | duration: PT2H ➙ running from: 2021-01-23T19:00-05:00[America/Montreal] to: 2021-01-23T21:00-05:00[America/Montreal]
By the way, in real work on an app booking future appointments, we would use a different data type in Java and in the database.
We would have used LocalDateTime and ZoneId in Java. In the database, we would have used a data type akin to the SQL standard type TIMESTAMP WITHOUT TIME ZONE with a second column for the name of the intended time zone. When retrieving values from the database to build an scheduling calendar, we would apply the time zone to the stored date-time to get a ZonedDateTime object. This would allow us to book appointments for a certain time-of-day regardless of changes to the offset-from-UTC made by the politicians in that jurisdiction.
Very simple:
id int auto_increment primary key
H2 will create Sequence object automatically
You can also use default:
create table if not exists my(id int auto_increment primary key,s text);
insert into my values(default,'foo');
id bigint(size) zerofill not null auto_increment,

Resources