Question is: does ds.put(employee) happen in transaction? Or does the outer transaction get erased/overriden by the transaction in saveRecord(..)?
Once error is thrown at line datastore.put(..) at some point in the for-loop (let's say i==5), will previous puts originating on the same line get rollbacked?
What about puts happening in the saveRecord(..). I suppose those will not get rollbacked.
DatastoreService datastore = DatastoreServiceFactory.getDatastoreService()
Transaction txn = datastore.beginTransaction();
try {
for (int i=0; 1<10; i++) {
Key employeeKey = KeyFactory.createKey("Employee", "Joe");
Entity employee = datastore.get(employeeKey);
employee.setProperty("vacationDays", 10);
datastore.put(employee);
Entity employeeRecord = createRecord("record", employeeKey);
saveRecord(employeeRecord);
}
txn.commit();
} finally {
if (txn.isActive()) {
txn.rollback();
}
}
public void saveRecord(Entity entity) {
datastore.beginTransaction();
try {
// do some logic in here, delete activity and commit txn
datastore.put(entity);
} finally {
if (datastore.getCurrentTransaction().isActive()) {
datastore.getCurrentTransaction().rollback();
}
}
}
OK, I'll assume you are using low-level Datastore API. Note that getTransaction() does not exist. I'll assume you meant datastoreService.getCurrentTransaction().
DatastoreService.beginTransaction() will return a Transaction, that is considered a current transaction on the same thread until you call beginTransaction() again. Since you call beginTransaction() in a loop inside "outer" transaction, it breaks your "outer" code: after the loop is finished ds.getCurrentTransaction() does not return the same transaction. Also, put() implicitly uses current transaction.
So first you must fix outer code to save transaction as shown in example:
public void put(EventPlan eventPlan) {
Transaction txn = ds.beginTransaction();
try {
for (final Activity activity : eventPlan.getActivities()) {
save(activity, getPlanKey(eventPlan)); // PUT
// IMPORTANT - also pass transaction and use it
// (I assume this is some internal helper method)
ds.put(txn, activity, getSubPlanKey(eventPlan)); //subplan's parent is eventPlan
}
txn.commit();
} finally {
if (txn.isActive())
txn.rollback();
}
}
Now on to questions:
Yes, because all puts are now part of the same transaction (after you fix the code) and you call txn.rollback() on it in case of errors.
No, of course not. They are part of different transactions.
Related
Can you guys please help me with writing a gamma script for logging data into a database table? The tags are inside a domain created in OPC DA of Cogent Datahub. The only condition that needs to be satisfied is the script should be logging all points in the domain every one second along with their value and timestamp.
require ("Application");
require ("ODBCThreadSupport");
require ("Time");
require ("Quality");
class LogData Application
{
DSN = "your ODBC DSN name"; // The DSN name to use for the database
connection
username = "Database_User"; // The user name for connecting to the
database
password = "*****"; // The password for connecting to the database
tablename = "table1"; // The name of the database table
cachefile = "C:\Users\AppData\cache.txt"; // Base name for the disk cache file
domain = "Domain name"; // The domain in which to log all points
tableclass;
thread;
mappedPoints = new Dictionary();
prevcount = 0;
}
/* If there is something we only want to perform on the first connection, we can
test
* is_first_connect to perform the code only once.
*/
method LogData.onConnect()
{
princ ("Connection succeeded\n");
if (.thread.is_first_connect)
{
// Start the sequence defined by the AddInitStage calls in the constructor
.thread.BeginAsyncInit();
}
}
/* If the connection fails after having been
* connected, this method is called.
*/
method LogData.onConnectFail()
{
princ ("Connection closed: ", SQLResult.Description, "\n");
}
/* Map the table in the set of table definitions that matches the name in
.tablename
* into a Gamma class. This lets us easily convert between class instances and
rows
* in the table.
*/
method LogData.mapTable(name, tabledefinitions)
{
//princ("Mapping table\n");
.tableclass = .thread.ClassFromTable(name, tabledefinitions);
//princ("Table class = ", .tableclass, "\n");
}
method LogData.startLogging()
{
.registerPoints();
}
/* Set up the timer or event handler functions to write to the table. */
method LogData.registerPoints()
{
/* Find all points in the domain */
local info = datahub_domaininfo(.domain)[0];
if (info.n_points != .prevcount)
{
local points = datahub_points(.domain, nil, nil);
local pointsym;
princ(info.n_points - .prevcount, " new points are being added to
logging\n");
with point in points do
{
// Filter out branch points
if ((point.flags & 0x30) == 0)
{
pointsym = symbol(string(point.domain,":", point.name));
if (!.mappedPoints.contains(pointsym))
{
local PointName = string(pointsym);
.TimerEvery(01,`(#self).writeData(#PointName));
//.mappedPoints.add(pointsym, pointsym);
}
}
}
.prevcount = info.n_points;
}
}
method LogData.writeData(pointsymbol)
{
local row = new (.tableclass);
local pttime, ptltime;
local timestring;
local point;
// Generate a timestamp in database-independent format to the millisecond.
// Many databases strip the milliseconds from a timestamp, but it is harmless
// to provide them in case the database can store them.
point = PointMetadata(pointsymbol);
//princ(point,"\n");
if (point && number_p(point.value))
{
pttime = WindowsTimeToUnixTime(point.timestamp);
//princ(point,"\n");
ptltime = localtime(pttime);
//princ(ptltime,"\n");
if (!ptltime)
ptltime = localtime(0);
timestring = format("{'%04d-%02d-%02d %02d:%02d:%02d'}",
ptltime.year+1900, ptltime.mon+1, ptltime.mday, ptltime.hour, ptltime.min,
ptltime.sec);
//princ(timestring,"\n");
// Fill the row. Since we mapped the table into a Gamma class, we can
access
// the rows in the column as member variables of the mapped class.
row.ptname = string(pointsymbol);
row.ptvalue = point.value;
row.pttime = timestring;
// Perform the insertion. In this case we are providing no callback on
completion.
.thread.Insert(row, nil);
}
}
/* Write the 'main line' of the program here. */
method LogData.constructor ()
{
// Create and configure the database connection object
.thread = new ODBCThread();
.thread.Configure(.DSN, .username, .password, STORE_AND_FORWARD, .cachefile, 0);
// Query the table and map it to a class for each insertion. We want to run an
asynchronous event
// within the asynchronous initialization stage, so to do that we specify the
special method
// cbInitStage as the callback function of our asynchronous event
(GetTableInfo). We deal with
// the return from the GetTableInfo in the onSuccess argument of the init stage.
.thread.AddInitStage(`(#.thread).GetTableInfo("", "", (#.tablename),
"TABLE,VIEW",
`(#.thread).cbInitStage()),
`(#self).mapTable(#.tablename, SQLTables), nil);
//.thread.AddInitStage(`(#.thread).GetTableInfo("", "", (#.tablename),
"TABLE,VIEW",
// `(#self).mapTable(#.tablename, SQLTables)),
`(#.thread).cbInitStage(), nil);
// Do not start writing data to the table until we have successfully created and
mapped
// the table to a class. If we wanted to start writing data immediately, then
we would
// create the table class beforehand instead of querying the database for the
table
// definition. Then, even if the database were unavailable we could still cache
to the
// local disk until the database was ready.
.thread.AddInitStage(nil, `(#self).startLogging(), nil);
// Set up the callback functions for various events from the database thread
.thread.OnConnectionSucceeded = `(#self).onConnect();
.thread.OnConnectionFailed = `(#self).onConnectFail();
.thread.OnFileSystemError = `princ("File System Error: ", SQLResult, "\n");
.thread.OnODBCError = `princ("ODBC Error: ", SQLResult, "\n");
.thread.OnExecuteStored = nil;
.thread.Start();
// Create a menu item in the system tray that allows us to open a window to
monitor
// the performance of the ODBC thread. The menu strings can be edited as
desired.
.AddCustomSubMenu("ODBC Logging");
.AddCustomMenuItem("Monitor Performance",
`(#.thread).CreateMonitorWindow((#self), "ODBC Monitor"));
// Automatically update the point list every 1 seconds in case new points are v
added
// to the domain.
//.TimerEvery(01, `(#self).registerPoints());
}
/* Any code to be run when the program gets shut down. */
method LogData.destructor ()
{
if (instance_p(.thread))
destroy(.thread);
}
/* Start the program by instantiating the class. */
ApplicationSingleton (LogData);
Major parts of this Gamma script are the constructor,destructor,classes and methods. This program first initializes an ODBC connection using provided details and write each row of data using 'registerpoints' and 'writedata' methods. Please find additional details of each lines from comments in the program.
We are using JDBC driver to connect to Snowflake and perform inserts. Using setQueryTimeout on preparedStatement to get the desired timeout behavior. Auto commit is kept default i.e. enabled.
We are observing, On timeout driver tries to cancel the query, however query still committing data into table.
Below is the sample program which uses 1 second as timeout for quick reproducible scenario -
boolean flag = false;
PreparedStatement ps = connection.prepareStatement("insert into Test_Int values (?)");
ps.setQueryTimeout(1);
for (int v =1; v<200; v++) {
ps.setInt(1, v);
ps.addBatch();
flag = true;
if(v%50 == 0) {
try {
ps.executeBatch();
flag = false;
} catch (SQLException se) {
//do not stop execution continue with other batches
}
}
}
if(flag) {
try {
ps.executeBatch();
} catch (SQLException se) {
//do not stop execution continue with other batches
}
}
As per requirement, we are continuing with next batch on SQLException, and all data get committed into table eventhough there is timeout.
Questions -
How does timeout work?
Is there any retry or connection renewal also done by driver in this case?
If the driver initiate cancel command, would query cancellation on DB guaranteed with rollback or it depends?
How to handle timeout-related exceptions in better way in the code?
Thanks you for the help in advance.
Suppose that I have a telephony application. I have a feature that I want to try calling an array of users one by one and break the sequence whenever one of the users accepts call, or when the complete operation is cancelled.
I will try to simplify it like this in pseudocode:
for(user in users) {
result = callUserCommand(user);
if(result == "accepted" || result == "cancelled") {
break;
}
}
Here, the callUserCommand is a RACCommand that needs to be async. And it can actually have three return values: "accepted", "cancelled", "declined".
Accepted and Cancelled will break the sequence of operations and won't execute the rest.
Declined, should continue with the execution of the rest of the sequence.
I tried with something like the following, but really couldn't accomplish exactly the thing I described above.
RACSignal *signal = [RACSignal concat:[users.rac_sequence map:^(User * user) {
return [self.callUserCommand execute:user];
}]];
[signal subscribeNext:^(id x) {
} error:^(NSError *error) {
} completed:^{
}];
If I understood correctly you would like to execute the sequence one by one until one of the call gets accepted or cancelled.
Maybe you could give takeUntil or takeWhile a try. I would write this scenario with RAC like this:
NSArray* users = #[#"decline", #"decline", #"decline", #"accept", #"decline"];
[[[[[users.rac_sequence signal]
flattenMap:^RACStream *(NSString* userAction) {
NSLog(#"Calling user (who will %#):", userAction);
// return async call signal here
return [RACSignal return:userAction];
}]
takeWhileBlock:^BOOL(NSString* resultOfCall) {
return [resultOfCall isEqualToString:#"decline"];
}]
doCompleted:^{
NSLog(#"Terminated");
}]
subscribeNext:^(NSString* userAction) {
NSLog(#"User action: %#", userAction);
}];
In the sample code above the last user who would decline the call won't be called.
Is Hazelcast always blocking in case initial.min.cluster.size is not reached? If not, under which situations is it not?
Details:
I use the following code to initialize hazelcast:
Config cfg = new Config();
cfg.setProperty("hazelcast.initial.min.cluster.size",Integer.
toString(minimumInitialMembersInHazelCluster)); //2 in this case
cfg.getGroupConfig().setName(clusterName);
NetworkConfig network = cfg.getNetworkConfig();
JoinConfig join = network.getJoin();
join.getMulticastConfig().setEnabled(false);
join.getTcpIpConfig().addMember("192.168.0.1").addMember("192.168.0.2").
addMember("192.168.0.3").addMember("192.168.0.4").
addMember("192.168.0.5").addMember("192.168.0.6").
addMember("192.168.0.7").setRequiredMember(null).setEnabled(true);
network.getInterfaces().setEnabled(true).addInterface("192.168.0.*");
join.getMulticastConfig().setMulticastTimeoutSeconds(MCSOCK_TIMEOUT/100);
hazelInst = Hazelcast.newHazelcastInstance(cfg);
distrDischargedTTGs = hazelInst.getList(clusterName);
and get log messages like
debug: starting Hazel pullExternal from Hazelcluster with 1 members.
Does that definitely mean there was another member that has joined and left already? It does not look like that would be the case from the log files of the other instance. Hence I wonder whether there are situtations where hazelInst = Hazelcast.newHazelcastInstance(cfg); does not block even though it is the only instance in the hazelcast cluster.
The newHazelcastInstance blocks till the clusters has the required number of members.
See the code below for how it is implemented:
private static void awaitMinimalClusterSize(HazelcastInstanceImpl hazelcastInstance, Node node, boolean firstMember)
throws InterruptedException {
final int initialMinClusterSize = node.groupProperties.INITIAL_MIN_CLUSTER_SIZE.getInteger();
while (node.getClusterService().getSize() < initialMinClusterSize) {
try {
hazelcastInstance.logger.info("HazelcastInstance waiting for cluster size of " + initialMinClusterSize);
//noinspection BusyWait
Thread.sleep(TimeUnit.SECONDS.toMillis(1));
} catch (InterruptedException ignored) {
}
}
if (initialMinClusterSize > 1) {
if (firstMember) {
node.partitionService.firstArrangement();
} else {
Thread.sleep(TimeUnit.SECONDS.toMillis(3));
}
hazelcastInstance.logger.info("HazelcastInstance starting after waiting for cluster size of "
+ initialMinClusterSize);
}
}
If you set the logging on debug then perhaps you can see better what is happening. Member joining and leaving should already be visible under info.
Some time ago we implemented a warehouse management app that keeps track of quantities of each product we have in the store. We solved the problem of concurrent access to data with database locks (select for update), but this approach led to poor performance when many clients try to consume product quantities from the same store. Note that we manage only a small set of product types (less than 10) so the degree of concurrency could be heavy (also, we don't care of stock re-fill). We thought to split each resource quantity in smaller "buckets", but this approach could lead to starvation for clients that try to consume a quantity that is bigger than each bucket capacity: we should manage buckets merge and so on...
My question is: there are some broadly-accepted solutions to this problem? I also looked for academic articles but the topic seems too wide.
P.S. 1:
our application runs in a clustered environment, so we cannot rely on the application concurrency control. The question aims to find an algorithm that structures and manages the data in a different way than a single row, but keeping all the advantages that a db transaction (using locks or not) has.
P.S. 2: for your info, we manage a wide number of similar warehouses, the example focuses on a single one, but we keep all the data in one db (prices are all the same, etc).
Edit: The setup below will still work on a cluster if you use a queueing program that can coordinate among multiple processes / servers, e.g. RabbitMQ.
You can also use a simpler queueing algorithm that only uses the database, with the downside that it requires polling (whereas a system like RabbitMQ allows threads to block until a message is available). Create a Requests table with a column for unique requestIds (e.g. a random UUID) that acts as the primary key, a timestamp column, a respourceType column, and an integer requestedQuantity column. You'll also need a Logs table with a unique requestId column that acts as the primary key, a timestamp column, a resourceType column, an integer requestQuantity column, and a boolean/tinyint/whatever success column.
When a client requests a quantity of ResourceX it generates a random UUID and adds a row to the Requests table using the UUID as the requestId, and then polls the Logs table for the requestId. If the success column is true then the request succeeded, else it failed.
The server with the database assigns one thread or process to each resource, e.g. ProcessX is in charge of ResourceX. ProcessX retrieves all rows from the Requests table where resourceType = ResourceX, sorted by timestamp, and then deletes them from Requests; it then processes each request in order, decrementing an in-memory counter for each successful request, and at the end of processing the requests it updates the quantity of ResourceX on the Resources table. It then writes each request and its success status to the Logs table. It then retrieves all of the requests from Requests where requestType = RequestX again, etc.
It may be slightly more efficient to use an autoincrement integer as the Requests primary key, and to have ProcessX sort by primary key instead of by timestamp.
One option is to assign one DAOThread per resource - this thread is the only thing that accesses that resource's database table so that there's no locking at the database level. Workers (e.g. web sessions) request resource quantities using a concurrent queue - the example below uses a Java BlockingQueue, but most languages will have some sort of concurrent queue implementation you can use.
public class Request {
final int value;
final BlockingQueue<ReturnMessage> queue;
}
public class ReturnMessage {
final int value;
final String resourceType;
final boolean isSuccess;
}
public class DAOThread implements Runnable {
private final int MAX_CHANGES = 10;
private String resourceType;
private int quantity;
private int changeCount = 0;
private DBTable table;
private BlockingQueue<Request> queue;
public DAOThread(DBTable table, BlockingQueue<Request> queue) {
this.table = table;
this.resourceType = table.select("resource_type");
this.quantity = table.select("quantity");
this.queue = queue;
}
public void run() {
while(true) {
Requester request = queue.take();
if(request.value <= quantity) {
quantity -= request.value;
if(++changeCount > MAX_CHANGES) {
changeCount = 0;
table.update("quantity", quantity);
}
request.queue.offer(new ReturnMessage(request.value, resourceType, true));
} else {
request.queue.offer(new ReturnMessage(request.value, resourceType, false));
}
}
}
}
public class Worker {
final Map<String, BlockingQueue<Request>> dbMap;
final SynchronousQueue<ReturnMessage> queue = new SynchronousQueue<>();
public class WorkerThread(Map<String, BlockingQueue<Request>> dbMap) {
this.dbMap = dbMap;
}
public boolean request(String resourceType, int value) {
dbMap.get(resourceType).offer(new Request(value, queue));
return queue.take();
}
}
The Workers send resource requests to the appropriate DAOThread's queue; the DAOThread processes these requests in order, either updating the local resource quantity if the request's value doesn't exceed the quantity and returning a Success, else leaving the quantity unchanged and returning a Failure. The database is only updated after ten updates to reduce the amount of IO; the larger MAX_CHANGES is, the more complicated it will be to recover from system failure. You can also have a dedicated IOThread that does all of the database writes - this way you don't need to duplicate any logging or timing (e.g. there ought to be a Timer that flushes the current quantity to the database after every few seconds).
The Worker uses a SynchronousQueue to wait for a response from the DAOThread (a SynchronousQueue is a BlockingQueue that can only hold one item); if the Worker is running in its own thread the you may want to replace this with a standard multi-item BlockingQueue so that the Worker can process the ReturnMessages in any order.
There are some databases e.g. Riak that have native support for counters, so this might improve your IO thoughput and reduce or eliminate the need for a MAX_CHANGES.
You can further increase throughput by introducing BufferThreads to buffer the requests to the DAOThreads.
public class BufferThread implements Runnable {
final SynchronousQueue<ReturnMessage> returnQueue = new SynchronousQueue<>();
final int BUFFERSIZE = 10;
private DAOThread daoThread;
private BlockingQueue<Request> queue;
private ArrayList<Request> buffer = new ArrayList<>(BUFFERSIZE);
private int tempTotal = 0;
public BufferThread(DAOThread daoThread, BlockingQueue<Request> queue) {
this.daoThread = daoThread;
this.queue = queue;
}
public void run() {
while(true) {
Request request = queue.poll(100, TimeUnit.MILLISECONDS);
if(request != null) {
tempTotal += request.value;
buffer.add(request);
}
if(buffer.size() == BUFFERSIZE || request == null) {
daoThread.queue.offer(new Request(tempTotal, returnQueue));
ReturnMessage message = returnQueue.take();
if(message.isSuccess()) {
for(Request request: buffer) {
request.queue.offer(new ReturnMessage(request.value, daoThread.resourceType, message.isSuccess));
}
} else {
// send unbuffered requests to DAOThread to see if any can be satisfied
for(Request request: buffer) {
daoThread.queue.offer(request);
}
}
buffer.clear();
tempTotal = 0;
}
}
}
}
The Workers send their requests to the BufferThreads, who then wait until they've buffered BUFFERSIZE requests or have waited for 100ms for a request to come through the buffer (Request request = queue.poll(100, TimeUnit.MILLISECONDS)), at which point they forward the buffered message to the DAOThread. You can have multiple buffers per DAOThread - rather than sending a Map<String, BlockingQueue<Request>> to the Workers you instead send a Map<String, ArrayList<BlockingQueue<Request>>>, one queue per BufferThread, with the Worker either using a counter or a random number generator to determine which BufferThread to send a request to. Note that if BUFFERSIZE is too large and/or if you have too many BufferThreads then Workers will suffer from long pause times as they wait for the buffer to fill up.