I'm trying to manage a decentralized DB around a huge number of partial DB instances. Each instance has a subset of the whole data and they are all nodes and clients, thus asking for some data the query must be spread to every (group) instance and which one have it will return the data.
Due to avoid lost of data if one instance goes down, I figured out they must replicate its contents with some others. How this scenario can be configured with Ignite?
Supose I have a table with the name and last access datetime of users in a distributed application, like ...
class UserLogOns
{
string UserName;
DateTime LastAccess;
}
Now when the program starts I prepare Ingite for work as a decentralizad DB ...
static void Main(string[] args)
{
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
// Override local port.
commSpi.LocalPort = 44444;
commSpi.LocalPortRange = 0;
IgniteConfiguration cfg = new IgniteConfiguration();
// Override default communication SPI.
cfg.CommunicationSpi = commSpi;
using (var ignite = Ignition.Start(cfg))
{
var cfgCache = new CacheConfiguration("mio");
cfgCache.AtomicityMode = CacheAtomicityMode.Transactional;
var cache = ignite.GetOrCreateCache<string, UserLogOns>(cfgCache);
cache.Put(Environment.MachineName, new UserLogOns { UserName = Environment.MachineName, LastAccess = DateTime.UtcNow });
}
}
And now ... I want to get LastAccess of other "computerB" when ever it was ..
Is this correct? How can it be implemented?
It depends on the exact use-case that you want to implement. In general, Ignite provides out of the box everything that you mentioned here.
This is a good way to start with using SQL in Ignite: https://apacheignite-sql.readme.io/docs
Create table with "template=partitioned" instead of "replicated" as it is shown in the example here: https://apacheignite-sql.readme.io/docs/getting-started#section-creating-tables, configure number of backups and select a field to be affinity key (a field that is used to map specific entries to cluster node) and just run some queries.
Also check out the concept of baseline topology if you are going to use native persistence: https://apacheignite.readme.io/docs/baseline-topology.
In-memory mode will trigger rebalance between nodes on each server topology change (node that can store data in/out) automatically.
Related
I get a Dotmim.Sync.SyncException when calling the agent.SynchronizeAsync(tables) function:
Exception: Seems you are trying another Setup tables that what is stored in your server scope database. Please make a migration or create a new scope
This is my code:
public static async Task SynchronizeAsync()
{
var serverProvider = new SqlSyncProvider(serverConnectionString);
// Second provider is using plain old Sql Server provider, relying on triggers and tracking tables to create the sync environment
var clientProvider = new SqliteSyncProvider(Path.Combine(FileSystem.AppDataDirectory, "treesDB.db3"));
// Tables involved in the sync process:
var tables = new string[] { "Trees" };
// Creating an agent that will handle all the process
var agent = new SyncAgent(clientProvider, serverProvider);
// Launch the sync process
var s1 = await agent.SynchronizeAsync(tables);
await agent.LocalOrchestrator.UpdateUntrackedRowsAsync();
var s2 = await agent.SynchronizeAsync();
}
I'm the author of Dotmim.Sync
Do not hesitate to to fill an issue on Github if you are still struggling.
Regarding your issue, I think you have made some tests with different tables.
You need to stick with a set of tables, because DMS needs to create different things (triggers / stored proc and so on)
If you want to test different setups, you need to define differents scopes.
You have a complete documentation on https://dotmimsync.readthedocs.io/
I have a WPF application where the user creates entities in the database. Each entity has some metadata and an interval field. For each entity I want to create a job with the interval provided and store them in the AdoJobStore.
Now since the WPF app will not always be running, I want a create a Windows Service that reads the jobs data from the AdoJobStore and run those jobs.
So essentially there are these 2 tiers. Now I have setup the Quartz tables already in my existing database. My question is:
How to create/edit/delete jobs from my WPF application
How to inform my windows service to run the jobs (every time an entity is created in database)
I have read through a lot of blogs but these 2 primary questions are a bit unclear to me. I would really appreciate some example code on how to achieve and may be structure my solution.
Thanks
You use Zero Thread Scheduler to schedule jobs. Example scheduler initialization code:
var properties = new NameValueCollection();
properties["quartz.scheduler.instanceId"] = "AUTO";
properties["quartz.threadPool.type"] = "Quartz.Simpl.ZeroSizeThreadPool, Quartz";
properties["quartz.jobStore.type"] = "Quartz.Impl.AdoJobStore.JobStoreTX, Quartz";
properties["quartz.jobStore.driverDelegateType"] = "Quartz.Impl.AdoJobStore.SqlServerDelegate, Quartz";
properties["quartz.jobStore.useProperties"] = "true";
properties["quartz.jobStore.dataSource"] = "default";
properties["quartz.jobStore.tablePrefix"] = tablePrefix;
properties["quartz.jobStore.clustered"] = "false";
properties["quartz.dataSource.default.connectionString"] = connectionString;
properties["quartz.dataSource.default.provider"] = "SqlServer-20";
schedFactory = new StdSchedulerFactory(properties);
BaseScheduler = schedFactory.GetScheduler();
Example scheduling function:
protected ITrigger CreateSimpleTrigger(string tName, string tGroup, IJobDetail jd, DateTime startTimeUtc,
DateTime? endTimeUtc, int repeatCount, TimeSpan repeatInterval, Dictionary<string, string> dataMap,
string description = "")
{
if (BaseScheduler.GetTrigger(new TriggerKey(tName, tGroup)) != null) return null;
var st = TriggerBuilder.Create().
WithIdentity(tName, tGroup).
UsingJobData(new JobDataMap(dataMap)).
StartAt(startTimeUtc).
EndAt(endTimeUtc).
WithSimpleSchedule(x => x.WithInterval(repeatInterval).WithRepeatCount(repeatCount)).
WithDescription(description).
ForJob(jd).
Build();
return st;
}
Obviously, you'll need to provide all relevant fields in your UI and pass values from those fields into the function. Example screenshot of some of the required fields:
Your Windows Service will initialize a Multi Thread Scheduler in OnStart() method in a very similar fashion to the way that Zero Thread Scheduler was initialized above and that Multi Thread Scheduler will monitor all the triggers in your database and start your jobs as specified in those triggers. Quartz.net will do all the heavy lifting in that regard. Once you scheduled your jobs and triggers are in the database all you need to do is initialize that Multi Thread Scheduler, connect it to the database containing triggers and it will keep on firing those jobs and execute your code as long as the service is running.
We noticed a slight oddity in the Sitecore API code. The code is below for your reference. The code is trying to get a database by doing new Database(database). But randomly it was failing.
This code worked for a while with Database db = new Database(database); but started failing randomly yesterday. When we changed the code to Database db = Database.GetDatabase(database);, the code started working again. What is the difference between the two approaches and what is recommended by Sitecore?
I've seen this happen twice now - multiple times in production and a couple of times in my development environment.
public static void DeleteItem(string id, stringdatabase)
{
//get the database
Database db = new Database(database);
//get the item
item = db.GetItem(new ID(id));
if (item != null)
{
using(new Sitecore.SecurityModel.SecurityDisabler())|
{
//delete the item
item.Delete();
}
}
}
A common way you will see people get a specific database is:
Sitecore.Data.Database master = Sitecore.Configuration.Factory.GetDatabase("master");
This is equivalent to Sitecore.Data.Database.GetDatabase("master").
When you call either of these methods it will first check the cache for the database. If not found it will build up the database with all of the configuration values within the config file via reflection. Once the database is created it will be placed in the cache for future use.
When you use the constructor on the database it is simply creating a rather empty database object. I am rather suprised to hear it was working at all when you used this method.
The proper approach to get a specific database would be to use:
Sitecore.Configuration.Factory.GetDatabase("master");
// or
Sitecore.Data.Database.GetDatabase("master");
If you are looking to get the database used with the current request (aka context database) you can use Sitecore.Context.Database. You can also use Sitecore.Context.ContentDatabase.
Here is some background about my app:
I am developing an Android app that will display a random quote or verse to the user. For this I am using an SQLite database. The size of the DB would be approximately 5K to 10K records, possibly increasing to upto 1M in later versions as new quotes and verses are added. Thus the user would need to update the DB as and when newer versions are of the app or DB are released.
After reading through some forums online, there seem to be two feasible ways I could provide the DB:
1. Bundle it along with the .APK file of the app, or
2. Upload it to my app's website from where users will have to download it
I want to know which method would be better (if there is yet another approach other than these, please do let me know).
After pondering this problem for some time, I have these thoughts regarding the above approaches:
Approach 1:
Users will obtain the DB along with the app, and won't have to download it separately. Installation would thereby be easier. But, users will have to reinstall the app every time there is a new version of the DB. Also, if the DB is large, it will make the installable too cumbersome.
Approach 2:
Users will have to download the full DB from the website (although I can provide a small, sample version of the DB via Approach 1). But, the installer will be simpler and smaller in size. Also, I would be able to provide future versions of the DB easily for those who might not want newer versions of the app.
Could you please tell me from a technical and an administrative standpoint which approach would be the better one and why?
If there is a third or fourth approach better than either of these, please let me know.
Thank you!
Andruid
I built a similar app for Android which gets periodic updates with data from a government agency. It's fairly easy to build an Android compatible db off the device using perl or similar and download it to the phone from a website; and this works rather well, plus the user gets current data whenever they download the app. It's also supposed to be possible to throw the data onto the sdcard if you want to avoid using primary data storage space, which is a bigger concern for my app which has a ~6Mb database.
In order to make Android happy with the DB, I believe you have to do the following (I build my DB using perl).
$st = $db->prepare( "CREATE TABLE \"android_metadata\" (\"locale\" TEXT DEFAULT 'en_US')");
$st->execute();
$st = $db->prepare( "INSERT INTO \"android_metadata\" VALUES ('en_US')");
$st->execute();
I have an update activity which checks weather updates are available and if so presents an "update now" screen. The download process looks like this and lives in a DatabaseHelperClass.
public void downloadUpdate(final Handler handler, final UpdateActivity updateActivity) {
URL url;
try {
close();
File f = new File(getDatabasePath());
if (f.exists()) {
f.delete();
}
getReadableDatabase();
close();
url = new URL("http://yourserver.com/" + currentDbVersion + ".sqlite");
URLConnection urlconn = url.openConnection();
final int contentLength = urlconn.getContentLength();
Log.i(TAG, String.format("Download size %d", contentLength));
handler.post(new Runnable() {
public void run() {
updateActivity.setProgressMax(contentLength);
}
});
InputStream is = urlconn.getInputStream();
// Open the empty db as the output stream
OutputStream os = new FileOutputStream(f);
// transfer bytes from the inputfile to the outputfile
byte[] buffer = new byte[1024 * 1000];
int written = 0;
int length = 0;
while (written < contentLength) {
length = is.read(buffer);
os.write(buffer, 0, length);
written += length;
final int currentprogress = written;
handler.post(new Runnable() {
public void run() {
Log.i(TAG, String.format("progress %d", currentprogress));
updateActivity.setCurrentProgress(currentprogress);
}
});
}
// Close the streams
os.flush();
os.close();
is.close();
Log.i(TAG, "Download complete");
openDatabase();
} catch (Exception e) {
Log.e(TAG, "bad things", e);
}
handler.post(new Runnable() {
public void run() {
updateActivity.refreshState(true);
}
});
}
Also note that I keep a version number in the filename of the db files, and a pointer to the current one in a text file on the server.
It sounds like your app and your db are tightly bound -- that is, the db is useless without the database and the database is useless without the app, so I'd say go ahead and put them both in the same .apk.
That being said, if you expect the db to change very slowly over time, but the app to change quicker, and you don't want your users to have to download the db with each new app revision, then you might want to unbundle them. To make this work, you can do one of two things:
Install them as separate applications, but make sure they share the same userID using the sharedUserId tag in the AndroidManifest.xml file.
Install them as separate applications, and create a ContentProvider for the database. This way other apps could make use of your database as well (if that is useful).
If you are going to store the db on your website then I would recommend that you just make rpc calls to your webserver and get data that way, so the device will never have to deal with a local database. Using a cache manager to avoid multiple lookups will help as well so pages will not have to lookup data each time a page reloads. Also if you need to update the data you do not have to send out a new app every time. Using HttpClient is pretty straight forward, if you need any examples please let me know
Can I do nested transactions in NHibernate, and how do I implement them? I'm using SQL Server 2008, so support is definitely in the DBMS.
I find that if I try something like this:
using (var outerTX = UnitOfWork.Current.BeginTransaction())
{
using (var nestedTX = UnitOfWork.Current.BeginTransaction())
{
... do stuff
nestedTX.Commit();
}
outerTX.Commit();
}
then by the time it comes to outerTX.Commit() the transaction has become inactive, and results in a ObjectDisposedException on the session AdoTransaction.
Are we therefore supposed to create nested NHibernate sessions instead? Or is there some other class we should use to wrap around the transactions (I've heard of TransactionScope, but I'm not sure what that is)?
I'm now using Ayende's UnitOfWork implementation (thanks Sneal).
Forgive any naivety in this question, I'm still new to NHibernate.
Thanks!
EDIT: I've discovered that you can use TransactionScope, such as:
using (var transactionScope = new TransactionScope())
{
using (var tx = UnitOfWork.Current.BeginTransaction())
{
... do stuff
tx.Commit();
}
using (var tx = UnitOfWork.Current.BeginTransaction())
{
... do stuff
tx.Commit();
}
transactionScope.Commit();
}
However I'm not all that excited about this, as it locks us in to using SQL Server, and also I've found that if the database is remote then you have to worry about having MSDTC enabled... one more component to go wrong. Nested transactions are so useful and easy to do in SQL that I kind of assumed NHibernate would have some way of emulating the same...
NHibernate sessions don't support nested transactions.
The following test is always true in version 2.1.2:
var session = sessionFactory.Open();
var tx1 = session.BeginTransaction();
var tx2 = session.BeginTransaction();
Assert.AreEqual(tx1, tx2);
You need to wrap it in a TransactionScope to support nested transactions.
MSDTC must be enabled or you will get error:
{"Network access for Distributed Transaction Manager (MSDTC) has been disabled. Please enable DTC for network access in the security configuration for MSDTC using the Component Services Administrative tool."}
As Satish suggested, nested transactions are not supported in NHibernate. I've not come across scenarios where nested transactions were needed, but certainly I've faced problems where I had to ignore creating transactions if other ones were already active in other units of work.
The blog link below provides an example implementation for NHibernate, but should also work for SQL server:
http://rajputyh.blogspot.com/2011/02/nested-transaction-handling-with.html
I've been struggling with this for a while now. Am going to have another crack at it.
I want to implement transactions in individual service containers - because that makes them self-contained - but then be able to nest a bunch of those service methods within a larger transaction and rollback the whole lot if necessary.
Because I'm using Rhino Commons I'm now going to try refactoring using the With.Transaction method. Basically it allows us to write code as if transactions were nested, though in reality there is only one.
For example:
private Project CreateProject(string name)
{
var project = new Project(name);
With.Transaction(delegate
{
UnitOfWork.CurrentSession.Save(project);
});
return project;
}
private Sample CreateSample(Project project, string code)
{
var sample = new Sample(project, code);
With.Transaction(delegate
{
UnitOfWork.CurrentSession.Save(sample);
});
return sample;
}
private void Test_NoNestedTransaction()
{
var project = CreateProject("Project 1");
}
private void TestNestedTransaction()
{
using (var tx = UnitOfWork.Current.BeginTransaction())
{
try
{
var project = CreateProject("Project 6");
var sample = CreateSample(project, "SAMPLE006", true);
}
catch
{
tx.Rollback();
throw;
}
tx.Commit();
}
}
In Test_NoNestedTransaction(), we are creating a project alone, without the context of a larger transaction. In this case, in CreateSample a new transaction will be created and committed, or rolled back if an exception occurs.
In Test_NestedTransaction(), we are creating both a sample and a project. If anything goes wrong, we want both to be rolled back. In reality, the code in CreateSample and CreateProject will run just as if there were no transactions at all; it is entirely the outer transaction that decides whether to rollback or commit, and does so based on whether an exception is thrown. Really that's why I'm using a manually created transaction for the outer transaction; so we I have control over whether to commit or rollback, rather than just defaulting to on-exception-rollback-else-commit.
You could achieve the same thing without Rhino.Commons by putting a whole lot of this sort of thing through your code:
if (!UnitOfWork.Current.IsInActiveTransaction)
{
tx = UnitOfWork.Current.BeginTransaction();
}
_auditRepository.SaveNew(auditEvent);
if (tx != null)
{
tx.Commit();
}
... and so on. But With.Transaction, despite the clunkiness of needing to create anonymous delegates, does that quite conveniently.
An advantage of this approach over using TransactionScopes (apart from the reliance on MSDTC) is that there ought to be just a single flush to the database in the final outer-transaction commit, regardless of how many methods have been called in-between. In other words, we don't need to write uncommitted data to the database as we go, we're always just writing it to the local NHibernate cache.
In short, this solution doesn't offer ultimate control over your transactions, because it doesn't ever use more than one transaction. I guess I can accept that, since nested transactions are by no means universally supported in every DBMS anyway. But now perhaps I can at least write code without worrying about whether we're already in a transaction or not.
That implementation doesn't support nesting, if you want nesting use Ayende's UnitOfWork implementation. Another problem with the implementation your are using (at least for web apps) is that it holds onto the ISession instance in a static variable.
I just rewrote our UnitOfWork yesterday for these reasons, it was originally based off of Gabriel's.
We don't use UnitOfWork.Current.BeginTransaction(), we use UnitofWork.TransactionalFlush(), which creates a separate transaction at the very end to flush all the changes at once.
using (var uow = UnitOfWork.Start())
{
var entity = repository.Get(1);
entity.Name = "Sneal";
uow.TransactionalFlush();
}