I am trying to create a Workflow logic for the multiple-approval process using logic apps.
My requirement is we have 3 approvers 2 are primary 1 is secondary approver.
If one of the primary approvers is not approved in 15 minutes it will go to a secondary approver.
If one of the primary approvers is rejected it will stop the process.
If both primary approvers have approved the file will move to another location.
Could you please someone help me with this.
Related
Here is the context for the problem I am trying to solve.
There are computers A and B, as well as a server S. Server S implements some backend which handles incoming requests in a RESTful manner.
The backend S has a shelf. The goal of users A and B is to make S create and place numbered boxes on that shelf. A unique constraint is that no two boxes can have the same number. Once a box is created, S should return that box (JSON, or xml...) back to A and B with its allocated number.
The problem boils down to concurrency, as A and B's POST ("create-numbered-box") transactions may arrive at the exact same time at the database - hence get cancelled (?). I remind, there is a unique constraint - no two boxes are allowed to have a same number.
What are possible ways to solve this problem? I wouldn't like to lock the database, so I am looking for alternatives of that. You are allowed to imagine that between the database and the backend layer calling the database we may have an extra layer of abstraction, e.g. a microservice, messaging queue... whatever or nothing at all - a direct backend - db exec. query call. If you think a postgres database is not a good choice to say a graph one, or document one, key-value one - feel free to substitute it.
The goal is in the end given concurrent writes users A and B to get responses to their create (POST) requests and each of them have a box on that shared shelf with a unique number, with no "Oops, something went wrong. Please retry" type of server response.
I described a simple world with users A and B but that can in theory go up to 10 000 users writing, not just 2.
As a secondary question, I'd like to ask, is there a way to test conflicting concurrent transactions in postgres?
I will go first.
My idea is, let A and B send requests and fail. Once they fail, have retries with random timeouts in some interval. Let's say up to 3 retries. This way for A and B I will try to separate the requested writes to the db and this would allow for some degree of successful resolution of the scenario. However, I don't think this is a clean solution and I am looking for alternatives you can think of. Just, please keep in mind the constraints and freedoms I mentioned above.
Databases such as Posgres include capabilities to have a unique number generated by the database (see PostgreSQL - SERIAL - Generate IDs (Identity, Auto-increment)). So the logic for your backend service S could be:
lookup if user has a record in the database already
return the id if it does
otherwise, create a record and return the newly allocated id
To avoid creating multiple boxes for the same user you need to serialize the lookup/create logic based on user id. Approaches to that vary from merely handling one request at a time in your service S to, for example, having Kafka topics that partition requests to different instances of service S based on user ids -- all depends on the scale.
I am implementing a license key system on Google AppEngine. Keys are generated ahead of time and emailed to users. Then they log into the system and enter the key to activate a product.
I could have potentially several hundred people submitting their keys for validation at the same time. I need the transactions to be strongly consistent so that the same license key cannot be used more than once.
Option 1: Use the datastore
To use the datastore, I need it to be strongly consistent, so I will use an EntityGroup for the license keys. However, there is a limit of 1 write / second to an entity group. Appengine requests must complete within 60 seconds, so this would mean either notifying users offline when their key was activated, or having them poll in a loop until their key was accepted.
Option 2: Use Google Cloud SQL
Even the smallest tier of Google Cloud SQL can handle 250 concurrent connections. I don't expect these queries to take very long. This seems like it would be a lot faster and would handle hundreds or thousands of simultaneous license key requests without any issues.
The downside to Google Cloud SQL is that it is limited in size to 500GB per instance. If I run out of space, I'll have to create a new database instance and then query both for the submitted license key. I think it will be a long time before I use up that 500GB and it looks like you can even increase the size by contacting Google.
Seems like Option2 is the way to go - but I'm wondering what others think. Do you find Entity Group performance for transactions acceptable?
Option 2 seems more feasible, neat and clean in your case but you have to take care of db connections by yourself and its a hassle with increasing load if connection pooling is not properly used.
Datastore can also be used in license key system by defining multiple EntityGroups with dummy ancestors based on few leading or trailing digits of key to deal with 1 write / second to an entity group. In this way you can also easily determine EntityGroup of a generated or provided license key.
For example 4321 G42T 531P 8922 is license key so 4321 can be used as EntityGroup and all keys starting with 4321 will be part of this EntityGroup. This is sort of sharding like mechanism to avoid the potential of simultaneous writes to single entity group.
If you need to perform queries on some columns other than license key then a separate mapping table can be maintained without an EntityGroup.
You can mixed them , Google Cloud SQL is only have Keys and Email , with 500G i belived you can store key for all of people in the planet .
In other hand you can request google to increase data size limit .
I will go with Option 1 datastore, it's much faster and scalable.
And I don't know why you need to create EntityGroup, you could make the "license key" itself as the Key, so each Entity is in it's own EntityGroup... only this will make things scalable.
Apologies if a similar question has been addressed elsewhere but I'm struggling to find the obvious answer to my issue....
I have rolled out a split end database (.accdb created in Access 2013) to 6 members of my team by providing each with a copy of the front end which links to a back end on a shared network drive. Four of the users are opening the db through Access 2013, one through Access Runtime 2013 and one through Runtime 2010 (32 bit).
The primary job of the database is to allow users to allocate and manage tasks for a set of campaigns. The db centres around a task table which is updated via a bound form. When new task records are created, usually via a control from a parent 'campaign' form, some fields are pre-populated.
The (frequent) bug seems to occur when a two users are editing different task records via the task form at the same time. Occasionally, one of the task records becomes corrupted (hashed out or Chinese characters!) but more often one of the tasks becomes duplicated in place of the other. This then leads to duplicate task IDs and the loss of the primary key on this field.
I have tried setting record locking to both no locks (optimistic locking) - on users' access clients (except the Runtime versions where I can't see there is an option to do this) and on the task form itself - and edit record (pessimistic locking) using the setting in the task form properties.
I am having trouble diagnosing whether the error lies with locking and/or the point at which a record is saved (currently just on form close) or whether there is a bigger weakness in the set up. Does anyone have any ideas as to why this duplication and sometimes corruption might occur? Thanks
I'm using Delphi XE4 Architect (Delphi Xe3 is ok as well)
I need to find a smart solution to the following problem
and I would like to use one of these frameworks: kbmMW or RemOjects SDK / DataAbstract or RealThinClient
Currently I have an application using a very simple MSSQL database on a site A that is used by users of a site B through the remote desktop.
The application sometimes needs to show some pictures and also view some PDF, but it is mostly text data entry.
There is no particular reason for me to use MSSQL,
but it is a database that I found already active and populated and I have not built it myself.
And now, it would be complicated to change it.
(Database is not important, not using specific features nor stored procedures nor triggers)
Users of the Site B are connected to the A site via a network connection very slow
and occasionally the connection is not available for a few hours and up to one day (this is the major problem).
The situation of the connection, unfortunately, can not be improved for various reasons.
The database is quite simple has many tables that hardly ever change,
about ten instead undergo daily updates and potentially they may be subject to competing changes.
Mainly the records of these tables contain data that are locked in update
from a single user to edit some fields and then he saves releasing the lock.
I would like to get something very different to optimize performance.
Users of the A site have higher priority, they are more important, because the A site is the headquarters.
I would like to have a copy of the database at Site A to Site B,
so that users of site B can work in local, much faster without using the remote desktop connecting to the site A.
The RDP protocol is not very optimized and in any case if the connection is absent, users could not work.
Synchronize and update databases lock records between the two databases may not be a big problem.
Basically when a user of the Site B acquires edit a record in the database B,
obviously a user of the site A should not be able to modify the same record on the database of the site A.
This should also work in the opposite direction of course.
My big problem is figuring out how handling to the best the situation that occurs
when the connection between B and A is not available for some hours. (And transaction/events is increasing on site B).
Events on Site A have generally priority (on collision) on events on Site B.
Users of the Site B must be able to continue working.
When the connection becomes active, the changes should be sent to the database at Site A.
Obviously this can result in conflicts, but the changes made on the record
possibly by users B can be discarded or it can be done under the supervision of a selective merge
and approval record by record user of the site B.
Well, I hope the scenario is almost explained clearly.
Additional infos:
DB schema is very simple, only tables, no triggers, stored procedure. So I can build one as example but imagine 10 tables that can be updated in concurrency.
DB is used by a desktop app of sales departement, so it contains most secret data.
Remote connection is typically max 512Kbit, but the main problem here is that the connection sometimes may be not active
and user on remote site must work anyway. THis is the main focus.
Total data of daily updates could be at max 10 Mb, compressed, only for DB connections. There are some other data synchronized
on the same connection but they are not part of this job.
I don't want to use specific MSSQL tools or services (replications or so on), because DB could change in future.
Thanks
We do almost exactly this using a Delphi client app, a kbmMW based Delphi server app, MSSQL database (though it used to work quite happily on on DBISAM database too).
We have some tables that only the head office site users are allowed to modify. The smaller tables are transferred in their entirety each time there is a "merge". The larger tables and the transaction type tables all have a date added and/or a date modified field and only those records that have been changed or added in the last 3 weeks or so (configurable) are transferred. This means sites can still update to the latest data even if they have been disconnected for quite some time - we used to have clients in remote places on dubious dial up lines!
We only run the merge routines once or twice a day but it would work equally well on an hourly basis or other time schedule.
At given times of day each site (including head office) "export" their changed/new records to files (eg client dataset tables or similar). These are then zipped up by the application and placed in an "outgoing" folder. The zip file is named based on the location id, date, time etc. The files are transferred by some external means eg via FTP / file share / email etc etc. Each branch office sends/transfers its data files to head office and head office transfers its data to each branch. The files are transferred by whatever means to an "incoming" folder.
On a regular basis (eg hourly) each location does a check on the incoming folder to see if there is anything new for it to import. If so it adds all the new records, branch locations overwrite the head-office data tables with the new ones and edited records are merged in "somehow". This is the tricky bit. The easiest policy is "head office wins" so all edits are accepted unless there is a conflict in which case the head office version wins. Alternatively you could use "last edited wins" - but then you need to make sure clocks are in sync across locations. The other option is to add conflicting records to some form of "suspense" status and let an end user decide at some point in the future. We do this on one data set. Whichever conflict method you choose you need to record each decision in some form of log table and prompt an administrative level user to check occasionally.
When the head office imports data or when data is added at the head office then a field is set to indicate the data is part of the master data. When branches add data this field is empty to indicate it has yet to reach the master set. This helps when branches export their data as they can include all data that doesn't have this field set.
We have found that you can't run the merge interactively as you'll end up never getting any work done and you won't be able to run the merge at night etc. It needs to be fully automated with the ability for an admin user to make adjustments at some point after the fact.
We've been running this approach for several years now on multi-site operations and once it settled down it has worked pretty much flawlessly. With 2 export/import schedules per day we have found the branch offices run perfectly well and are only ever missing less than a days worth of transactions. Works well in our scenario where we don't often have conflicts. Exported data is in the region of 5-10MB which zips up plenty small enough.
Primary keys are vital! We use a GUID and it hasn't let us down yet.
The choice of database server and n-tier framework are, actually, irrelevant. It's the process that matters here.
Basically when a user of the Site B acquires edit a record in the database B, obviously a user of the site A should not be able to modify the same record on the database of the site A. This should also work in the opposite direction of course.
I can't see how you're ever going to make this bit work reliably if both sites have their own copy of the database and you're allowing for dropped/non-existent inter-site connections on occasion.
I have a scenario where multiple users will be doing import processes, but all of them will be working for different clients.
I have one core table which gets the most hits whenever import processes run. I have 2 options now
To have one core table and do the sequential imports by making queue for the import processes.
To have 300 core table, one for each client, it will allow the users to work on the import processes simultaneously without waiting for one another.
Can anyone suggest which one is better and why?
I am giving my requirements in more detailed this time. Can you all once again have a look at and provide your comments after going through the requirements.
The query is regarding data modeling for core functionality of my application.
I have a scenario where multiple users will be doing import processes, but all of them will be working for different clients. Also, at the same time client's data could be shown to the user and can be modified/inserted too, while the import process for the same or different client is in process.
I have 2 core tables which get the most hits whenever import processes run.
I have 2 options now
1. To have 2 core tables and do the sequential imports by making queue for the import processes.
Table 1
ID
ClientID
SourceID
Count
AnotherCol1
AnotherCol2
AnotherCol3
Table 2
ID
ClientID
OrderID
Count
AnotherCol4
AnotherCol5
AnotherCol6
To have 1000 core table, 2 for each client (I may have maximum 500 clients), it will allow the users to work on the import processes simultaneously without waiting for one another.
More information about the import process:
1. These table is not going to be used in any Reporting.
2. Each import process will insert 20k-30k records (7 columns) in these each table. And there will be around 40-50 such imports in a day.
3. While the import process is going on, data could be retrieved from these tables by some other user and INSERT OR UPDATED too.
4. These are going to be one of the most usable tables in the application.
5. BULK INSERT will be used for insertion.
6. Clustered index is on the Primary Key which is an Identity column.
7. We are considering the table partitioning too.
Can you please suggest which option is better and why?
Also, if you suggest to go with option 2, then would it not be a performance hit to create so many tables in the database? Should we create a separate database for these 1000 tables in this case?
I am giving my requirements in more detailed this time. Can you all once again have a look at and provide your comments after going through the requirements.
The query is regarding data modeling for core functionality of my application.
I have a scenario where multiple users will be doing import processes, but all of them will be working for different clients. Also, at the same time client's data could be shown to the user and can be modified/inserted too, while the import process for the same or different client is in process.
I have 2 core tables which get the most hits whenever import processes run.
I have 2 options now
1. To have 2 core tables and do the sequential imports by making queue for the import processes.
Table 1
ID
ClientID
SourceID
Count
AnotherCol1
AnotherCol2
AnotherCol3
Table 2
ID
ClientID
OrderID
Count
AnotherCol4
AnotherCol5
AnotherCol6
To have 1000 core table, 2 for each client (I may have maximum 500 clients), it will allow the users to work on the import processes simultaneously without waiting for one another.
More information about the import process:
1. These table is not going to be used in any Reporting.
2. Each import process will insert 20k-30k records (7 columns) in these each table. And there will be around 40-50 such imports in a day.
3. While the import process is going on, data could be retrieved from these tables by some other user and INSERT OR UPDATED too.
4. These are going to be one of the most usable tables in the application.
5. BULK INSERT will be used for insertion.
6. Clustered index is on the Primary Key which is an Identity column.
7. We are considering the table partitioning too.
Can you please suggest which option is better and why?
Also, if you suggest to go with option 2, then would it not be a performance hit to create so many tables in the database? Should we create a separate database for these 1000 tables in this case?
It's not really a question with a definitive answer as each has it's own benefits and drawbacks.
Scenario 1: Central core table
Pros: Central table, easy global modifications
Cons: Slower import, more difficult client-level modifications
Scenario 2: 300 core tables
Pros: Faster imports, easy client customization
Cons: More difficult to deploy changes against all 300 core tables, reporting which needs to touch all tables will be more complicated and probably slower as well
In the end the answer is whatever really works for you
Another option is a third scenario where you have a single table, but you still can do the imports in parallel by having a batch identifier in the table which stops people stepping on each other.
The main problem with having multiple people in the same table is that you cannot do things like TRUNCATE.
For me the decision would be related to about where the data eventually goes. Is this just a staging table for convenience because there is going to be some SQL for transform or lookup run against it after load? Would it be possible to make such tables in a separate database or schema and with unique names which would make it easy for them to be cleaned up without interfering with or bloating the transaction log in your primary database> Do you need to insert in bulk, then apply indexes and eventually drop the table? Is such a table even necessary if you are using SSIS to load the data, you can often do a lot of work in the pipeline without needing a staging table?
All these things would play into my decision making process on the architecture.