In CQRS with multiple databases, should I call database functions inside Command or inside repositories? - database

I'm using cqrs pattern with multiples databases (one for query and another for search). Should I put the insert inside Repository
CommunityRepository{
Add(Community community){
Database1.Insert(community);
Database2.Insert(community);
}
}
and then:
CommunityCommands{
Handler(AddCommunityCommand community){
communityRepository.Add(community);
}
or should i put this in Commands, like this:
CommunityCommands{
Handler(AddCommunityCommand community){
db1.Insert(community);
db2.Insert(community);
}
or maybe something like this, using the main repository + database2
CommunityCommands{
Handler(AddCommunityCommand community){
communityRepository.Add(community);
db2.Insert(community);
}

I would do neither of those options as you'd be basically coupling the Command and Query immplementations.
Instead, publish events from the Command side, like OrderPlacedEvent and subscribe to them from the Query side. This not only allows you to separate the implementations of the Command and Query sides, but it will also allow you to implement other side effects of the events without coupling the code from the multiple features (eg. "when an order is placed, send a confirmation email").
You can implement the pub/sub synchronously (in process) or asynchronously (with a messaging system). If you use messaging, note that you'll have to deal with eventual consistency (the read data is slightly behind the write data, but eventually it catches up).

refreshing the Query Models should be handled in an offline operation. You should do something like this:
process your Command logic (whatever it is)
right before your Command handler returns, send an Event to a message bus
then in a background service you can listen to those Events and update the Query side.
Bonus tip: you can use the Outbox pattern to get more reliability. The idea is to store the Event messages in a table on your Write DB in the same transaction as your previous write operation, instead of sending them directly. A background service would check for "pending" messages and dispatch them. The rest is unchanged.

Related

Using Broadcast State To Force Window Closure Using Fake Messages

Description:
Currently I am working on using Flink with an IOT setup. Essentially, devices are sending data such as (device_id, device_type, event_timestamp, etc) and I don't have any control over when the messages get sent. I then key the steam by device_id and device_type to preform aggregations. I would like to use event-time given that is ensures the timers which are set trigger in a deterministic nature given a failure. However, given that this isn't always a high throughput stream a window could be opened for a 10 minute aggregation period, but not have its next point come until approximately 40 minutes later. Although the calculation would aggregation would eventually be completed it would output my desired result extremely late.
So my work around for this is to create an additional external source that does nothing other than pump fake messages. By having these fake messages being pumped out in alignment with my 10 minute aggregation period, even if a device hadn't sent any data, the event time windows would have something to force the windows closed. The critical part here is to make it possible that all parallel instances / operators have access to this fake message because I need to close all the windows with this single fake message. I was thinking that Broadcast state might be the most appropriate way to accomplish this goal given: "Broadcast state is replicated across all parallel instances of a function, and might typically be used where you have two streams, a regular data stream alongside a control stream that serves rules, patterns, or other configuration messages." Quote Source
Questions:
Is broadcast state the best method for ensuring all parallel instances (e.g. windows) receive my fake messages?
Once the operators have access to this fake message via the broadcast state can this fake message then be used to advance the event time watermark?
You can make this work with broadcast state, along the lines you propose, but I'm not convinced it's the best solution.
In an ideal world I'd suggest you arrange for the devices to send occasional keepalive messages, but assuming that's not possible, I think a custom Trigger would work well here. You can extend the EventTimeTrigger so that in addition to the event time timer it creates via
ctx.registerEventTimeTimer(window.maxTimestamp());
you also create a processing time timer, as a fallback, and you FIRE the window if the window still exists when that processing time timer fires.
I'm recommending this approach because it's simpler and more directly addresses the specific need. With the broadcast state approach you'll have to introduce a source for these messages, add a broadcast state descriptor and stream, add special fake watermarks for the non-broadcast stream (set to Watermark.MAX_WATERMARK), connect the broadcast and non-broadcast streams and implement a BroadcastProcessFunction (that probably doesn't really do anything), etc. It's a lot of moving parts spread across several different operators.

Is there a way to broadcast a message into all task managers

i wonder if there is a way in flink to broadcast an event (or something like that) if specific event read from the source into all the task managers.
To be more specific I am aggregating state data with a map state and if some events are read from the source I want that all task managers perform a specific action
Is it possible?
Yes, this is possible. The broadcast state pattern is meant for exactly this sort of use case.
As David noted, using a broadcast stream is the right way to send data to all (parallel) sub-tasks. As for only broadcasting some data, take a look at side outputs as a way to do special processing for a sub-set of your data. So you could have a ProcessFunction that passes through all data un-modified, and if an incoming event is one that wants to be broadcast, then you also emit it as a side output.

Coalescing Flux Actions

This is a detailed Flux Architecture question.
Say I have a resource that gets asynchronously prepared. Let's take User Data as an example.
There are multiple different ways to get this user data - in our example it may be that it requires a few different subsequent queries to generate from the server or is stored locally as a whole.
Case 1:
User data needs sequential steps. Fire USER_DATA_1_SUCESS, USER_DATA_2_SUCCESS. Other stores listen for USER_DATA_2_SUCCESS.
Case 2:
User Data is locally available as a whole. Fire a USER_DATA_READY action.
I'm trying to figure out how to go from a linear state completion (USER_DATA_2_SUCESS) to a resource ready event (USER_DATA_READY) in the stores. I can't call USER_DATA_READY directly from the stores - I get a can't call action in the middle of dispatch error. At the same time I want granularity - I want to control the different stages of putting the data together.
I'd like to have one way to condense these calls with good design. The option I can think of is:
Add a convenience 'Ready' function in a client class that is visible to the store. Call it with a tiny timeout in the stores callback for USER_DATA_2_SUCCESS.
Can anyone suggest a better flow?

Why should I use Actions in Flux?

An application that I develop was initially built with Flux.
However, over time the application became harder to maintain. There was a very large number of actions. And usually one action is only listened to in one place (store).
Actions make it possible to not write all event handler code in one place. So instead of this:
store.handleMyAction('ha')
another.handleMyAction('ha')
yetAnotherStore.handleMyAction('ha')
I can write:
actions.myAction('ha')
But I never use actions that way. I am almost sure, that this isn't an issue of my application.
Every time I call an action, I could have just called store.onSmthHappen instead of action.smthHappen.
Of course there are exceptions, when one action is processed in several places. But when that happens it feels like something went wrong.
How about if instead of calling actions I call methods directly from the store? Will my application not be so flexible? No! Occurs just rename (with rare exceptions). But at what cost! It becomes much harder to understand what is happening in the application with all these actions. Each time, when tracking the processing of complex action, I have to find in stores where they are processed. Then in these Stores I should find the logic that calls another action. Etcetera.
Now I come to my solution:
There are controllers that calls methods from stores directly. All logic to how handle action is in the Store. Also Stores calls to WebAPI (as usually one store relating to one WebAPI). If the event should be processed in several Stores (usually sequentially), then the controller handles this by orchestrating promises returned from stores. Some of sequentials (common used) in private methods of itself. And method of controllers can use them as simple part of handling. So I will never be duplicating code.
Controller methods do not return anything (one-way flow).
In fact the controller does not contain the logic of how to process the data. It's only points where, and in what sequence.
You can see almost the complete picture of the data processing in the Store. There is no logic in stores about how to interact with another stores (with flux it's like a many-to-many relation but just through actions). Now the store is a highly cohesive module that is responsible only for the logic of domain model (collection).
The main (in my opinion) advantages of flux are still here.
As a result, there are Stores, which are the only true source of the data. Components can subscribe to the Stores. And the components calls the same methods as before, but instead of actions uses controller. Interaction with React did not change at all.
Also, event processing becomes much obvious. Now I can just look at the handler in the controller and all becomes clear, and it's much easier to debug.
The question is:
Why were actions created in flux? And what are their advantages that I have missed?
Actions where implemented to capture a certain interaction on the view or from the server which can then be dispatched to as many different stores as you like. The developers explained this with the example of the facebookchat.
There is a messageStore and a threadstore. When the action eg. messagePost was emitted it got dispatched into both stores doing different work to update their attributes. Threadstore increased the number of unread messages and messageStore added the new message to its messagearray.
So its basicly channeling one action to perform datachanges in more than one store.
I had the same questions and thought process as you, and now I started using Flummox which makes it cleaner to have the Flux architecture.
I define my Actions in the same file where I define my Store, and that's close enough. I can still subscribe to the dispatcher to log events so I can see all actions being called, and I have the option to create multi-store Actions if needed.
It comes with a nice FluxComponent that lets you wrap all store-related code in a single place so its children are stateless components that get updated on store changes, like
<FluxComponent connectToStores={['storeA', 'storeB']}>
<InnerComponent />
</FluxComponent>
Keeping the Flux architecture (it uses Facebook's Flux behind the scenes) will hopefully make it easy to use other technologies like GraphQL.

Can I save changes to objects to another TR besides those they are locked?

When I try to switch to edit mode for a Report source, a popup comes up telling me
"A new task will be created for the following request of user XXX".
A transport request is also being suggested.
I don't want to save my changes in this request however, but in another existing one. I am not aware of any versioning systems being implemented in my system, and don't know how to check that.
Is what i'm trying to achieve possible? And if so, how?
No, this is not possible. There are very good reasons for this being an exclusive lock -- reasons that you should know about before you attempt to change anything. Briefly speaking
The CTS only notes that an object was touched, not what change was made.
When the transport is released, the entire object in its current state is exported - there is no delta/diff logic involved.
Therefore you can't separately transport changes to the same development object. Furthermore, if you serialize this manually, the second transport will always comprise the changes of the first one.
Things get slightly more complicated with partial objects - you can have LIMU METH objects (methods of a class) in different transports, but as soon as you try to lock the R3TR CLAS main class, you'll have to resolve that.

Resources