Following is just requirement:
Read from DB
Create 3 files from data with only small setter addition on if statement
Deliever 3 to same sftp
Coding:
1 Route Timer which to(direct)
1 Route trigger which call same (direct)
1 ROUTE - from(direct) then calls DB and fetches data and assign it to Bean and multicast to 3 transformation routes
3 routes - from(direct:transform) where similar transformation and ofr 3rd route a small different transformation and put to SFTP Route
1 Route to SFTP
This could be done in 3 routes only i.e timer route, trigger route and single route from 3 to last.
Just want to ask if Creating multiple route considered as costly operation where 50 project of similar types are deployed?
From a resource operation, it doesn't appear to make a huge difference. I've setup some scenarios that have many small routes, and some with larger routes. In general, the cost is in the work performed by the processors and such. And smaller routes are often more reusable (compose-able) and easier to test. So that's my preference, given that the performance doesn't seem impacted in any significant way.
Related
I've created a GraphQL Schema in AWS App Sync(1100 lines before autogenerating the resources, queries, mutations and subscriptions, 4800 lines afterwards) to use in react. In my react folder I executed
amplify init
amplify add codegen --apiId myId
then I was asked during execution about the desired nesting depth. My types are very nested so it would be great to have a deeper depth but unfortunately I discovered the enormous increase of file size.
3 levels is 2,5mb
4 levels is 9mb
5 levels is 50mb
from the nesting point I would prefer 5 levels or maybe even more but I want the app to be as performant as possible.
No customer wants to wait until 50mb are loaded
what is your experience about loading large files in graphql? Is it possible to split the large files automatically into smaller ones? At level 5 the files have up to 1m lines of code each.
I use lazy loading in React for my components, it would be great if something similar would be possible here as well.
Do you prefer one large nested query with many levels of depth or several cascaded smaller queries with a low depth?
Regards Christian
Lazy...
... probably you don't need full info about each node (all properties) at once.
You can fetch only id and name how deep you need to get structure/relations. Later you can fetch for additional/required (for rendering) properties.
We are investigating Camel for use in a new system; one of our situations is that a set of steps is started, and some of the steps in the set can take hours or days to execute. We need to execute other steps after such long-elapsed-time steps are finished.
We also need to have the system survive reboot while some of the "routes" with these steps are in progress, i.e., the system might be rebooted, and we need the "routes" in progress to remain in their state and pick up where they left off.
I know we can use a queued messaging system such as JMS, and that messages put into such a queue can be handled to be persisted. What isn't clear to me is how (or whether) that fits into Camel -- would we need to treat the step(s) after each queue as its own route, so that it could, on startup, read from the queue? That's going to split up our processing steps into many more 'routes' than we would have otherwise, but maybe that's the way it's done.
Is/are there Camel construct/constructs which assist with this kind of system? If I know their terms and basic outline, I can probably figure it out, but what I really need is an explanation of what the constructs do.
Camel is not a human workflow / long-lasting tasks system. For that kind look at BPMS systems. Camel is more fitting for real time / near real time integrations.
For long tasks you persist their state in some external system like a message broker or database or BPMS, and then you can use Camel routes to process and move from one state to the next - or where Camel fit in such as integrating with the many different systems you can do OOTB with the 200+ Camel components.
Camel do provide graceful shutdown so you can safely shutdown or reboot Camel. But in the unlikely event of a crash, you may want to look at transactions and idempotency if you are talking about surviving a system crash.
You are referring to asynchronous processing of messages in routes. Camel has a couple of components that you can use to achieve this.
SEDA: In memory not persistent and can only call end points in the same route.
VM: In memory not persistent and can call endpoints in different camel contexts but limited to the same JVM. This component is a extension of SEDA.
JMS: Persistence is configurable on the queue stack. Much more heavy weight but also more fault tolerant than SEDA/JVM.
SEDA/JVM can be used as low overhead replacements for JMS components and in some cases I would use them exclusively. In your case the persistence element is a required so SEDA/JVM is not an option, but to keep things simple the examples will use SEDA as you can get some basics up and running quickly.
The example will assume the following we have a timer that kicks off and then there is two processes it needs to run. See screenshot below:
In this route the message flow is synchronous between the timer and the two process beans.
If we wanted to make these steps asynchronous we would need to break each step into a route of its own. We would then connect these routes using one of the components listed in the beginning. See the screenshot below:
Notice we have three routes each route only has one "processing" step in it. Route Test only has a timer which fires a messages to the SEDA queue called processOne. This message is received on the SEDA queue and sent to the Process_One bean. After this it is the send to the SEDA queue called processTwo, where it is received and passed to the Process_Two bean. All this is done asynchronously.
You can replace the SEDA components with JMS once you get to understand the concepts. I suspect that state tracking is going to be the most complicated part as Camel makes the asynchronous part easy.
I am new to Camel and am looking for patterns or strategies to manage the availability of a target system in a Camel route.
For example, say I want:
- to read input data from an file server
- process the data (data -> targetData)
- to send the target data (TargetData) to a target web site using Rest services (call it TargetSystem)
My question is what is the best strategy if the TargetSytem is down?
I understand that if a route fails it is possible to rollback the overall process. But in case TargetSystem is an external system and can be down for hours, I don't think trying to rollback the process untill the target system is up is a good approach.
Is there any pattern or strategy that fits well with this issue?
Regards
Gilles
This is the pattern I use with a couple of systems.
Persist your TargetData somewhere (Database table, JMS queue, Redis, ...)
Create a route that reads unsent TargetData and sends it to TargetSystem
If transfer is OK, mark TargetData accordingly (eg: set a flag in the table row, remove from Queue, etc...)
Trigger such route periodically from a timer and/or other routes. You can even trigger it with a shell command to manually "force" a resend of unsent data.
You can customize this to your needs and for example add logging where appropriate, keep track in a DB table when each tentative send occurs, how many times it failed, how many retries and so on...
You can now modularize your application in 2 modules: one that receives Data and process it to TargetData and another that manages the transfer of TargetData to TargetSystem.
Module can mean 2 CamelContextes, 2 OSGi bundles, 2 totally separate Java applications.
We are using camel-cxf as consumer (soap) in our project and asked ourself if camel-cxf uses multiple threads to react on requests.
We think it uses multiple threads, right?!
But what does this mean for the rest of the route? Is all multithreaded after "from" or is there a point of synchronization?
And what does this mean for "parallelProcessing" or "threads"?
In our case we use jdbc component later in the route. IS camel-jdbc also using multiple threads?
How to know in general what threading model is used by a given component?
Let's start with your last question:
How to know in general what threading model is used by a given
component?
You are probably asking which component is single-threaded by default and which ones are multi-threaded?
You need to ask yourself which approach makes most sense for a component and read the component's documentation. Normally the flags will tell you what behavior is applied by default. CXF is a component that requires a web server, jetty in this case, for a SOAP (over HTTP) client to be able to call the service. HTTP is a stateless protocol, a web server has to scale to many clients, thus it makes a lot of sense for a web server to be multi-threaded. So yes, two simultanious requests to a CXF endpoint are handled by two separate (jetty) threads. The route starting at the CXF endpoint is executed simultaniously by the jetty threads that received the request.
On the contrary, if you are polling for file system changes, e.g. you want to check if a certain file was created, it makes no sense to apply multiple threads to the task of polling. Thus the file consumer is single threaded. The thread employed by the file consumer to do the polling will also execute your route that processes the file(s) that were picked up during a poll.
If processing the files identified by a poll takes a long time compared to your polling intervall, and you cannot afford to miss a poll, then you need to hand of the processing of the rest of the route to another thread so your polling thread is again free to do, well, polling. Enter the Threads DSL.
Then you have processors like the splitter that create many tasks from a single task. To make the splitter work for everyone it must be assumed that the tasks created by the splitter cannot be performed out of order and/or fully independent of each other. So the safe default is to run the steps wrapped by the split step in the thread that executes the route as a whole. But if you the route author knows that the individual split items can be processed independent of each other, then you can parallelize the processing of the steps wrapped by the split step by setting parallelProcessing="true".
Both the threads DSL and the using parallelProcessing="true" acquire threads from a thread pool. Camel creates a pool for you. But if you want to use multiple pools or a pool with a different configuration, then you can always supply your own.
This is something that's been bothering me.
In a typical web application, request goes to the server and the server does it's 'stuff' and returns the template along with the data, ready to be displayed on the browser.
In Angular however, there is a response to get the template and, in most cases, another one to get the data.
Doesn't this add pressure on the bandwidth – 2 responses sent over the wire as compared to one from a typical web app. Doesn't it also imply a (possibly) higher page-load time?
Thanks,
Arun
The short answer is that it depends. :)
For example, in terms of bandwidth, it depends on how big the 'server-side' view render payload is. If the payload is very big, then separating out the calls might reduce bandwidth. This is because the JSON payload might be lighter (less angle brackets) so the overall bandwidth could reduce. Also, many apps that use ajax are already making more than one call (for partial views etc).
As for performance, you should test it in your application. There is an advantage to calling twice if you are composing services and one service might take longer than another. For example, if you need to layout a product and the service which provides 'number in the warehouse' takes longer than the product details. Then, the application could see a perceived performance gain. In the server-side view render model, you would have to wait for all the service calls to finish before returning a rendered view.
Further, client side view rendering will reduce the CPU burden on the server, because view rendering will be distributed on each client's machine. This could make a very big difference in the number of concurrent users and application can handle.
Hope this helps.