Generating multiple files based on DB query in a nice way - apache-camel

I have a following question.
I have to generate many files based on sql query.
Let's say for example, I have get from database a list of orders made today and genarate file for each order and later store each file on ftp.
Ideally I would like to get follewing.
Not quite sure how to get it.
from(MyBean).to(Ftp)
The problem and main question is how to generate multiple messages by custom bean (for example).
I am not sure if splitter EIP is ok in this case
because in my case I have not just one message to split, but I just have to generate and send many messages.
http://camel.apache.org/splitter.html
I hope, someone meet this problem before.
If the task will be to generate just one file - everything is quite simple - you need just fill Exchange.OutMessage (or something like this). But what about multiple files - I really can't get, how to manage this situation.
P.S. Sorry if this question is stupid.
I am novice in Camel (working with it just for coupe weeks).
It's a great tool.
Actually, that's why I want to use in in the best way.
Thanks a lot.

from("bean:myBean").split(body()).to("bean:ftpBean")
just make sure myBean returns a List of messages and they will be split and sent individually to your FTP endpoint...

Related

Apache Camel FTP - Get List of Files

Hy,
I'm currently facing the challenge, that I need to get a List of all children in an FTP folder.
Unfortunatelly I'm not searching for a specific file, or I would just use a filter, but I need the actual list of files, with all their meta-data.
I do this because I then compare that list to the children of a local folder to create a kind of report.
I tried using pollEnrich() with the consumer to get the files individually and then aggregate them, but the issue is, that I don't have a completion condition as I don't know the number of files, that will arrive.
I thought about using the FTPClient directly, but it does not make much sense to create a client myself, when camel already has such good handling of the FTPClient. I would basically copy paste the camel component code, which is nonsense.
Any ideas?
Edit:
I just realized, that the ftp consumer also skips folders, which makes sense for its original use case (loading files), but makes this even harder, as I also need the folder.
Best regards
Chris

How is ElasticSearch supposed to work in CakePHP 3?

I've been trying my very best not to ask any nosy question here in stackoverflow, but it has been almost one week since I got stuck in this problem and I couldn't find any solution.
I already have my working website built with CakePHP 3.2. What the website basically does is scrape Twitter for tweets containing a given search term, check if it's already in my database, and store it if it doesn't yet exist. Twitter's JSON response has this "tweet_id" property, and I've been using that value to check for whether I should ignore or append a specific tweet to my DB. While this might be okay while my database is small, I suspect it's going to slow things down considerably when my tables grow bigger. Thus my need for ElasticSearch.
My ElasticSearch server is running on my Arch Linux install, and I've configured my app to point to the said server. Also, I have my "Type" object named the same way as my "Tweets" table (I followed the documentation until the overview part http://book.cakephp.org/3.0/en/elasticsearch.html). This craps out an "Unknown method "alias" error, and following Google searches led me to creating an alternate pagination class since that was what some found to be the cause of the error (https://github.com/lorenzo/audit-stash/issues/4), which still doesn't fix things.
I'm not sure if I got this right. I installed the ElasticSearch plugin with the assumption that all I have to do is name the Types the same name as my tables, since to me the documentation "implies" that this should be done on top of the Blog Tutorial they did to "improve query performance".
TLDR, how is this supposed to work? Is my above assumption right? Do I name the Types differently and index everything myself? I'm not sure if there's just too much automagic, or I'm just poor at these sort of things. And yes, I'm new to frameworks (but not PHP, among other languages)
Thanks in advance!

How do websites use information on a database to create pages?

Sorry about the broad question. I'm just curious if someone could point me in the right direction.
Say there's a database of contact information, and there's a site where you can input a persons name and it brings you to a page with all of their information on that database. How does this happen exactly? The server would have to dynamically create this page, but does it have a generic format that it just fills with the information? And how does this happen?
Like you said, this is a an extremely broad question. It could be either way. The server could generate the entire contents dinamically, or it could be "filling the blanks" into a preformatted layout.
Google some PHP basic tutorials. That should give you a good idea about how this "dynamism" works. Sorry but your question is too broad to ellaborate more.
The server would dynamically create the page using PHP and SQL. There is a quick tutorial at http://www.mysqltutorial.org/php-querying-data-from-mysql-table/ that shows how it would be setup.
If I understood your question right, you are asking things like how this page was created, for example, in which case it can be as simple as a basic PHP and SQL combination. You can check an example on the w3sschools website:
Try it yourself example
There would be special place holders for the data and a query will extract the data and put it into the given place holders, please note, you can also use loops to add things like tables, fetch through multiple rows and so on.

What is the best solution to manage data from csv file and do some logic and action

Intro
I have learn basic programming in the past at school (vb) so i understand the logic behind an application and the way it think.
I started this week to learn python because ... I would like to be able to build what i need without having to smash my head in the keyboard.
In the mean time i would like to resolve my problem and i'm asking to all of you in case someone have seen something somewhere.
I have try to search but not sure how to ask so respond was not relevant.
My Question
I know i can do this in excel but i'm looking for another way if possible.
Does someone knows if there existe an application online or on a mac that give me the ability to:
1- Import data from a csv file and add it in a database. So every day i will log 100 line of data.
2- After that i would like to manage condition related on data store in store in file 2 and 3.
3- That will generated a file 4 where i can see the only important case i have to take care for the day.
I imagine a little bit Access from microsoft but i don't remember if i was able to add parameter on the data i will visualise or condition.
What i'm trying to achieve
I have to process a lot of data manually right now and i'm trying to find a way to take out automatically the recurring problem from a list that i receive every day And there is no other way for me than build my own validation process.
Thank for your help. I'm new but i will try my best to bring something is this community (Python studies lv1, VB -15ans) . For now i'm building my startup And my expertise is more in technical skill on security products and technical support.
In case i'm not alone that would like to be able to do some basic thing like i was needed without having to program from scratch a database program i found those 2 solution yet:
https://www.zoho.com/creator/
and
http://www.ifreetools.com/

Processing.js - Listing files from sketch directory

I'm new here, so hello everyone!
I wrote a few things in Processing language and now I need to switch to Processing.js. I need to write an app that first scans the sketch folder to prepare a list of provided files. And what was straightforward in Processing is not in PJS.
I'm currently searching the web but I only found solutions for classic Processing. I know that JavaScript has restrictions and in general can't access the user-side files, but is there any way to list the sketch-itself files?
The only way that comes to my mind is to list them on server side via PHP and generate the .pde file dynamically depending on the sketch folder. But the catch is to not use any other language.
Thanks in advance for help!
Processing.js running on a website can only get information that URLs can provide it, and since there are no "dir listings" on the web, it can't grab dir listing content for a URL for you work with. However, depending on what you really want to do, there might be a way to make it work without resorting to PHP.
Assuming you have your Pjs page running on www.example.org/index.html, and you want to list content for www.example.org/sketch/, one option is to simply have a file www.example.org/sketch/list.txt containing all the filenames that the sketch can access, and simply grab that with a
String[] fileNames = loadStrings("./sketch/list.txt")
instruction.
If you can give an example of what you mean with "I need to write an app that first scans the sketch folder to prepare a list of provided files", a more specific solution is probably possible (i.e., what are the files, what does the user need them for, etc)

Resources