Following on from this (apologies, had a different user): Kafka Key access on Ingress of a Python Flink Stateful function
Our use case is that we make use of the Kafka headers as a means of tracing and lineage as well as required metadata. Looking at this:
https://github.com/apache/flink-statefun/blob/master/statefun-flink/statefun-flink-io-bundle/src/main/java/org/apache/flink/statefun/flink/io/kafka/binders/ingress/v1/RoutableKafkaIngressDeserializer.java#L45-L61 It looks like using the standard deserializer, the headers are dropped.
Effectively what I'd want, is a way to inject my own deserializer that would return a message containing this and any other metadata from the record. I'd want to add something like the UniversalKafkaIngress so that I could configure it using a remote module.
Looking at the code, I can see that I could register a new ExtensionModule, and replace the deserializer (and create a custom kind). Is this recommended? If so - are there any docs on this (if not, how could I configure statefun to pick this up)?
Or, is there another preferred method?
Thanks again...
Ah - found out where I was going wrong.
You can load a ExtensionModule using the standard module SPI process - and therefore register it as a new 'universal' ingress, so that it can be loaded remotely. I had a typo - which is why I battled.
There are a few gotchas - and I'll post a gist a little later to show how it can be done.
Related
I would like to know if this scenario would be possible in any programming language combined with any database technology.
I would like to automatically save received pdf files that are attached in emails into a database. Is this possible? Is there any library or framework available to do so?
Yes, I would recommend using Google Apps Script for this. The approach you should follow is to use the GmailApp class (Documentation here) to get the messages you need, you can use methods like getInboxThreads() (Documentation), to retrieve the messages.
After you've found the message and retrieved the attachment (which you can do withgetAttachments() (Documentation)), you can use the JDBC Service to connect with external databases. The specifics here depend a lot on what database you want to connect with, but the documentation will lead you in the right direction.
Does anyone had accomplished this?.
The big picture would be to develop the entire asterisk GUI from filemaker, but right now I'm asking you help to connect both.
Asterisk controls our entire Call Center. I would like the info from incoming calls and queues to be written in a FileMaker database.
Disclaimer: I don't know the first thing about FileMaker. But, if it's like any other programming language (which from what I know, I'm not sure that's true) then let's look at the options on how we'd accomplish this generically with other programming languages...
If you just want the results of your call, the CDRs (call detail records), you can configure Asterisk to output custom CDRs in cdr_custom.conf (check it out if you'd generated the sample configurations)
Here's an example cdr_custom.conf:
[mappings]
Simple.csv => ${CSV_QUOTE(${EPOCH})},${CSV_QUOTE(${CDR(src)})},${CSV_QUOTE(${CDR(dst)})}
It will drop a file typically in /var/log/asterisk/ if you haven't changed it otherwise in your configuration.
Then, either restart asterisk, or more gracefully just reload the cdr module:
asterisk*CLI> cdr show status
asterisk*CLI> module reload cdr_custom.so
Using the resulting file, parse the CSV and format it in a friendly fashion for Filemaker / "your language of choice".
If you're looking for real-time information about calls, it does get more complicated. Probably for just reporting purposes, you can use the Asterisk AMI (Asterisk Manager Interface). (Canonical wiki page linked)
This is a TCP IP application, open a socket to it, and you're good to go. There's also the AJAM interface (Asynchronous Javascript Asterisk Manger). Which you can make HTTP calls to.
Lastly, if you want to do further processing during the routing of the call via the dialplan, you'd want to use AGI (Asterisk Gateway Interface) which is called from the dialplan, and is all over STDIO.
Actually you can create a ODBC connection to the asterisk databases and use filemaker to access the tables direct. It will give you a 'live' connection and save you all the import <-> export fuss. If you google on filemaker odbc you'll get results on setting this up, it works quite easy (not always fast depending on your query but certainly a lot quicker than the manual method)
How do you create a custom database driver to extend CodeIgniter's functionalities to other types of database systems? I'm using iRODS (www.irods.org). I have a version of the site created using MySQL, but I want to be able to change the database backend with minimal changes. Is there an easy way to add this function, like how you can add a custom library in CI? I haven't been able to find any so far.
I'm assuming you mean how do you create a custom Active Record driver for codeigniter? Otherwise I'm probably far off the mark here but:
There is no way I know of to simply extend or override the DB classes it is not a common thing. You can implement your own and patch up your CI config to use the new DB though.
Under system/database/drivers you find all the AR driver source. You would need to reimplement each function in each of the four files (may be able to skip on forge if you don't use it.)
I'd use the MySQL driver as a starting template as you mention you already use that, in which case you'll want to make sure all the features you use are re-implemented.
It sounds like a daunting task if you're not too experienced but I assure you the code is pretty simple.
We have several applications that use log4j for logging. I need to get a log4j parser working so we can combine multiple log files and run automated analysis on them. I'm not looking to reinvent the wheel, so can someone point me to a decent pre-existing parser? I do have the log4j conversion pattern if that helps.
If not, I'll have to roll our own.
I didn't realize that Log4J ships with an XML appender.
Solution was: specify an XML appender in the logging configuration file, include that output XML file as an entity into a well formed XML file, then parse the XML using your favorite technique.
The other methods had the following limitations:
Apache Chainsaw - not automated enough
jdbc - poor performance in a high performance distributed app
You can use OtrosLogViewer with batch processing. You have to:
Define you log format, you can use Log4j pattern layout parser or Log4j XmlLayout
Create java class that implements LogDataParsedListener. Method public void logDataParsed(LogData data, BatchProcessingContext context) will be called on every parsed log event.
Create jar
Run OtrosLogViewer with specifying your log processing jar, LogDataParsedListener implementation and log files.
What you are looking for is called SawMill, or something like it.
Log4j log files aren't really suitable for parsing, they're too complex and unstructured. There are third party tools that can do it, I believe (e.g. Sawmill).
If you need to perform automated, custom analysis of the logs, you should consider logging to a database, and analysing that. JDBC ships with the JdbcAppender which appends all messages to a database of your choice, but it has performance implications, and it's a bit flaky. There are other, similar, alternatives on the interweb, though (like this one).
You -can- use Log4j's Chainsaw V2 to process the various log files and collect them into one table, and either output those events as xml or use Chainsaw's built-in expression-based filtering, searching & colorizing support to slice & dice the logs.
Steps:
- Start Chainsaw V2
- Create a chainsaw configuration file by copying the example configuration file available from the Welcome tab - define one LogFilePatternReceiver 'plugin' entry for each log file that you want to process
- Start Chainsaw with that configuration
- Each log file will end up as a separate tab in the UI
- Pause the chainsaw-log tab and clear the events from that tab
- Create a new tab which aggregates the events from the various tabs by going to the 'view, crate custom expression logpanel' menu item and enter 'level >= DEBUG' in the box. It will create a new tab containing events from all of the tabs with level >= debug (which is why you cleared the chainsaw-log tab).
You can get an overview of the expression syntax used to filter, colorize and search from the tutorial (available from the Help menu).
If you don't want to use Chainsaw, you can do something similar - start a simple app that doesn't log but loads a log4j.xml config file with the 'plugin' entries you defined for the Chainsaw configuration, but also define a FileAppender with an xmllayout - all of the events received by the 'receivers' will be sent to the single appender.
How would one create a REST web service to write a row into a databse table. Use the follwoing scenario:
The table is called Customer - the
data to be inserted into the row would
be the name, addresss, telephone
number, email.
I think its impossible to describe the whole thing end to end in Java or C#, and I would never expect that, but here are the questions I have popping into my head as I prepare for coding:
How the URI would look (eg. if you use this URL - http://www.example.com/)?
What info would go into the HTTP envelope?
Would I use POST when writing to the database in this way?
Do I use a resource to store the posted data from the client? Is this even necessary if the data is being written to a database anyway?
When the data to be writeen into the db is recieved by the server - how do I physically insert it into the database - do I call some method on the server to actually write the data (in Java)? - this doesn't seem to fit with truely REST architecture - shunning RPC calls.
Should I even be bothering writing to a DB - should I be storing my data as a resource?
As you can see I need a few issues clearing in my head. Any help much appreciated.
First of all, I'm not either java nor c# expert and I don't exactly know what means do these languages have to support REST design, but in general:
http://www.example.com/customers - customers is a collection of resources and you want to add a new resource to this collection
It depends on various things - you should probably set the content-type header (according to the data format in which you are sending the representation) and set some authentication headers if you need it.
Yes, you always use POST to create a new entry in a collection of resources.
I don't fully understand this question, to be honest. What do you mean by "inmediately writing data into the database"?
REST is primarily just a style of communication between server and a client. It doesn't say anything about how you should handle the data received by using it. The usual way how modern web approaches (MVC style frameworks) solve it, is by routing every REST action to a method of some class (usually a controller instance) where you handle the received parameters (eg. save them to the database) and generate a response to be sent back.
For a very brief and very clear introduction to REST have a look at this short video.
RESTful Web Services, published by O'Reilly and Associates, seems to fit the bill you're looking for.
As far as doing it in Java, Sun has a page on it.