I am implementing a RichParallelSourceFunction which reads files over SFTP. RichParallelSourceFunction inherits cancel() from SourceFunction and close() from RichFunction(). As far as I understand it, both cancel() and close() are invoked before the source is teared down. So in both of them I have to add logic for stopping the endless loop which reads files.
When I set the parallelism of the source to 1 and I run the Flink job from the IDE, Flink runtime invokes stop() right after it invokes start() and the whole job is stopped. I didn't expect this.
When I set the parallelism of the source to 1 and I run the Flink job in a cluster, the job runs as usual.
If I leave the parallelism of the source to the default (in my case 4), the job runs as usual.
Using Flink 1.7.
public class SftpSource<TYPE_OF_RECORD>
extends RichParallelSourceFunction<TYPE_OF_RECORD>
{
private final SftpConnection mConnection;
private boolean mSourceIsRunning;
#Override
public void open(Configuration parameters) throws Exception
{
mConnection.open();
}
#Override
public void close()
{
mSourceIsRunning = false;
}
#Override
public void run(SourceContext<TYPE_OF_RECORD> aContext)
{
while (mSourceIsRunning)
{
synchronized ( aContext.getCheckpointLock() )
{
// use mConnection
// aContext.collect() ...
}
try
{
Thread.sleep(1000);
}
catch (InterruptedException ie)
{
mLogger.warn("Thread error: {}", ie.getMessage() );
}
}
mConnection.close();
}
#Override
public void cancel()
{
mSourceIsRunning = false;
}
}
So I have workarounds and the question is more about the theory. Why is close() invoked if parallelism is 1 and the job is run from the IDE (i.e. from the command line)?
Also, do close() and cancel() do the same in a RichParallelSourceFunction?
Why is close() invoked if parallelism is 1 and the job is run from the
IDE.
close is called after the last call to the main working methods (e.g. map or join). This method can be used for clean up work.
It will be called independent of the number defined in parallelism.
Also, do close() and cancel() do the same in a RichParallelSourceFunction?
They aren't the same thing, take a look at how it's described.
Cancels the source. Most sources will have a while loop inside the run(SourceContext) method. The implementation needs to ensure that the source will break out of that loop after this method is called.
https://ci.apache.org/projects/flink/flink-docs-master/api/java/org/apache/flink/streaming/api/functions/source/SourceFunction.html#cancel--
The following link may help you to understand the task lifecycle:
https://ci.apache.org/projects/flink/flink-docs-stable/internals/task_lifecycle.html#operator-lifecycle-in-a-nutshell
I think javadocs are more than self-explanatory:
Gracefully Stopping Functions
Functions may additionally implement the {#link org.apache.flink.api.common.functions.StoppableFunction} interface. "Stopping" a function, in contrast to "canceling" means a graceful exit that leaves the state and the emitted elements in a consistent state.
-- SourceFunction.cancel
Cancels the source. Most sources will have a while loop inside the run(SourceContext) method. The implementation needs to ensure that the source will break out of that loop after this method is called.
A typical pattern is to have an "volatile boolean isRunning" flag that is set to false in this method. That flag is checked in the loop condition.
When a source is canceled, the executing thread will also be interrupted (via Thread.interrupt()). The interruption happens strictly after this method has been called, so any interruption handler can rely on the fact that this method has completed. It is good practice to make any flags altered by this method "volatile", in order to guarantee the visibility of the effects of this method to any interruption handler.
-- SourceContext.close
This method is called by the system to shut down the context.
Note, you can cancel SourceFunction, but stop SourceContext
I found a bug in my code. Here is the fix
public void open(Configuration parameters) throws Exception
{
mConnection.open();
mSourceIsRunning = true;
}
Now close() is not invoked until I decide to stop the workflow in which case first is invoked cancel() and then close(). I am still wondering how did parallelism affect the behaviour.
Related
I am trying to shutdown a StreamExecutionEnvironment that is started during one of our junit Integration tests. Once all the items in the stream are processed i want to be able to shutdown this execution environment in a deterministic fashion.
Right now when i call StreamExecutionEnvironment.execute method it never returns from that call.
[May be I am too late here, but will answer for those people who are looking for an answer or hint]
Actually what you need to do is gracefully exit from the SourceFunction<T>.
Then the whole StreamExecutionEnvironment will be auto closed. To do that, you may need a special end event to send into your source function.
Write or extend (if you are using pre-defined source function) to check the special incoming event, which will be emitted at the end of your integration tests, and break the loop or unsubscribe from the source. The basic pattern would be shown below.
public class TestSource implements SourceFunction<Event> {
public void run(SourceContext<T> ctx) throws Exception {
while (hasContent()) {
Event event = readNextEvent();
if (isAnEndEvent(event)) {
break;
}
ctx.collect(event);
}
}
}
Depends on your situation, in junit tests, you should send this special event at the end of each/all test cases.
// or #AfterClass
#After
public void doFinish() {
// send the special event from a different thread.
}
Sometimes you might have to do this from a different thread (basically where you generate test events). Not like above.
It is recommended you having a separate source function implementation for your tests because of this matter, so it is easier to modify to accept a special close event. But never in your actual source function which is expecting to go into the production. :)
The javadoc of SourceFunction also explains about a stop() function, which you can see an example from how TwitterSource has been implemented.
Gracefully Stopping Functions
Functions may additionally implement the {#link
org.apache.flink.api.common.functions.StoppableFunction} interface.
"Stopping" a function, in contrast to "canceling" means a graceful
exit that leaves the state and the emitted elements in a consistent
state
.
I'm trying to figure out how to get System.AbortJob() to actually work. My assumption (could be wrong) is that when I pass the current jobId to System.AbortJob(), the job will stop right there and abort. Here is my test that isn't working, as I am seeing the System.debug() showing up in my logs.
Executing from execute anonymous:
queueableTest tst = new queueableTest();
System.enqueueJob(tst);
Queueable class:
public class queueableTest implements Queueable {
public static void execute(QueueableContext Context)
{
ID jobID = Context.getJobId();
System.AbortJob(jobID);
shouldntExecute();
}
public static void shouldntExecute()
{
System.debug('Why is this executing?');
}
}
Any help/feedback greatly appreciated!
The System.abortJob() call will only take effect after its execution context is completed. Since you are calling abortJob from the same context that you want to abort, by the time it takes effect your code has already finished executing which makes the System.abortJob() call irrelevant.
If you want to abort the current job, you need to use return; or System.assert(false, 'Aborting'); In the first case your job will terminate with a status of 'Completed', and in the second case with a 'Failed' status. Throwing an exception would also have the same result.
I am creating a server which consumes commands from numerous sources such as JMS, SNMP, HTTP etc. These are all asynchronous and are working fine. The server maintains a single connection to a single item of legacy hardware which has a request/reply architecture with a custom TCP protocol.
Ideally I would like a single command like this blocking type method
public Response issueCommandToLegacyHardware(Command command)
or this asynchronous type method
public Future<Response> issueCommandToLegacyHardware(Command command)
I am relatively new to Netty and asynchronous programming, basically learning it as I go along. My current thought is that my LegacyHardwareClient class will have public synchronized issueCommandToLegacyHardware(Command command), will make a write to the client channel to the legacy hardware, then take() from a SynchronousQueue<Response> which will block. The ChannelInboundHandler in the pipeline will offer() a Response to the SynchronousQueue>Response> which will allow the take() to unblock and receive the data.
Is this too convoluted? Are there any examples around of synchronous Netty client implementations that I can look at? Are there any best practices for Netty?
I could obviously use just standard Java sockets however the power of Netty for parsing custom protocols along with the ease of maintaniability is far too great to give up.
UPDATE:
Just regarding the implementation, I used an ArrayBlockingQueue<>() and I used put() and remove() rather than offer() and remove(). Because I wanted to ensure that subsequent requests to the legacy hardware were only sent when any active requests had been replied to as the legacy hardware behaviour is not known with certainty otherwise.
The reason offer() and remove() did not work for me was that the offer() command would not pass anything if there was not an actively blocking take() request no the other side. The converse is true that remove() would not return anything unless there was a blocking put() call inserting data.
I couldn't use a put()/remove() since the remove() statement would never be reached since there was no request written to the channel to trigger the event from where the remove() would be called. I couldn't use offer()/take() since the offer() statement would return false since the take() call hadn't been executed yet.
Using the ArrayBlockingQueue<>() with a capacity of 1, it ensured that only one command could be executed at once. Any other commands would block until there was sufficient room to insert, with a capacity of 1 this meant it had to be empty. The emptying of the queue was done once a response had been received from the legacy hardware. This ensured a nice synchronous behaviour toward the legacy hardware but provided an asynchronous API to the users of the legacy hardware, for which there are many.
Instead of designing your application on a blocking manner using SynchronousQueue<Response>, design it in a nonblocking manner using SynchronousQueue<Promise<Response>>.
Your public Future<Response> issueCommandToLegacyHardware(Command command) should then use offer() to add a DefaultPromise<>() to the Queue, and then the netty pipeline can use remove() to get the response for that request, notice I used remove() instead of take(), since only under exceptional circumstances, there is none element present.
A quick implementation of this might be:
public class MyLastHandler extends SimpleInboundHandler<Response> {
private final SynchronousQueue<Promise<Response>> queue;
public MyLastHandler (SynchronousQueue<Promise<Response>> queue) {
super();
this.queue = queue;
}
// The following is called messageReceived(ChannelHandlerContext, Response) in 5.0.
#Override
public void channelRead0(ChannelHandlerContext ctx, Response msg) {
this.queue.remove().setSuccss(msg); // Or setFailure(Throwable)
}
}
The above handler should be placed last in the chain.
The implementation of public Future<Response> issueCommandToLegacyHardware(Command command) can look:
Channel channel = ....;
SynchronousQueue<Promise<Response>> queue = ....;
public Future<Response> issueCommandToLegacyHardware(Command command) {
return issueCommandToLegacyHardware(command, channel.eventLoop().newPromise());
}
public Future<Response> issueCommandToLegacyHardware(Command command, Promise<Response> promise) {
queue.offer(promise);
channel.write(command);
return promise;
}
Using the approach with the overload on issueCommandToLegacyHardware is also the design pattern used for Channel.write, this makes it really flexable.
This design pattern can be used as follows in client code:
issueCommandToLegacyHardware(
Command.TAKE_OVER_THE_WORLD_WITH_FIRE,
channel.eventLoop().newPromise()
).addListener(
(Future<Response> f) -> {
System.out.println("We have taken over the world: " + f.get());
}
);
The advantage of this design pattern is that no unneeded blocking is used anywhere, just plain async logic.
Appendix I: Javadoc:
Promise Future DefaultPromise
I am using Camel for my messaging application. In my use case I have a producer (which is RabbitMQ here), and the Consumer is a bean.
from("rabbitmq://127.0.0.1:5672/exDemo?queue=testQueue&username=guest&password=guest&autoAck=false&durable=true&exchangeType=direct&autoDelete=false")
.throttle(100).timePeriodMillis(10000)
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
MyCustomConsumer.consume(exchange.getIn().getBody())
}
});
Apparently, when autoAck is false, acknowledgement is sent when the process() execution is finished (please correct me if I am wrong here)
Now I don't want to acknowledge when the process() execution is finished, I want to do it at a later stage. I have a BlockingQueue in my MyCustomConsumer where consume() is putting messages, and MyCustomConsumer has different mechanism to process them. I want to acknowledge message only when MyCustomConsumer finishes processing messages from BlockingQueue. How can I achieve this?
You can consider to use the camel AsyncProcessor API to call the callback done once you processing the message from BlockingQueue.
I bumped into the same issue.
The Camel RabbitMQConsumer.RabbitConsumer implementation does
consumer.getProcessor().process(exchange);
long deliveryTag = envelope.getDeliveryTag();
if (!consumer.endpoint.isAutoAck()) {
log.trace("Acknowledging receipt [delivery_tag={}]", deliveryTag);
channel.basicAck(deliveryTag, false);
}
So it's just expecting a synchronous processor.
If you bind this to a seda route for instance, the process method returns immediately and you're pretty much back to the autoAck situation.
My understanding is that we need to make our own RabbitMQ component to do something like
consumer.getAsyncProcessor().process(exchange, new AsynCallback() {
public void done(doneSync) {
if (!consumer.endpoint.isAutoAck()) {
long deliveryTag = envelope.getDeliveryTag();
log.trace("Acknowledging receipt [delivery_tag={}]", deliveryTag);
channel.basicAck(deliveryTag, false);
}
}
});
Even then, the semantics of the "doneSync" parameter is not clear to me. I think it's merely a marker to identify whether we're dealing with a real async processor or a synchronous processor that was automatically wrapped into an async one.
Maybe someone can validate or invalidate this solution?
Is there a lighter/faster/stronger alternative?
Or could this be suggested as the default implementation for the RabbitMQConsumer?
I am writing my first camel application. it is a standalone application with a main method. As starting point i used the maven camel java archetype. It provides a simple main method that calls main.run().
Now i re-factored it a little bit and pulled the main.run out in a new class (and method) that will be my main-control of all camel stuff.
Now i want to create the "opposite" method of run(). At the moment i want to implement tests for single routs that start (run()) the context then wait (at the moment i am unsure how to wait 'til a route is finished) and the stop the context.
But now i discovered many method that could start and stop stuff all in Main class. The Jvadoc didn't help - that some methods are inherited doesn't make it easier ;-). So someone please tell me the exact meaning (or use case) for:
Main.run()
Main.start()
Main.stop()
Main.suspend()
Main.resume()
Thanks in advance.
See this page about the lifecycle of the various Camel services
http://camel.apache.org/lifecycle
And for waiting until a route is finished, then you can check the inflight registry if there is any current in-flight exchanges to know if a route is finished.
http://camel.apache.org/maven/current/camel-core/apidocs/org/apache/camel/spi/InflightRepository.html
We must separate the methods into 2 groups.
The first is the one described in the life cycle http://camel.apache.org/lifecycle
The second is composed of run and shutdown.
run runs indefinitely and can be stopped when invoking shutdown, the latter must be invoked in a different thread and sent before the run invocation.
Example:
import org.apache.camel.main.Main;
public class ShutdownTest {
public static void main(String[] args) throws Exception {
Main camel = new Main();
camel.addRouteBuilder( new MyRouteBuilder() );
// In this case the thread will wait a certain time and then invoke shutdown.
MyRunnable r = new MyRunnable(5000, camel);
r.excecute();
camel.run();
}
}
Simple Runnable class
public class MyRunnable implements Runnable {
long waitingFor = -1;
Main camel;
public MyRunnable(long waitingFor, Main camel){
this.waitingFor = waitingFor;
this.camel = camel;
}
public void excecute(){
Thread thread = new Thread(this);
thread.start();
}
#Override
public void run() {
try {
synchronized (this) {
this.wait( waitingFor );
}
} catch (InterruptedException e) {
}
try {
System.out.println("camel.shutdown()");
camel.shutdown();
} catch (Exception e) {
e.printStackTrace();
}
}
}