Ok, I understand that maybe very stupid question, but i never did it before, so i ask this question. How can i download file (let's say, from the internet) using Thread class?
What do you mean with "using Thread class"? I guess you want to download a file threaded so it does not block your UI or some other part of your program.
Ill assume that your using C++ and WINAPI.
First create a thread. This tutorial provides good information about WIN32 threads.
This thread will be responsible for downloading the file. To do this you simply connect to the webserver on port 80 and send a HTTP GET request for the file you want. It could look similar to this (note the newline characters):
GET /path/to/your/file.jpg HTTP/1.1\r\n
Host: www.host.com\r\n
Connection: close\r\n
\r\n
\r\n
The server will then answer with a HTTP response containing the file with a preceding header. Parse this header and read the contents.
More information on HTTP can be found here.
If would suggest that you do not use threads for downloading files. It's better to use asynchronous constructs that are more targeted towards I/O, since they will incur a lower overhead than threads. I don't know what version of the .NET Framework you are working with, but in 4.5, something like this should work:
private static Task DownloadFileAsync(string uri, string localPath)
{
// Get the http request
HttpWebRequest webRequest = WebRequest.CreateHttp(uri);
// Get the http response asynchronously
return webRequest.GetResponseAsync()
.ContinueWith(task =>
{
// When the GetResponseAsync task is finished, we will come
// into this contiuation (which is an anonymous method).
// Check if the GetResponseAsync task failed.
if (task.IsFaulted)
{
Console.WriteLine(task.Exception);
return null;
}
// Get the web response.
WebResponse response = task.Result;
// Open a file stream for the local file.
FileStream localStream = File.OpenWrite(localPath);
// Copy the contents from the response stream to the
// local file stream asynchronously.
return response.GetResponseStream().CopyToAsync(localStream)
.ContinueWith(streamTask =>
{
// When the CopyToAsync task is finished, we come
// to this continuation (which is also an anonymous
// method).
// Flush and dispose the local file stream. There
// is a FlushAsync method that will flush
// asychronously, returning yet another task, but
// for the sake of brevity I use the synchronous
// method here.
localStream.Flush();
localStream.Dispose();
// Don't forget to check if the previous task
// failed or not.
// All Task exceptions must be observed.
if (streamTask.IsFaulted)
{
Console.WriteLine(streamTask.Exception);
}
});
// since we end up with a task returning a task we should
// call Unwrap to return a single task representing the
// entire operation
}).Unwrap();
}
You would want to elaborate a bit on the error handling. What this code does is in short:
See the code comments for more detailed explanations of how it works.
Related
I checked the docs and stackoverflow but didn't find exactly a suiting approach.
E.g. this post seems very close: Dispatch a blocking service in a Reactive REST GET endpoint with Quarkus/Mutiny
However, I don't want so much unneccessary boilerplate code in my service, at best, no service code change at all.
I generally just want to call a service method which uses entity manager and thus is a blocking action, however, want to return a string to the caller immidiately like "query started" or something. I don't need a callback object, it's just a fire and forget approach.
I tried something like this
#NonBlocking
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public Uni<String> triggerQuery() {
return Uni.createFrom()
.item("query started")
.call(() -> service.startLongRunningQuery());
}
But it's not working -> Error message returned to the caller:
You have attempted to perform a blocking operation on a IO thread. This is not allowed, as blocking the IO thread will cause major performance issues with your application. If you want to perform blocking EntityManager operations make sure you are doing it from a worker thread.",
I actually expected quarkus takes care to distribute the tasks accordingly, that is, rest call to io thread and blocking entity manager operations to worker thread.
So I must using it wrong.
UPDATE:
Also tried an proposed workaround that I found in https://github.com/quarkusio/quarkus/issues/11535 changing the method body to
return Uni.createFrom()
.item("query started")
.emitOn(Infrastructure.getDefaultWorkerPool())
.invoke(()-> service.startLongRunningQuery());
Now I don't get an error, but service.startLongRunningQuery() is not invoked, thus no logs and no query is actually sent to db.
Same with (How to call long running blocking void returning method with Mutiny reactive programming?):
return Uni.createFrom()
.item(() ->service.startLongRunningQuery())
.runSubscriptionOn(Infrastructure.getDefaultWorkerPool())
Same with (How to run blocking codes on another thread and make http request return immediately):
ExecutorService executor = Executors.newFixedThreadPool(10, r -> new Thread(r, "CUSTOM_THREAD"));
return Uni.createFrom()
.item(() -> service.startLongRunningQuery())
.runSubscriptionOn(executor);
Any idea why service.startLongRunningQuery() is not called at all and how to achieve fire and forget behaviour, assuming rest call handled via IO thread and service call handled by worker thread?
It depends if you want to return immediately (before your startLongRunningQuery operation is effectively executed), or if you want to wait until the operation completes.
If the first case, use something like:
#Inject EventBus bus;
#NonBlocking
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public void triggerQuery() {
bus.send("some-address", "my payload");
}
#Blocking // Will be called on a worker thread
#ConsumeEvent("some-address")
public void executeQuery(String payload) {
service.startLongRunningQuery();
}
In the second case, you need to execute the query on a worker thread.
#POST
#Produces(MediaType.TEXT_PLAIN)
#Path("/query")
public Uni<String> triggerQuery() {
return Uni.createFrom(() -> service.startLongRunningQuery())
.runSubscriptionOn(Infrastructure.getDefaultWorkerPool());
}
Note that you need RESTEasy Reactive for this to work (and not classic RESTEasy). If you use classic RESTEasy, you would need the quarkus-resteasy-mutiny extension (but I would recommend using RESTEasy Reactive, it will be way more efficient).
Use the EventBus for that https://quarkus.io/guides/reactive-event-bus
Send and forget is the way to go.
Now I'm learning Bidirectional streaming in asynchronous GRPC++.
Thanks for the master:https://github.com/Mityuha/grpc_async. I get much useful information to know the realization principle of this mode.But I have a question about it:
Not much to say,the code is following:
the server:
if(!ok || mcounter >= greeting.size())//ctx_.IsCancelled() doesn't work
{
std::cout << "[ProceedMM]: Trying finish" << std::endl;
status_ = FINISH;
responder_.Finish(Status(), (void*)this);
}
the client:
void AsyncCompleteRpc()
{
void* got_tag;
bool ok = false;
while(cq_.Next(&got_tag, &ok))
{
AbstractAsyncClientCall* call = static_cast<AbstractAsyncClientCall*>(got_tag);
call->Proceed(ok);
}
std::cout << "Completion queue is shutting down." << std::endl;
}
in this server,the end of ClientStream is judged by the bool value of OK which is send by client.It isn't similar to the way of synchronous GRPC,which is judged the steaming end by the return of bool Read(RequestType* request) in the class of ServerReaderWriter in many times.It's so strange to find the same way in the class of ServerAsyncReaderWriter which is void Read(R* msg, void* tag).Though I know it's because of the asynchronous way.But if I don't know how much times of asynchronous streaming without the judgement of "OK", how to find the way like synchronous streaming to judge the end of client streaming.Because I test the performance by java which is the same code between synchronous with asynchronous ways,which don't have the bool value of OK in asynchronous ways.
So can someone help me?Or tell me some ways to deal with it or find a way to test the performance testing of GRPC++ by Bazel of in my another question.
I'm not 100% sure that I get the question, but what ok tells you is (when false) that the operation you requested couldn't be completed and nothing else will ever complete successfully on that side of the stream. So if you issue a Read operation and the Next gives you a !ok value, then you can be sure that no more data will ever come back from the client. A more detailed explanation is given in the comments for the CompletionQueue class.
Thanks and good luck with gRPC.
In the case of receiving a stream in an asynchronous client of gRPC, you will use a ClientAsyncReader<> class to receive data. This class differs when both send and receive are stream, but logic is the same.
This class has a Finish() method which you need to call after finishing sending your rpc data to server. When answer stream from server is finished, a message to CompletionQueue will be added which corresponds to this method. This Finish method returns final status when its message is returned in CQ. You can find out that your stream is finished. Your code will be similar to this:
response_reader_ = stub->PrepareAsyncXYZ(ctx_, req, cq);
response_reader_->StartCall(&start_data_);
response_reader_->Finish(&status_, &finish_data_);
in this sample, message in CQ will have finish_data_ tag and you can use it to handle it properly. You will probably will need to manage messages for Finish() and Read() by reference counting, because you will probably get an additional failed read message too. when message with finish_data_ is received in CQ, status_ will have the valid value of status.
At least it is how I wrote it.
I am creating a server which consumes commands from numerous sources such as JMS, SNMP, HTTP etc. These are all asynchronous and are working fine. The server maintains a single connection to a single item of legacy hardware which has a request/reply architecture with a custom TCP protocol.
Ideally I would like a single command like this blocking type method
public Response issueCommandToLegacyHardware(Command command)
or this asynchronous type method
public Future<Response> issueCommandToLegacyHardware(Command command)
I am relatively new to Netty and asynchronous programming, basically learning it as I go along. My current thought is that my LegacyHardwareClient class will have public synchronized issueCommandToLegacyHardware(Command command), will make a write to the client channel to the legacy hardware, then take() from a SynchronousQueue<Response> which will block. The ChannelInboundHandler in the pipeline will offer() a Response to the SynchronousQueue>Response> which will allow the take() to unblock and receive the data.
Is this too convoluted? Are there any examples around of synchronous Netty client implementations that I can look at? Are there any best practices for Netty?
I could obviously use just standard Java sockets however the power of Netty for parsing custom protocols along with the ease of maintaniability is far too great to give up.
UPDATE:
Just regarding the implementation, I used an ArrayBlockingQueue<>() and I used put() and remove() rather than offer() and remove(). Because I wanted to ensure that subsequent requests to the legacy hardware were only sent when any active requests had been replied to as the legacy hardware behaviour is not known with certainty otherwise.
The reason offer() and remove() did not work for me was that the offer() command would not pass anything if there was not an actively blocking take() request no the other side. The converse is true that remove() would not return anything unless there was a blocking put() call inserting data.
I couldn't use a put()/remove() since the remove() statement would never be reached since there was no request written to the channel to trigger the event from where the remove() would be called. I couldn't use offer()/take() since the offer() statement would return false since the take() call hadn't been executed yet.
Using the ArrayBlockingQueue<>() with a capacity of 1, it ensured that only one command could be executed at once. Any other commands would block until there was sufficient room to insert, with a capacity of 1 this meant it had to be empty. The emptying of the queue was done once a response had been received from the legacy hardware. This ensured a nice synchronous behaviour toward the legacy hardware but provided an asynchronous API to the users of the legacy hardware, for which there are many.
Instead of designing your application on a blocking manner using SynchronousQueue<Response>, design it in a nonblocking manner using SynchronousQueue<Promise<Response>>.
Your public Future<Response> issueCommandToLegacyHardware(Command command) should then use offer() to add a DefaultPromise<>() to the Queue, and then the netty pipeline can use remove() to get the response for that request, notice I used remove() instead of take(), since only under exceptional circumstances, there is none element present.
A quick implementation of this might be:
public class MyLastHandler extends SimpleInboundHandler<Response> {
private final SynchronousQueue<Promise<Response>> queue;
public MyLastHandler (SynchronousQueue<Promise<Response>> queue) {
super();
this.queue = queue;
}
// The following is called messageReceived(ChannelHandlerContext, Response) in 5.0.
#Override
public void channelRead0(ChannelHandlerContext ctx, Response msg) {
this.queue.remove().setSuccss(msg); // Or setFailure(Throwable)
}
}
The above handler should be placed last in the chain.
The implementation of public Future<Response> issueCommandToLegacyHardware(Command command) can look:
Channel channel = ....;
SynchronousQueue<Promise<Response>> queue = ....;
public Future<Response> issueCommandToLegacyHardware(Command command) {
return issueCommandToLegacyHardware(command, channel.eventLoop().newPromise());
}
public Future<Response> issueCommandToLegacyHardware(Command command, Promise<Response> promise) {
queue.offer(promise);
channel.write(command);
return promise;
}
Using the approach with the overload on issueCommandToLegacyHardware is also the design pattern used for Channel.write, this makes it really flexable.
This design pattern can be used as follows in client code:
issueCommandToLegacyHardware(
Command.TAKE_OVER_THE_WORLD_WITH_FIRE,
channel.eventLoop().newPromise()
).addListener(
(Future<Response> f) -> {
System.out.println("We have taken over the world: " + f.get());
}
);
The advantage of this design pattern is that no unneeded blocking is used anywhere, just plain async logic.
Appendix I: Javadoc:
Promise Future DefaultPromise
In my java code, I am processing huge amount of data. So I moved the code as servlet to Cron Job of App Engine. Some days it works fine. After the amount of the data increases, the cron job is not working and shows the following error message.
2012-09-26 04:18:40.627
'ServletName' 'MethodName': Inside SQLExceptionjava.sql.SQLRecoverableException:
Connection is already in use.
I 2012-09-26 04:18:40.741
This request caused a new process to be started for your application, and thus caused
your application code to be loaded for the first time. This request may thus take
longer and use more CPU than a typical request for your application.
W 2012-09-26 04:18:40.741
A problem was encountered with the process that handled this request, causing it to
exit. This is likely to cause a new process to be used for the next request to your
application. If you see this message frequently, you may be throwing exceptions during
the initialization of your application. (Error code 104)
How to handle this problem?
This exception is typical when a single connection is shared between multiple threads. This will in turn happen when your code does not follow the standard JDBC idiom of acquiring and closing the DB resources in the shortest possible scope in the very same try-finally block like so:
public Entity find(Long id) throws SQLException {
Connection connection = null;
// ...
try {
connection = dataSource.getConnection();
// ...
} finally {
// ...
if (connection != null) try { connection.close(); } catch (SQLException ignore) {}
}
return entity;
}
Your comment on the question,
#TejasArjun i used connection pooling with servlet Init() method.
doesn't give me the impression that you're doing it the right way. This suggests that you're obtaining a DB connection in servlet's init() method and reusing the same one across all HTTP requests in all HTTP sessions. This is absolutely not right. A servlet instance is created/initialized only once during webapp's startup and reused throughout the entire remaining of the application's lifetime. This at least confirms the exception you're facing.
Just rewrite your JDBC code according the standard try-finally idiom as demonstrated above and you should be all set.
See also:
Is it safe to use a static java.sql.Connection instance in a multithreaded system?
In SL4 application I perform a lot of requests (30) to the server in the same time (+~3 sec) by using HttpWebRequest. As a result I receive the same count of responses and start to process streams
HttpWebRequest request = (HttpWebRequest) asynchronousResult.AsyncState;
HttpWebResponse response = (HttpWebResponse) request.EndGetResponse(asynchronousResult)
using (var stream = new StreamReader(response.GetResponseStream()))
{
var str = stream.ReadToEnd();
var doc = XDocument.Parse(str);
}
All responsies are received (server returned 304 - not modified)
For each response program creates new thread (because of asynchronusResult).
When I try to fetch data from stream it takes a lot of time when stream.ReadToEnd() method works. I don't now what inside ReadToEnd() but it seems that stream reading performs in one thread or any locking occurs. Any ideas??
Why it happes? Everything incapsulated in separate threads and works fast except this method.
Presumably you are using the browser http stack, in which case your browser will almost certainly have a limit on the number of connections you can make to a given subdomain at once- in many browser this is two.
A common trick, is to use multiple sub domains to serve content.