I want to get the sender of all first messages in each thread in my G-mail inbox. To do so, it looks like I need to list all threads, fetch the thread itself with all of it's, get the first message, and then loop through its headers. Is there a more efficient way to do this?
Get sender mail in gmail-api
I've implemented the above and it works, but for 100 threads, it makes 1 List threads request + 100 Get thread requests.
There is no way to do this as of now.
You could however submit a Feature Request in the Google Issue Tracker and ask that the Gmail.Users.threads:list method get an option to retrieve the first message of each thread.
Related
I'm attempting to use a pull queue to create a queue of image processing tasks that could take longer that the acktimeout limit of 10 minutes. I'm using node.js api and I'm wondering how I could have a worker grab a message off the pull queue, mark it as in progress so no other workers attempt to grab it, do its work and acknowledge the message after the processing is done. This processing could take up to an hour per worker. If an exception occurs, I'd like to remove the "in progress" status and allow other workers to pick up this message and attempt to work on it.
I was hoping there was something in pubsub that would allow me to do this. My alternative is to, before processing, store an entity (inProgressMessage) with the message id, ack id, status=pending, timestamp=now() into datastore, have the worker immediately return the ackid after receiving the message (this will allow other workers to attempt other messages), then the worker can work on the lengthy task. If successful, mark the entity status as complete, if failed in a non permanent way, requeue the task into pubsub, if failed in a permanent way that won't allow reqeueing, I can have cron that checks datastore for pending tasks older than several hours and have them either be deleted or requeued.
My alternative feels like i'm re-implementing alot of what pub sub is supposed to help with.
Let me know if you can think of a better way.
To take longer than the ack deadline to process a message, you'll want to use modifyAckDeadline. You can extend the deadline as many times as you need up to 10 minutes per call. Your workflow would be as follows:
Pull the message.
Start to process the message.
While you are not done with the message, if you are close to the 10 minute ack deadline, call modifyAckDeadline to extend the deadline.
Once done processing the message, ack it.
Please note that calling modifyAckDeadline does not guarantee that the message won't be delivered to another task. In certain circumstances like server restarts, the message may end up being delivered to another of your subscribers. However, in most normal circumstances, as long as you call modifyAckDeadline before the current ack deadline, you can prevent a message's redelivered as long as necessary.
When creating a topic (only), you can configure the acknowledge time to be whatever up to 10 minutes (https://cloud.google.com/pubsub/subscriber). Once a message has been pulled from the queue, no other worker (of the same subscriber) will be able to take it for processing, unless the ack ttl was reached, and then the message is automatically returned to queue.
Since you need a longer period, you will have to implement something on your own, or seek another queuing solution. I think the design you suggested is fairly simple to implement, and is not really a re-implementation of what pubsub does.
What I want to achieve:
I am coding a Java program that uses IMAP to connect to some gmail accounts every 5 minutes and extract information from some messages.
I want to check all the messages (incoming and outgoing) and take only the ones I have not processed. By "processed" I do not mean only "read" or "seen" messages. My application does not care whether or not another user has accessed that account and read a message. My application needs to keep track of which was the last message it processed and, the next time it goes through the messages, start with the first non-processed message.
I do not want to change anything in the messages. I do not want to mark them as seen or read.
What I have implemented:
Establish IMAP connection.
Open and access all messages in "[Gmail]/All Mail" folder.
What I have tried:
I have been reading about UID and message number, but I am not sure if any of them could help me achieve what I want. Maybe UID could, but: how do I retrieve it with JavaMail?
I found Folder.getMessages(int start, int end), but I think it refers to the index of the message in a folder, which I believe can easily change.
Can anyone provide some guidance at what is the best approach to take here?
Thanks!
IMAP UIDs are relative to the folder containing the message. I don't know how Gmail handles UIDs for messages in the "[Gmail]/All Mail" folder, but if it does the right thing you could use the UIDFolder interface to get the UIDs. And as described, once you've processed a certain UID, all the new messages will have larger UIDs, which can make processing more efficient.
The alternative is to use Message-IDs, which has a different set of problems...
I am attempting to construct a route which will do the following:
Consume a message from jms:sender-in. I am using a INOUTrequest reply pattern. The JMSReplyTo = sender-out
The above message will be routed to multiple recipients like jms:consumer1-in, jms:consumer2-in and jms:consumer3-in. All are using a request reply pattern. The JMSReplyTo is specified per consumer ( in this case, the JMSReplyTo are in this order jms:consumer1-out, jms:consumer2-out, jms:consumer3-out
I need to aggregate all the replies together and send the result back to jms:sender-out.
I constructed a route which will resemble this:
from("jms:sender-in")
.to("jms:consumer1-in?exchangePattern=InOut&replyTo=queue:consumer1-out&preserveMessageQos=true")
.to("jms:consumer2-in?exchangePattern=InOut&replyTo=queue:consumer2-out&preserveMessageQos=true")
.to("jms:consumer3-in?exchangePattern=InOut&replyTo=queue:consumer3-out&preserveMessageQos=true");
I then send the replies back to some queue to gather and aggreagte:
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:consumer1-out?preserveMessageQos=true").to("jms:gather");
from("jms:gather").aggregate(header("TransactionID"), new GatherResponses()).completionSize(3).to("jms:sender-out");
To emulate the behavior of my consumers, I added the following route:
from("jms:consumer1-in").setBody(body());
from("jms:consumer2-in").setBody(body());
from("jms:consumer3-in").setBody(body());
I am getting a couple off issues:
I am getting a timeout error on the replies. If I comment out the gather part, then no issues. Why is there a timeout even though the replies are coming back to the queue and then forwarded to another queue.
How can I store the original JMSReplyTo value so Camel is able to send the aggregated result back to the sender's reply queue.
I have a feeling that I am struggling with some basic concepts. Any help is appreciated.
Thanks.
A good question!
There are two things you need to consider
Don't mix the exchange patterns, Request Reply (InOut) vs Event
message (InOnly). (Unless you have a good reason).
If you do a scatter-gather, you need to make the requests
multicast, otherwise they will be pipelined which is not
really scatter-gather.
I've made two examples which are similar to your case - one with Request Reply and one with (one way) Event messages.
Feel free to replace the activemq component with jms - it's the same thing in these examples.
Example one, using event messages - InOnly:
from("activemq:amq.in")
.multicast()
.to("activemq:amq.q1")
.to("activemq:amq.q2")
.to("activemq:amq.q3");
from("activemq:amq.q1").setBody(constant("q1")).to("activemq:amq.gather");
from("activemq:amq.q2").setBody(constant("q2")).to("activemq:amq.gather");
from("activemq:amq.q3").setBody(constant("q3")).to("activemq:amq.gather");
from("activemq:amq.gather")
.aggregate(new ConcatAggregationStrategy())
.header("breadcrumbId")
.completionSize(3)
.to("activemq:amq.out");
from("activemq:amq.out")
.log("${body}"); // logs "q1q2q3"
Example two, using Request reply - note that the scattering route has to gather the responses as they come in. The result is the same as the first example, but with less routes and less configuration.
from("activemq:amq.in2")
.multicast(new ConcatAggregationStrategy())
.inOut("activemq:amq.q4")
.inOut("activemq:amq.q5")
.inOut("activemq:amq.q6")
.end()
.log("Received replies: ${body}"); // logs "q4q5q6"
from("activemq:amq.q4").setBody(constant("q4"));
from("activemq:amq.q5").setBody(constant("q5"));
from("activemq:amq.q6").setBody(constant("q6"));
As for your question two - of course, it's possible to pass around JMSReplyTo headers and force exchange patterns along the road - but you will create hard to debug code. Keep your exchange patterns simple and clean - it keep bugs away.
I tried to use the list thread method:
GET https://www.googleapis.com/gmail/v1/users/{MyEmailAddress}/threads?labelIds=INBOX&labelIds=UNREAD&key={YOUR_API_KEY}
The resultSizeEstimate does not match the number next to my inbox in Gmail web app.
Is there another way?
The Gmail API now provides total and unread counts for messages and threads on each Label using the Labels.Get() method. See:
https://developers.google.com/gmail/api/release-notes
and
https://developers.google.com/gmail/api/v1/reference/users/labels
Turn conversation view off in the Gmail UI in order to get an accurate count of the message (not threads) in your inbox.
You can also use:
in:inbox is:unread
as the query in both the UI and API. Remove the labelId parameters.
GET https://www.googleapis.com/gmail/v1/users/{email}/messages?labelIds=UNREAD&labelIds=INBOX&fields=messages%2FthreadId
"messages/threadId" will count the number of unread threads but will list the same thread multiple times if the thread has multiple messages. Store them in an array and check the array as you fill it for matching "threadId" identifiers and ignore duplicates.
I send push notifications using PushSharp.
I call the method StopAllServices of PushService object to let it send the messages.
I assume that after PushService.StopAllServices finishes , I got the responses for all the Notification objecs.
Afterwards I kill the thread in which the PushService object is instanciated.
PushService = New PushService()
...
...
...
PushService.StopAllServices(True)
...
'the thread dies here
I noticed that sometimes I dont get responses in the callback functions.
Can it be that the method StopAllServices doesnt wait for all responses (from APNS/GCM) ?
I need to recieve responses for all Notification objects. How to achive that ?
I'll be glad for sugestions
From the Apple push Service (APNS) you won't get response for all the successfully delivered notifications and that is "by design".
Redth (the author of pushSharp) posted a detailed explanation of this : http://redth.info/the-problem-with-apples-push-notification-ser/
So you'll barely never get one response-per-notification-sent anyway, not useful to wait for all answers to stop the service.
And keep in mind that delivering the notification to the provider (android/apple/microsoft) doesn't guarantee you the effective device delivery of the notification anyway...