We are using azure search and need to implement a retry stratgey as well as storing the Ids of failed documents as described.
Is there any documentation/samples on how to implement a RetryPolicy strategy in Azure Search.
Thanks
This is what I used:
private async Task<DocumentIndexResult> IndexWithExponentialBackoffAsync(IndexBatch<IndexModel> indexBatch)
{
return await Policy
.Handle<IndexBatchException>()
.WaitAndRetryAsync(5, retryAttempt => TimeSpan.FromSeconds(Math.Pow(2, retryAttempt)), (ex, span) =>
{
indexBatch = ((IndexBatchException)ex).FindFailedActionsToRetry(indexBatch, x => x.Id);
})
.ExecuteAsync(async () => await _searchClient.IndexAsync(indexBatch));
}
It uses the Polly library to handle exponential backoff. In this case I use a model IndexModel that has a id field named Id.
If you like to log or store the ids of the failed attempts you can do that in the WaitAndRetryAsync function like
((IndexBatchException)ex)ex.IndexingResults.Where(r => !r.Succeeded).Select(r => r.Key).<Do something here>
There is currently no sample showing how to properly retry on IndexBatchException. However, there is a method you can use to make it easier to implement: IndexBatchException.FindFailedActionsToRetry. This method extracts the IDs of failed documents from the IndexBatchException, correlates them with the actions in a given batch, and returns a new batch containing only the failed actions that need to be retried.
Regarding the rest of the retry logic, you might find this code in the ClientRuntime library useful. You will need to tweak the parameters based on the characteristics of your load. The important thing to remember is that you should use exponential backoff before retrying to help your service recover, since otherwise your requests may be throttled.
Related
I'm using Apollo React Native client working with a query for which my request body has become too large to use (it's being rejected by our CDN for a request-too-large rule). So, I'm hoping to split/chunk this request into smaller requests and particularly curious if it's possible to do parallelized.
I think this is better illustrated with an example, so we can imagine I'm building a WhatsApp challenger -- WhoseApp -- for which we want users to be able to see who of their contacts have a WhoseApp account upon signup.
For our implementation, we'll take all of the phone numbers stored on our user's device and send them to our GraphQL query GetPhoneNumberAccountStatus which accepts an array of phone numbers and which returns an Account for each number associated to an account (and nothing for those that are not).
If we send the contacts as one request, we'll have a request body that looks something like this:
[
"+15558675309",
"+15558675308",
"+15558675307"
"+15558675306"
...
// 500+ numbers for some users
]
What's the correct way to split this request into multiple?
I'm curious of both:
What's the 'optimal' way to approach this using a sequential approach (e.g., send one group, wait for response, send next group), or
Is there a way to do this parallelized (e.g., send all groups at beginning and then receive responses as they arrive)?
I initially figured it might be possible to use useLazyQuery and send tranches of ~50 numbers at a time, firing each group and then awaiting the responses but this GitHub thread for the library makes it clear that that's not the correct approach.
I think it's readable
const promises = [];
const chunkSize = 50;
for (let i = 0; i <= contacts.length; i += chunkSize) {
const promise = apollo.query({...dataHere});
promises.push(promise);
}
await Promise.all(promises);
I was trying to debug a problem related to refunding Paypal orders (in a sandbox environment) using order IDs (which were stored previously). Every time I tried to perform a refund, the Paypal API would return an INVALID_RESOURCE_ID error, meaning that no such order existed. After much debugging, I have made a revelation with the initial process when I stored said order ID. The following method is how I am retrieving and storing said order id:
const onApprove = (data, actions) => {
// Redux method of saving checkout in backend with order ID via using data.orderID
dispatch(saveCheckout(data.orderID);
return actions.order.capture();
}
<PayPalButton
amount={totalPrice}
currency= "AUD"
createOrder={(data, actions) => createOrder(data, actions)}
onApprove={(data, actions) => onApprove(data, actions)}
options={{
clientId: "<placeholder>",
currency: "AUD"
}}
/>
I am using the recommended data.orderID from the docs and yet, upon inspecting the network tab, the following is shown:
{"id":"5RJ421191B663801G","intent":"CAPTURE","status":"COMPLETED","purchase_units":[{"reference_id":"default","amount":{"currency_code":"AUD","value":"24.00"},"payee":{"email_address":"sb-sg4zd7438633#business.example.com","merchant_id":"EJ7NSJGC6SRXQ"},"shipping":{"name":{"full_name":"John Doe"},"address":{"address_line_1":"1 Cheeseman Ave Brighton East","admin_area_2":"Melbourne","admin_area_1":"Victoria","postal_code":"3001","country_code":"AU"}},"payments":{"captures":[{"id":"7A2856455D561633D","status":"COMPLETED","amount":{"currency_code":"AUD","value":"24.00"},"final_capture":true,"seller_protection":{"status":"ELIGIBLE","dispute_categories":["ITEM_NOT_RECEIVED","UNAUTHORIZED_TRANSACTION"]},"create_time":"2021-10-11T00:40:58Z","update_time":"2021-10-11T00:40:58Z"}]}}],"payer":{"name":{"given_name":"John","surname":"Doe"},"email_address":"sb-432azn7439880#personal.example.com","payer_id":"KMEQSKCLCLUZ4","address":{"country_code":"AU"}},"create_time":"2021-10-11T00:40:48Z","update_time":"2021-10-11T00:40:58Z","links":[{"href":"https://api.sandbox.paypal.com/v2/checkout/orders/5RJ421191B663801G","rel":"self","method":"GET"}]}
The id saved by onApprove is 5RJ421191B663801G but there is another ID under captures and id which is 7A2856455D561633D. This is the actual order id I need to save in order to make the refund later on. However, I am struggling as to how I can retrieve this value as that id value seems to be only visible via the network. The objects returned via the onApprove and action.order.get() methods only return the first "false" id. Any advice would be greatly appreciated.
These are two separate types of IDs, the order ID (used only during buyer checkout approval), and the payment/transaction ID (which only exists after an order is captured, and is the one needed for any later refund or accounting purposes)
Since you are capturing on the client side with actions.order.capture(), this is where you would need to add a .then(function(data){ ... }) to do something with the capture data (particularly data.purchase_units[0].payments.captures[0].id). That is the id you would use for a refund.
In actual best practice, if anything important needs to be done with the capture id -- such as storing it in a database for reference -- you should not be creating and capturing orders on the client side, and instead calling a server-side integration where that database write will be performed.
Follow the Set up standard payments guide and make 2 routes on your server, one for 'Create Order' and one for 'Capture Order', documented here. Both routes should return only JSON data (no HTML or text). Inside the 2nd route, when the capture API is successful you should store its resulting payment details in your database (particularly the aforementioned purchase_units[0].payments.captures[0].id, which is the PayPal transaction ID) and perform any necessary business logic (such as sending confirmation emails or reserving product) immediately before forwarding your return JSON to the frontend caller.
Pair those 2 routes with the frontend approval flow: https://developer.paypal.com/demo/checkout/#/pattern/server
Or for react, use the official react-paypal-js
I am thinking should I audit user queries in HttpRequestInterceptor or DiagnosticEventListener for Hot Chocolate v11. The problem with latter is that if the audit failed to write to disk/db, the user will "get away" with the query.
Ideally if audit fail, no operation should proceed. Therefore in theory I should use HttpRequestInterceptor.
But How do I get IRequestContext from IRequestExecutor or IQueryRequestBuilder. I tried googling but documentation is limited.
Neither :)
The HttpRequestInterceptor is meant for enriching the GraphQL request with context data.
The DiagnosticEventListener, on the other hand, is meant for logging or other instrumentations.
If you want to write an audit log, you should instead go for a request middleware. A request middleware can be added like the following.
services
.AddGraphQLServer()
.AddQueryType<Query>()
.UseRequest(next => async context =>
{
})
.UseDefaultPipeline();
The tricky part here is to inspect the request at the right time. Instead of appending to the default pipeline, you can define your own pipeline like the following.
services
.AddGraphQLServer()
.AddQueryType<Query>()
.UseInstrumentations()
.UseExceptions()
.UseTimeout()
.UseDocumentCache()
.UseDocumentParser()
.UseDocumentValidation()
.UseRequest(next => async context =>
{
// write your audit log here and invoke next if the user is allowed to execute
if(isNotAllowed)
{
// if the user is not allowed to proceed create an error result.
context.Result = QueryResultBuilder.CreateError(
ErrorBuilder.New()
.SetMessage("Something is broken")
.SetCode("Some Error Code")
.Build())
}
else
{
await next(context);
}
})
.UseOperationCache()
.UseOperationResolver()
.UseOperationVariableCoercion()
.UseOperationExecution();
The pipeline is basically the default pipeline but adds your middleware right after the document validation. At this point, your GraphQL request is parsed and validated. This means that we know it is a valid GraphQL request that can be processed at this point. This also means that we can use the context.Document property that contains the parsed GraphQL request.
In order to serialize the document to a formatted string use context.Document.ToString(indented: true).
The good thing is that in the middleware, we are in an async context, meaning you can easily access a database and so on. In contrast to that, the DiagnosticEvents are sync and not meant to have a heavy workload.
The middleware can also be wrapped into a class instead of a delegate.
If you need more help, join us on slack.
Click on community support to join the slack channel:
https://github.com/ChilliCream/hotchocolate/issues/new/choose
I'm working on a UWP app that needs to check to see if a mobile service table has any records.
I realize I could do something like this and check to see if any records are returned:
records = await MyTable.Select(x => x).ToCollectionAsync();
However, is there a way to do this that won't attempt to download every record into a collection? What would be great is if something like this were allowed, but it isn't:
setupOfferors = await setupOfferorsTable.Select(so => so).FirstOrDefault();
Thanks!
There's no built-in method for you to check if the Azure mobile service table is empty. But you could try to apply the specified take clause to the source query and execute this query to see if it will return data.
For example:
await setupOfferorsTable.Select(so => so).Take(1).ToCollectionAsync();
I'm using gcloud-node.
createTopic api returns error 409, if that topic exist already. Is there a flag that can implicitly create a topic when publishing a message or Is there an API to check if a topic exist already?
Its easy to use getTopics API, iterate thru the response topic array and determine if a topic exist. Just wanted to make sure I dont write something, if it exists already.
Is there a flag that can implicitly create a topic when publishing a message or Is there an API to check if a topic exist already?
I believe the problem you'll run into is that if a message is published to a topic that doesn't exist, it is immediately dropped. So, it won't hang around and wait for a subscription to be created; it'll just disappear.
However, gcloud-node does have methods that will create a topic if necessary:
var topic = pubsub.topic('topic-that-maybe-exists');
topic.get({ autoCreate: true }, function(err, topic) {
// topic.publish(...
});
In fact, almost all gcloud-node objects have the get method that will work the same way as above, i.e. a Pub/Sub subscription or a Storage bucket or a BigQuery dataset, etc.
Here's a link to the topic.get() method in the docs: https://googlecloudplatform.github.io/gcloud-node/#/docs/v0.37.0/pubsub/topic?method=get
ran into this recently, and the accepted answer runs you into http 429 errors. topic.get is an administrative function which has a significantly lower rate limit than normal functions. You should only call them when neccessary eg. error code 404 during publish (topic doesn't exist), something like so:
topic.publish(payload, (err) => {
if(err && err.code === 404){
topic.get({ autoCreate: true }, (err, topic) => {
topic.publish(payload)
});
}
});
Personally use this one
const topic = pubsub.topic('topic-that-maybe-exists');
const [exists] = await topic.exists();
if (!exists) {
await topic.create();
}