Solr - How to trigger the DIH from SolrNet API with C# code - solr

Solr version :: 6.6.1
SolrNet API with the C# based application
I wish to invoke or trigger the data import handler from the C# code with
the help of SolrNet. But i am unable to locate any tutorial in the SolrNet
API. I can easily invoke the DIH from the solr admin UI, but my need is to invoke it from an external application.
Please suggest the code snippet how do i invoke the data import action from the C# based
application ?

I don't think it's possible to do completely from Solr.NET, a brief look gives me an idea, that currently there is only class responsible for DIH status page, which is good, but not covering the initial process. I think this was discarded recently, since this functionality wasn't needed.
In the SolrBasicServer class you have:
public SolrDIHStatus GetDIHStatus(KeyValuePair<string, string> options) {
var response = connection.Get("/dataimport", null);
var dihstatus = XDocument.Parse(response);
return dihStatusParser.Parse(dihstatus);
}
which is getting the DIH. Most likely, you need to extend this class and do something similar (I'm not C# developer, so not 100% sure about the code):
connection.Post("/dataimport?command=full-import", null);
or something similar with delta-import command and then later get the status part.
If updating Solr.NET is not the case for you, you still could just trigger it via usual HTTP call with some preferable C# library and do POST request to http://host:port/solr/collection-name/dataimport?command=full-import

string solrTargetDIHUrl = "http://localhost:8983/solr/dih/dataimport?command=delta-import";
try
{
using (var solrClient = new HttpClient())
{
var resultObj = solrClient.GetAsync(new Uri(solrTargetDIHUrl)).Result;
Console.ForegroundColor = ConsoleColor.Green;
Console.WriteLine("\t\t Data Import Triggered Successfully !");
Console.ResetColor();
}
}
catch(Exception ex)
{
Console.WriteLine("ERROR in DIH Trigger >>>>> " + ex.Message + "||" + ex.StackTrace);
}

Related

How to retry requests using MS JAVA SDK for GRAPH API?

I am a beginner using Graph API and currently trying to handle throttling related errors in my code. In the sdk's github page there is a RetryHandler class. But i can't seem to find an implementation of the code for a sample request. Is there any examples which i can inspect and try?
As an example i can give my own code below.
#Override
public JsonObject getUserList(String accessToken){
LOGGER.trace("Querying directRoutingEnabled users");
final String selectQuery = "mail,givenName,surname,displayName";
final List<Option> requestOptions = new LinkedList<>();
final String filterQuery = createFilterQuery();
requestOptions.add(new HeaderOption(CONSISTENCY_LEVEL, EVENTUAL));
requestOptions.add(new QueryOption(SELECT, selectQuery));
requestOptions.add(new QueryOption(COUNT, true));
requestOptions.add(new QueryOption(FILTER, filterQuery));
IGraphServiceClient graphServiceClient = GraphServiceClient.builder().authenticationProvider(new SimpleAuthProvider(accessToken)).buildClient();
IUserCollectionRequest request = graphServiceClient
.users()
.buildRequest(requestOptions);
IUserCollectionPage users = request.get();
return users.getRawObject();
}
How can i cover this code with retry-after mechanism (and maybe also Error Handling) using Graph SDK for Java?.
The default middleware from the core library comes with the default RetryHandler implementation. And thanks to a recent fix, it works with throttling errors as well.
Starting from V3 of the SDK, you can specify custom OkHttpClient with custom interceptors. Customization guidelines can be found here: https://learn.microsoft.com/en-us/graph/sdks/customize-client?tabs=java.
Default RetryHandler implementation: https://github.com/microsoftgraph/msgraph-sdk-java-core/blob/dev/src/main/java/com/microsoft/graph/httpcore/RetryHandler.java
Default RetryHandler design: https://microsoftgraph.github.io/msgraph-sdk-design/middleware/RetryHandler.html

SolrJ and Custom Solr Handler

I am trying to implement a simple custom request handler in Solr 7.3. I needed some clarifications on the methods available via the Solr Java API.
As per my understanding, I have extended my Java Class with "SearchHandler" and then overridden the "handleRequestBody" method. I am trying to understand the flow from the beginning. Here is a sample query in the browser.
http://localhost:8983/solr/customcollection/customhandler?
q=John&fl=id,last_name&maxRows=10
1) Once you enter the above query in the browser and press
"return" the Solr customhandler will be triggered. It will look
for the necessary jars from where the handler is created.
2) Once it finds the main class it will execute the following
method, which is overridden from the "SearchHandler" parent
class.
public void handleRequestBody(SolrRequest req, SolrResponse
resp) throws Exception
3) The SolrRequest req object will hold all the Solr Parameters
on the query, in this case, q,fl and maxRows.
4) Using the following code I unpack these parameters.
SolrParams params = req.getParams();
String q = params.get(CommonParams.Q);
String fl = params.getParams(CommonParams.FL);
String rows = params.get(CommonParams.ROWS);
5)I create a Solr object that let's me connect to my Solr Cloud
String zkHostString = "localhost:5181";
SolrClient solr = new
CloudSolrClient.Builder().withZkHost(zkHostString).build();
6) Here is where I need help
a) How do I use the unpacked Solr Parameters from the
original query and make a call to the "solr" object to
return results.
b) How do I make use of the "resp" object?
c) Most of the examples that I found on the internet show
how to print the results to STDOUT. However, since I am
using a custom handler I would like to display the results
back to the user (in this case, SOLR Admin or the browser).```
Any help is truly appreciated.
Thanks
public class SolrQueryTest extends
org.apache.solr.handler.component.SearchHandler {
String zkHostString = "localhost:5181";
SolrClient solr = new
CloudSolrClient.Builder().withZkHost(zkHostString).build();
private static final Logger log =
Logger.getLogger(SolrQueryTest.class.getName());
public void handleRequestBody(SolrRequest req, SolrResponse
resp) throws Exception {
SolrParams params = req.getParams();
String q = params.get(CommonParams.Q);
String rows = params.get(CommonParams.ROWS);
SolrQuery query = new SolrQuery(q);
query.setShowDebugInfo(true);
query.set("indent", "true");
// need to know how to call SOLR using the above query
parameters
//Once the response is received how to send it back to the
browser and NOT STDOUT
}
}

How to execute blocking calls within a Spring Webflux / Reactor Netty web application

In my use case where I have a Spring Webflux microservice with Reactor Netty, I have the following dependencies:
org.springframework.boot.spring-boot-starter-webflux (2.0.1.RELEASE)
org.springframework.boot.spring-boot-starter-data-mongodb-reactive (2.0.1.RELEASE)
org.projectreactor.reactor-spring (1.0.1.RELEASE)
For a very specific case I need to retrieve some information from my Mongo database, and process this into query parameters send with my reactive WebClient. As the WebClient nor the UriComponentsBuilder accepts a Publisher (Mono / Flux) I used a #block() call to receive the results.
Since reactor-core (version 0.7.6.RELEASE) which has been included in the latest spring-boot-dependencies (version 2.0.1.RELEASE) it is not possible anymore to use: block()/blockFirst()/blockLast() are blocking, which is not supported in thread xxx, see -> https://github.com/reactor/reactor-netty/issues/312
My code snippet:
public Mono<FooBar> getFooBar(Foo foo) {
MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
parameters.add("size", foo.getSize());
parameters.addAll("bars", barReactiveCrudRepository.findAllByIdentifierIn(foo.getBarIdentifiers()) // This obviously returns a Flux
.map(Bar::toString)
.collectList()
.block());
String url = UriComponentsBuilder.fromHttpUrl("https://base-url/")
.port(8081)
.path("/foo-bar")
.queryParams(parameters)
.build()
.toString();
return webClient.get()
.uri(url)
.retrieve()
.bodyToMono(FooBar.class);
}
This worked with spring-boot version 2.0.0.RELEASE, but since the upgrade to version 2.0.1.RELEASE and hence the upgrade from reactor-core to version 0.7.6.RELEASE it is not allowed anymore.
The only real solution I see is to include a block (non-reactive) repository / mongo client as well, but I'm not sure if that is encouraged. Any suggestions?
The WebClient does not accept a Publisher type for its request URL, but nothing prevents you from doing the following:
public Mono<FooBar> getFooBar(Foo foo) {
Mono<List<String>> bars = barReactiveCrudRepository
.findAllByIdentifierIn(foo.getBarIdentifiers())
.map(Bar::toString)
.collectList();
Mono<FooBar> foobar = bars.flatMap(b -> {
MultiValueMap<String, String> parameters = new LinkedMultiValueMap<>();
parameters.add("size", foo.getSize());
parameters.addAll("bars", b);
String url = UriComponentsBuilder.fromHttpUrl("https://base-url/")
.port(8081)
.path("/foo-bar")
.queryParams(parameters)
.build()
.toString();
return webClient.get()
.uri(url)
.retrieve()
.bodyToMono(FooBar.class);
});
return foobar;
}
If anything, this new reactor-core inspection saved you from crashing your whole application with this blocking call in the middle of a WebFlux handler.

Setting request timeout with Netflix Feign and Hystrix

I'm creating a REST client using Feign. I've got my calls working, but I want to add some timeout support, and I'm having a heck of a time figuring out how to do that.
Feign's documentation says "to use Hystrix with Feign, add the Hystrix module to your classpath. Then use the HystrixFeign builder." Ok, so now I've got this:
service = HystrixFeign.builder()
.decoder(new GsonDecoder())
.target(ProjectService.class, URL_TO_BE_MOVED_TO_PROPS);
Now all of my methods are returning HystrixCommands, which I can execute or queue, but I still can't see how to configure them.
The Hystrix wiki (https://github.com/Netflix/Hystrix/wiki/Configuration) says that configuration should be added into the HystrixCommand constructor like this:
public HystrixCommandInstance(int id) {
super(Setter.withGroupKey(HystrixCommandGroupKey.Factory.asKey("ExampleGroup"))
.andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds(500)));
this.id = id;
But my commands are being built/return by Feign, so I don't have access to the constructors.
One more thing worth noting is that the Feign-Hystrix readme (https://github.com/Netflix/feign/tree/master/hystrix) says "To use Hystrix with Feign, add the Hystrix module to your classpath. Then, configure Feign to use the HystrixInvocationHandler," but a Google search for HystrixInvocationHandler points me toward a non-Netflix repo. Even if I used that, I don't see how to configure Feign to use it.
Please tell me I'm being dumb and that this is super simple, which will make me feel gladness that I'm past this issue, and shame for not being able to figure it out on my own.
TL;DR: I want to set timeouts on requests made by my Feign client. How do?
Turns out you can set Hystrix properties using an instance of com.netflix.config.ConfigurationManager (from com.netflix.archaius:archaius-core).
Feign uses method names as HystrixCommandKeys, so you can access their properties using those names:
ConfigurationManager.getConfigInstance().setProperty("hystrix.command." + methodName + ".execution.isolation.thread.timeoutInMilliseconds", 1500);
This is assuming you've used HystrixFeign to construct your client, which wraps each call in HystrixCommand objects.
To simplify, I created a loop of my methods so I could apply the timeout service-wide:
private void configureHystrix() {
Method[] methods = ProjectService.class.getMethods();
String methodName;
for(int i = 0; i < methods.length; i++) {
methodName = methods[i].getName();
ConfigurationManager.getConfigInstance().setProperty(String.format("hystrix.command.%s.execution.isolation.thread.timeoutInMilliseconds", methodName), config.getTimeoutInMillis());
}
}
After some debugging I managed to set Hystrix timeout as follows:
HystrixFeign.builder()
.setterFactory(getSetterFactory())
.target(...);
// copy-paste feign.hystrix.SetterFactory.Default, just add andCommandPropertiesDefaults
private SetterFactory getSetterFactory() {
return (target, method) -> {
String groupKey = target.name();
String commandKey = Feign.configKey(target.type(), method);
return HystrixCommand.Setter
.withGroupKey(HystrixCommandGroupKey.Factory.asKey(groupKey))
.andCommandKey(HystrixCommandKey.Factory.asKey(commandKey))
.andCommandPropertiesDefaults(HystrixCommandProperties.Setter()
.withExecutionTimeoutInMilliseconds(15000));
};
}

Using SolrNet to query Solr from a console application?

I'm trying to use SolrNet in a command line application (or more accurately, from LINQPad) to test some queries, and when trying to initialize the library, I get the following error:
Key 'SolrNet.Impl.SolrConnection.UserQuery+Resource.SolrNet.Impl.SolrConnection' already registered in container
However, if I catch this error and continue, the ServiceLocator gives me the following error:
Activation error occured while trying to get instance of type ISolrOperations`1, key ""
With the inner exception:
The given key was not present in the dictionary.
My full code looks like this:
try
{
Startup.Init<Resource>("http://localhost:8080/solr/");
Console.WriteLine("Initialized\n");
}
catch (Exception ex)
{
Console.WriteLine("Already Initialized: " + ex.Message);
}
// This line causes the error if Solr is already initialized
var solr = ServiceLocator.Current.GetInstance<ISolrOperations<Resource>>();
// Do the search
var results = solr.Query(new SolrQuery("title:test"));
I'm running Tomcat 7 on Windows 7x64 with Solr 3.4.0 installed.
There's another message about the same problem on StackOverflow, though the accepted answer of putting the Startup.Init code in Global.asax is only relevant to ASP.NET.
Restarting the Tomcat7 service resolves the problem, but having to do this after every query is a pain.
What is the correct way to use the SolrNet library to interact with Solr from a C# console application?
The correct way to use SolrNet in a console application is to only execute the line
Startup.Init<Resource>("http://localhost:8080/solr/");
once for the life of your console application. I typically put it as the first line in my Main method as shown below...
static void Main(string[] args)
{
Startup.Init<Resource>("http://localhost:8080/solr/");
//Call method or do work to query from solr here...
//Using your code in a method...
QuerySolr();
}
private static void QuerySolr()
{
var solr = ServiceLocator.Current.GetInstance<ISolrOperations<Resource>>();
// Do the search
var results = solr.Query(new SolrQuery("title:test"));
}
Your error is coming from the fact that you are trying to initialize the SolrNet connection multiple times. You only need to initialize it once when the console application starts and then reference (look up) via the ServiceLocator when needed.
My Solution is clear Startup before Init
Startup.Container.Clear();
Startup.InitContainer();
Startup.Init<Resource>("http://localhost:8080/solr/");

Resources