We have an instance of Solr, where we've found that turning on autoCommit in the solrconfig.xml actually may serve our needs well. However there are some instances and some batch operations where we want to temporarily disable autocommit. I have not been able to find anything, but I'm wondering if anyone knew if via SolrJ you could disable autocommit for a certain process, and then re-enable it?
You can't disable and enable autocommit as it's configured in solrconfig.xml. However, you can leave it disabled in solrconfig.xml and use commitWithin for those add commands that need autocommit.
answering because this is the first result for "solr disable autocommit".
This is now possible with the new config API that allows to override some properties set in solrconfig.xml without reloading the core.
Solrj does not implement that new API yet.
You should not disable autocommits, see this article.
If you want to do a bulk indexing of many documents at once, set updateHandler.autoCommit.openSearcher=false and disable autoSoftCommits:
/**
* Disables the autoSoftCommit feature.
* Use {#link #reEnableAutoCommit()} to reenable.
* #throws IOException network error.
* #throws SolrServerException solr error.
*/
public void disableAutoSoftCommit() throws SolrServerException, IOException
{
// Solrj does not support the config API yet.
String command = "{\"set-property\": {" +
"\"updateHandler.autoSoftCommit.maxDocs\": -1," +
"\"updateHandler.autoSoftCommit.maxTime\": -1" +
"}}";
GenericSolrRequest rq = new GenericSolrRequest(SolrRequest.METHOD.POST, "/config", null);
ContentStream content = new ContentStreamBase.StringStream(command);
rq.setContentStreams(Collections.singleton(content));
rq.process(solrClient);
}
/**
* Undo {#link #disableAutoSoftCommit()}.
* #throws IOException network error.
* #throws SolrServerException solr error.
*/
public void reenableAutoSoftCommit() throws SolrServerException, IOException
{
// Solrj does not support the config API yet.
String command = "{\"unset-property\": [" +
"\"updateHandler.autoSoftCommit.maxDocs\"," +
"\"updateHandler.autoSoftCommit.maxTime\"" +
"]}";
GenericSolrRequest rq = new GenericSolrRequest(SolrRequest.METHOD.POST, "/config", null);
ContentStream content = new ContentStreamBase.StringStream(command);
rq.setContentStreams(Collections.singleton(content));
rq.process(solrClient);
}
You can see the overriden properties at http://localhost:8983/solr/<core>/config/overlay
Related
So starting to update ancient solr app to 9.1 and also the SolrJ indexer. When I try to add a document, I am getting
Exception in thread "main" org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://my.host:8983/solr/qmap:
Searching for Solr
You must type the correct path
Solr will respond
I can see the qmap core in the solr admin and solr is running.
Code is:
public class DocumentIndexer {
private final String fileToIndex;
private final ConcurrentUpdateHttp2SolrClient solrClient;
private final Http2SolrClient http2Client;
public DocumentIndexer(String solrUrl, String fileToIndex) {
this.fileToIndex =fileToIndex;
http2Client = new Http2SolrClient.Builder().build();
solrClient = new ConcurrentUpdateHttp2SolrClient.Builder(solrUrl, http2Client).build();
}
public void indexDocuments() throws IOException, SolrServerException{
ContentStreamUpdateRequest req = new ContentStreamUpdateRequest("/update/extract");
req.setAction(AbstractUpdateRequest.ACTION.COMMIT, true, true);
req.addFile(new File(fileToIndex),"application/xml");
req.setParam("id", fileToIndex);
req.process(solrClient);
solrClient.commit(true, true);
}
}
Simple enough - update/extract was not defined in the solrconfig. Recreating the core using the sample_techproducts_examples as template supplies this or alternatively setting up the solrconfig with the update/extract path defined.
Also, req.setParam("id", fileToIndex) needs to be changed to req.setParam("literal.id", fileToIndex)
I am trying to implement a simple custom request handler in Solr 7.3. I needed some clarifications on the methods available via the Solr Java API.
As per my understanding, I have extended my Java Class with "SearchHandler" and then overridden the "handleRequestBody" method. I am trying to understand the flow from the beginning. Here is a sample query in the browser.
http://localhost:8983/solr/customcollection/customhandler?
q=John&fl=id,last_name&maxRows=10
1) Once you enter the above query in the browser and press
"return" the Solr customhandler will be triggered. It will look
for the necessary jars from where the handler is created.
2) Once it finds the main class it will execute the following
method, which is overridden from the "SearchHandler" parent
class.
public void handleRequestBody(SolrRequest req, SolrResponse
resp) throws Exception
3) The SolrRequest req object will hold all the Solr Parameters
on the query, in this case, q,fl and maxRows.
4) Using the following code I unpack these parameters.
SolrParams params = req.getParams();
String q = params.get(CommonParams.Q);
String fl = params.getParams(CommonParams.FL);
String rows = params.get(CommonParams.ROWS);
5)I create a Solr object that let's me connect to my Solr Cloud
String zkHostString = "localhost:5181";
SolrClient solr = new
CloudSolrClient.Builder().withZkHost(zkHostString).build();
6) Here is where I need help
a) How do I use the unpacked Solr Parameters from the
original query and make a call to the "solr" object to
return results.
b) How do I make use of the "resp" object?
c) Most of the examples that I found on the internet show
how to print the results to STDOUT. However, since I am
using a custom handler I would like to display the results
back to the user (in this case, SOLR Admin or the browser).```
Any help is truly appreciated.
Thanks
public class SolrQueryTest extends
org.apache.solr.handler.component.SearchHandler {
String zkHostString = "localhost:5181";
SolrClient solr = new
CloudSolrClient.Builder().withZkHost(zkHostString).build();
private static final Logger log =
Logger.getLogger(SolrQueryTest.class.getName());
public void handleRequestBody(SolrRequest req, SolrResponse
resp) throws Exception {
SolrParams params = req.getParams();
String q = params.get(CommonParams.Q);
String rows = params.get(CommonParams.ROWS);
SolrQuery query = new SolrQuery(q);
query.setShowDebugInfo(true);
query.set("indent", "true");
// need to know how to call SOLR using the above query
parameters
//Once the response is received how to send it back to the
browser and NOT STDOUT
}
}
I have a case where I have a set of fields to be updated in solr. The input i recieve is in form of a map, key being field name and value is the updated value.
I had a doubt that in such a scenario should i be using curl to update the doc or solrj where I have to convert the map to solrInputDocument and then call add command. Will the first approach be faster than second?
You can convert the map to SolrInputdocument.
Curl uses HTTP URL. All requests go to HttpSolrServer(Im not quite sure about it).
However, in my experience, I would strongly recommend
ConcurrentUpdateSolrServer which is meant for updates
SolrJ has ConcurrentUpdateSolrServer class which describes like below.
ConcurrentUpdateSolrServer buffers all added documents and writes them
into open HTTP connections. This class is thread safe. Params from
UpdateRequest are converted to http request parameters. When params
change between UpdateRequests a new HTTP request is started. Although
any SolrServer request can be made with this implementation, it is
only recommended to use ConcurrentUpdateSolrServer with /update
requests. The class HttpSolrServer is better suited for the query
interface.
The below is example way to get Solr Server instance
/**
* Method lookUpConcurrentUpdateSolrServerInstance.
*
* #param baseURL String
* #param collection String
* #return ConcurrentUpdateSolrServer
* #throws SolrServerException
* #throws IOException
* #throws Exception
*/
public static ConcurrentUpdateSolrClient lookUpConcurrentUpdateSolrServerInstance(String baseURL, String collection)
throws SolrServerException, IOException, Exception {
ConcurrentUpdateSolrClient solrServer = null;
if (StringUtils.isNotBlank(baseURL) && StringUtils.isNotBlank(collection)) {
solrServer = new ConcurrentUpdateSolrClient(baseURL + collection, "queueSizeasInteger", 10),
"threadCount as integer", 10));
checkServerStatus(solrServer);
return solrServer;
}
return solrServer;
}
By default, CXF 3.0.5's JAXRSBeanValidationInInterceptor and JAXRSBeanValidationOutInterceptor do not support validation of arguments passed to request-scoped resource beans. This exclusion is enforced in org.apache.cxf.jaxrs.validation.ValidationUtils.getResourceInstance(Message). Attempting to use a request-scoped resource results in the following warning being logged:
Service object is not a singleton, use a custom invoker to validate
I've spent some time poking around and come up with the following workaround:
/**
* This is a replacement for CXF's builtin
* {#link JAXRSBeanValidationInInterceptor}. This customization supports
* validation of messages handled by non-singleton JAX-RS resource beans. This
* is needed as many of the beans in this project are request-scoped.
*/
public class BeanValidationInInterceptor extends
JAXRSBeanValidationInInterceptor {
/**
* This is a customization of the code in CXF's builtin
* {#link ValidationUtils#getResourceInstance(Message)}.
*
* #see org.apache.cxf.jaxrs.validation.JAXRSBeanValidationInInterceptor#getServiceObject(org.apache.cxf.message.Message)
*/
#Override
protected Object getServiceObject(Message message) {
final OperationResourceInfo ori = message.getExchange().get(
OperationResourceInfo.class);
if (ori == null) {
return null;
}
if (!ori.getClassResourceInfo().isRoot()) {
return message.getExchange().get(
"org.apache.cxf.service.object.last");
}
final ResourceProvider resourceProvider = ori.getClassResourceInfo()
.getResourceProvider();
return resourceProvider.getInstance(message);
}
}
It seems to work, but as I don't fully understand the reason this wasn't supported in the first place, I'm wondering if it's safe?
Any CXF devs around who can explain if/how I'm shooting myself in the foot here, and what I might do instead?
Im trying to implement google's Cloud Connection Server with Google App Engine following this tutorial -
Implementing an XMPP-based App Server. I copied latest smack jars from http://www.igniterealtime.org/projects/smack/ (smack.jar and smackx.jar), put them in WEB-INF/lib and added them to the classpath (im using eclipse).
In the code sample in the first link i posted, the XMPPConnection is initiated in a 'main' method. Since this is not really suitable to GAE i created a ServletContextListener and added it to web.xml.
public class GCMContextListener implements ServletContextListener {
private static final String GCM_SENDER_ID = "*GCM_SENDER_ID*";
private static final String API_KEY = "*API_KEY*";
private SmackCcsClient ccsClient;
public GCMContextListener() {
}
#Override
public void contextInitialized(ServletContextEvent arg0) {
final String userName = GCM_SENDER_ID + "#gcm.googleapis.com";
final String password = API_KEY;
ccsClient = new SmackCcsClient();
try {
ccsClient.connect(userName, password);
} catch (XMPPException e) {
e.printStackTrace();
}
}
#Override
public void contextDestroyed(ServletContextEvent arg0) {
try {
ccsClient.disconnect();
} catch (XMPPException e) {
e.printStackTrace();
}
}
}
web.xml
<web-app>
<listener>
<listener-class>com.myserver.bootstrap.GCMContextListener</listener-class>
</listener>
</web-app>
Now, when i start the GAE server i get the following exception :
java.lang.NoClassDefFoundError: javax.naming.directory.InitialDirContext is a restricted class. Please see the Google App Engine developer's guide for more details.
i searched the "Google App Engine developer's guide for more details" but couldnt find anything about this. can you please help me ?
Google App Engine restricts access to certain JRE classes. In fact they published a whitelist that shows you which classes are useable. It seems to me that the Smack library might require some reference to a directory context (maybe to create the XMPP messages?) and that is why your servlet causes this exception. The javax.naming.directory is not in the whitelist.
I'm currently working on setting up a GCM Server as well. It seems to me that you need to read through the example and see what that main method is doing. What I see is a connection to the GCM server:
try {
ccsClient.connect(userName, password);
} catch (XMPPException e) {
e.printStackTrace();
}
Then a downstream message being sent to a device:
// Send a sample hello downstream message to a device.
String toRegId = "RegistrationIdOfTheTargetDevice";
String messageId = ccsClient.getRandomMessageId();
Map<String, String> payload = new HashMap<String, String>();
payload.put("Hello", "World");
payload.put("CCS", "Dummy Message");
payload.put("EmbeddedMessageId", messageId);
String collapseKey = "sample";
Long timeToLive = 10000L;
Boolean delayWhileIdle = true;
ccsClient.send(createJsonMessage(toRegId, messageId, payload, collapseKey,
timeToLive, delayWhileIdle));
}
These operations would be completed at some point during your application's lifecycle, so your servlet should support them by providing the methods the example is implementing, such as the connect method that appears in the first piece of code that I pasted here. It's implementation is in the example at line 235 if I'm not mistaken.
As the documentation says, the 3rd party application server, which is what you're trying to implement using GAE, should be:
Able to communicate with your client.
Able to fire off properly formatted requests to the GCM server.
Able to handle requests and resend them as needed, using exponential back-off.
Able to store the API key and client registration IDs. The API key is included in the header of POST requests that send messages.
Able to store the API key and client registration IDs.
Able to generate message IDs to uniquely identify each message it sends.