I am unable build up a query in Solr - solr

I am trying to form a query in solr for data import but could not able to do so.
I need to form the below query:
http://salsa23q-XXX-08.XXX.XXX.com:8080/solr/#/geoloc_replica1/dataimport/?command=full-import&clean=true
The code I am trying:
public class SolrJDB
{
public static String url = "http://salsa23q-XXX-08.XXX.XXX.com:8080/solr:8080/solr";
public static SolrServer localserver;
public static CloudSolrServer cloudserver;// = new CloudSolrServer("url");
public static SolrQuery que;
public static SolrInputDocument doc;
public static SolrDocumentList list;
public static QueryResponse response;
public static String serverurl = "http://salsa23q-XXX-08.XXX.XXX.com:8080/solr";
public static void main(String[] args) throws MalformedURLException, SQLException, SolrServerException {
try{
System.out.println("+++++++++++++ Starting here +++++++++++++++++++++");
//Cloud Server
String url = "salsa23q-XXX-08.XXX.XXX.com:8080/solr";
cloudserver = new CloudSolrServer(url);
SolrQuery parameters = new SolrQuery();
parameters.set("qt","/geoloc_replica1");
parameters.set("qt","//dataimport");
parameters.set("command","full-import");
System.out.println("Query to be Executed ============"+parameters.toString());
QueryResponse response = cloudserver.query(parameters);
SolrDocumentList list = response.getResults();
}
catch(SolrServerException e){
System.out.println(e.toString());
e.printStackTrace();
}
catch(Exception e){
System.out.println(e.toString());
e.printStackTrace();
}
}
}
I am getting following error:
org.apache.solr.client.solrj.SolrServerException: Error executing query
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:98)
at org.apache.solr.client.solrj.SolrServer.query(SolrServer.java:301)
at SolrJDB.main(SolrJDB.java:37)
Caused by: java.lang.RuntimeException
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:115)
at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:83)
at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:138)
at org.apache.solr.client.solrj.impl.CloudSolrServer.connect(CloudSolrServer.java:140)
at org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:165)
at org.apache.solr.client.solrj.request.QueryRequest.process(QueryRequest.java:90)
... 2 more

You are mixing up things in your code. You want to perform a request to run a DataImportHandler. But you are constructing a SolrQuery. The latter represents a search request.
What you need to run a DataImportHandler via the Java API is an UpdateRequest.
public int runDataImportHandler() throws Exception {
// fill in the parameters you want to run your import with
ModifiableSolrParams tmpParams = new ModifiableSolrParams();
tmpParams.set("command", "full-import");
tmpParams.set("clean", true);
tmpParams.set("commit", true);
tmpParams.set("optimize", true);
// create the update request
UpdateRequest tmpRequest = new UpdateRequest("/dataimport");
tmpRequest.setParams(tmpParams);
SolrServer tmpServer = getServer();
tmpRequest.process(tmpServer);
ModifiableSolrParams tmpStatusParams = new ModifiableSolrParams();
tmpStatusParams.set("command", "status");
String tmpStatus = "busy";
int tmpProcessed = 0;
do {
System.out.println("waiting for import to finish, status was " + tmpStatus);
Thread.sleep(500);
UpdateRequest tmpStatusRequest = new UpdateRequest("/dataimport");
tmpStatusRequest.setParams(tmpStatusParams);
UpdateResponse tmpStatusResponse = tmpStatusRequest.process(tmpServer);
tmpStatus = tmpStatusResponse.getResponse().get("status").toString();
Map tmpMessages = (Map) tmpStatusResponse.getResponse().get("statusMessages");
System.out.println("import status is " + tmpStatus);
if (tmpMessages.get("Total Documents Processed") != null) {
tmpProcessed = Integer.valueOf(tmpMessages.get("Total Documents Processed").toString());
}
} while ("busy".equals(tmpStatus));
System.out.println("import done");
return tmpProcessed;
}

As I can see your code the exception is related to how CloudSolrServer instance is created :-
To make instance of CloudSolrServer you need to have Solr in Cloud mode and the URL used here will be zookeeper address. CloudSolrServer creates zkStateReader which gets live nodes and collection from which it query.
Following is the exact way how you would be creating the code for dataimport:-
public static void main(String[] args) throws SolrServerException, IOException {
ModifiableSolrParams params = new ModifiableSolrParams();
params.set(UpdateParams.COLLECTION, "collection1");
params.set(CommonParams.QT, "/dataimport");
params.set("command", "full-import");
params.set("claen", true);
params.set(UpdateParams.COMMIT, true);
params.set(UpdateParams.OPTIMIZE, true);
String url = "localhost:9983"; /*Zookeeper Address*/
CloudSolrServer cloudserver = new CloudSolrServer(url, true);
cloudserver.setDefaultCollection("test");/*This is necessary in case if you are not specifying any collection name in dataimport*/
cloudserver.query(params);
}

Related

BaseX parrallel Client

I have client like this :
import org.basex.api.client.ClientSession;
#Slf4j
#Component(value = "baseXAircrewClient")
#DependsOn(value = "baseXAircrewServer")
public class BaseXAircrewClient {
#Value("${basex.server.host}")
private String basexServerHost;
#Value("${basex.server.port}")
private int basexServerPort;
#Value("${basex.admin.password}")
private String basexAdminPassword;
#Getter
private ClientSession session;
#PostConstruct
private void createClient() throws IOException {
log.info("##### Creating BaseX client session {}", basexServerPort);
this.session = new ClientSession(basexServerHost, basexServerPort, UserText.ADMIN, basexAdminPassword);
}
}
It is a singleton injected in a service which run mulitple queries like this :
Query query = client.getSession().query(finalQuery);
return query.execute();
All threads query and share the same session.
With a single thread all is fine but with multiple thread I get some random (and weird) error, like the result of a query to as a result of another.
I feel that I should put a synchronized(){} arround query.execute() or open and close session for each query, or create a pool of session.
But I don't find any documentation how the use the session in parrallel.
Is this implementation fine for multithreading (and my issue is comming from something else) or should I do it differently ?
I ended creating a simple pool by adding removing the client from a ArrayBlockingQueue and it is working nicely :
#PostConstruct
private void createClient() throws IOException {
log.info("##### Creating BaseX client session {}", basexServerPort);
final int poolSize = 5;
this.resources = new ArrayBlockingQueue < ClientSession > (poolSize) {
{
for (int i = 0; i < poolSize; i++) {
add(initClient());
}
}
};
}
private ClientSession initClient() throws IOException {
ClientSession clientSession = new ClientSession(basexServerHost, basexServerPort, UserText.ADMIN, basexAdminPassword);
return clientSession;
}
public Query query(String finalQuery) throws IOException {
ClientSession clientSession = null;
try {
clientSession = resources.take();
Query result = clientSession.query(finalQuery);
return result;
} catch (InterruptedException e) {
log.error("Error during query execution: " + e.getMessage(), e);
} finally {
if (clientSession != null) {
try {
resources.put(clientSession);
} catch (InterruptedException e) {
log.error("Error adding to pool : " + e.getMessage(), e);
}
}
}
return null;
}

How to make BasicAuthentication in SolrJ - CloudSolrClient?

I am using Solr 7.5 Server and I had used External Zookeeper.When I browse using the Solr Admin UI It ask authentication to me.
For Java Client I had used the below Code
BasicAuthSolrClientCache bs = new BasicAuthSolrClientCache("solr", "SolrRocks");
CloudSolrClient solrCloudClient = bs.getCloudSolrClient(zkHost);
solrCloudClient.setDefaultCollection("sample");
SolrInputDocument doc = new SolrInputDocument();
doc.addField("cat", "book");
doc.addField("id", "book-1");
doc.addField("name", "The Legend of the Hobbit part 1");
solrCloudClient.add(doc);
solrCloudClient.commit();
solrCloudClient.close();
BasicAuthSolrClientCache.java
public class BasicAuthSolrClientCache extends SolrClientCache {
private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
private final Map<String, SolrClient> solrClients = new HashMap<>();
private final String username;
private final String password;
public BasicAuthSolrClientCache(String username, String password) {
this.username = username;
this.password = password;
}
#Override
public synchronized CloudSolrClient getCloudSolrClient(String zkHost) {
CloudSolrClient client;
if (solrClients.containsKey(zkHost)) {
client = (CloudSolrClient) solrClients.get(zkHost);
} else {
client = new CloudSolrClient.Builder()
.withZkHost(zkHost)
.withHttpClient(getHttpClient())
.build();
client.connect();
solrClients.put(zkHost, client);
}
return client;
}
#Override
public synchronized HttpSolrClient getHttpSolrClient(String host) {
HttpSolrClient client;
if (solrClients.containsKey(host)) {
client = (HttpSolrClient) solrClients.get(host);
} else {
client = new HttpSolrClient.Builder(host)
.withHttpClient(getHttpClient())
.build();
solrClients.put(host, client);
}
return client;
}
#Override
public synchronized void close() {
for(Map.Entry<String, SolrClient> entry : solrClients.entrySet()) {
try {
entry.getValue().close();
} catch (IOException e) {
log.error("Error closing SolrClient for " + entry.getKey(), e);
}
}
solrClients.clear();
}
private HttpClient getHttpClient() {
CredentialsProvider provider = new BasicCredentialsProvider();
UsernamePasswordCredentials credentials = new
UsernamePasswordCredentials(this.username, this.password);
provider.setCredentials(AuthScope.ANY, credentials);
return
HttpClientBuilder.create().setDefaultCredentialsProvider(provider).
build();
}
}
But it give the exception like the below,
Exception in thread "main" org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: IOException occured when talking to server at: http://192.168.0.104:8983/solr/gettingstarted_shard2_replica1 at
How to authenticate SolrCloud using SolrJ
I found a easy way to do this.
You add a request interceptor like this so you do not have to worry about creating a properly configured HttpClient instance yourself. This will just add the interceptor to the default HttpClient that Solrj creates.
org.apache.solr.client.solrj.impl.HttpClientUtil.addRequestInterceptor(new SolrPreemptiveAuthInterceptor());
The RequestInterceptor looks like this:
public class SolrPreemptiveAuthInterceptor implements HttpRequestInterceptor {
final static Logger log = LoggerFactory.getLogger(SolrPreemptiveAuthInterceptor.class);
#Override
public void process(HttpRequest request, HttpContext context) throws HttpException, IOException {
AuthState authState = (AuthState) context.getAttribute(HttpClientContext.TARGET_AUTH_STATE);
// If no auth scheme available yet, try to initialize it preemptively
if (authState.getAuthScheme() == null) {
log.info("No AuthState: set Basic Auth");
HttpHost targetHost = (HttpHost) context.getAttribute(HttpCoreContext.HTTP_TARGET_HOST);
AuthScope authScope = new AuthScope(targetHost.getHostName(), targetHost.getPort());
CredentialsProvider credsProvider = (CredentialsProvider) context.getAttribute(HttpClientContext.CREDS_PROVIDER);
Credentials creds = credsProvider.getCredentials(authScope);
if(creds == null){
log.info("No Basic Auth credentials: add them");
creds = getCredentials(authScope);
}
authState.update(new BasicScheme(), creds);
}
}
private Credentials getCredentials(AuthScope authScope) {
String user = "";
String password = "";
UsernamePasswordCredentials creds = new UsernamePasswordCredentials(user, password);
CredentialsProvider credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(authScope, creds);
log.info("Creating Basic Auth credentials for user {}", user);
return credsProvider.getCredentials(authScope);
}
}
You can also use UpdateRequest for indexing requests to do a basic authentication via SolrJ:
UpdateRequest ur = new UpdateRequest();
ur.add(doc);
ur.setBasicAuthCredentials("YOU USER NAME", "USER PASSWORD");
ur.setCommitWithin(COMMIT_WITHIN_INTERVAL);
ur.process(cloudSolrClient);

Lucene 6.5.1 TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset()

Hi I have written class to test lucene index creation getting below error.
Lucene version 6.5.1.
Exception in thread "main" java.lang.IllegalStateException: TokenStream contract violation: reset()/close() call missing, reset() called multiple times, or subclass does not call super.reset(). Please see Javadocs of TokenStream class for more information about the correct consuming workflow.
at org.apache.lucene.analysis.Tokenizer$1.read(Tokenizer.java:109)
at java.io.Reader.read(Reader.java:140)
at org.apache.lucene.analysis.pattern.PatternTokenizer.fillBuffer(PatternTokenizer.java:162)
at org.apache.lucene.analysis.pattern.PatternTokenizer.reset(PatternTokenizer.java:151)
at org.apache.lucene.analysis.TokenFilter.reset(TokenFilter.java:70)
at org.apache.lucene.analysis.TokenFilter.reset(TokenFilter.java:70)
at org.apache.lucene.analysis.TokenFilter.reset(TokenFilter.java:70)
at org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter.reset(RemoveDuplicatesTokenFilter.java:79)
at org.apache.lucene.analysis.TokenFilter.reset(TokenFilter.java:70)
at org.apache.lucene.analysis.ngram.EdgeNGramTokenFilter.reset(EdgeNGramTokenFilter.java:110)
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:742)
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:447)
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:403)
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocuments(DocumentsWriterPerThread.java:273)
at org.apache.lucene.index.DocumentsWriter.updateDocuments(DocumentsWriter.java:433)
at org.apache.lucene.index.IndexWriter.updateDocuments(IndexWriter.java:1382)
at org.apache.lucene.index.IndexWriter.addDocuments(IndexWriter.java:1358)
Please find below standalone program and please check Analyzer create component part i am not getting proper example for 6.5.1 version.
Below code is used to create index for Id and name using lucene 6.5.1
public class LuceneWriteIndexExample
{
private static final String INDEX_DIR = "c:/temp/lucene6index";
static private Analyzer analyzer = new Analyzer() {
#Override
protected TokenStreamComponents createComponents(String fieldName) {
PatternTokenizer tokenizer = new PatternTokenizer(Pattern.compile("[/\\-\\t ,;\\.]+"), -1);
TokenStream filter = new LowerCaseFilter(tokenizer);
filter = new TrimFilter(filter);
//filter = new ICUFoldingFilter(filter);
filter = new RemoveDuplicatesTokenFilter(filter);
filter = new EdgeNGramTokenFilter(filter, 1, 25);
return new TokenStreamComponents(tokenizer, filter);
}
};
public static void main(String[] args) throws Exception
{
IndexWriter writer = createWriter();
List<Document> documents = new ArrayList();
Document document1 = createDocument(1, "Test 1");
documents.add(document1);
Document document2 = createDocument(2, "Test 2");
documents.add(document2);
//Let's clean everything first
writer.deleteAll();
writer.addDocuments(documents);
// writer.flush();
writer.commit();
writer.close();
}
private static Document createDocument(Integer id, String name) throws IOException {
Document document = new Document();
document.add(new StringField("id", id.toString() , Field.Store.YES));
document.add(new TextField("name", name, Field.Store.YES));
TokenStream tokenStream = analyzer.tokenStream("customeName", name);
document.add(new TextField("customeName", tokenStream));
return document;
}
private static IndexWriter createWriter() throws IOException
{
FSDirectory dir = FSDirectory.open(Paths.get(INDEX_DIR));
IndexWriterConfig config = new IndexWriterConfig(new StandardAnalyzer());
IndexWriter writer = new IndexWriter(dir, config);
return writer;
}
}

Accept Multipart file upload as camel restlet or cxfrs endpoint

I am looking to implement a route where reslet/cxfrs end point will accept file as multipart request and process. (Request may have some JSON data as well.
Thanks in advance.
Regards.
[EDIT]
Have tried following code. Also tried sending file using curl. I can see file related info in headers and debug output, but not able to retrieve attachment.
from("servlet:///hello").process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
Message in = exchange.getIn();
StringBuffer v = new StringBuffer();
HttpServletRequest request = (HttpServletRequest) in
.getHeaders().get(Exchange.HTTP_SERVLET_REQUEST);
DiskFileItemFactory diskFile = new DiskFileItemFactory();
FileItemFactory factory = diskFile;
ServletFileUpload upload = new ServletFileUpload(factory);
List items = upload.parseRequest(request);
.....
curl :
curl -vvv -i -X POST -H "Content-Type: multipart/form-data" -F "image=#/Users/navaltiger/1.jpg; type=image/jpg" http://:8080/JettySample/camel/hello
following code works (but can't use as it embeds jetty, and we would like to deploy it on tomcat/weblogic)
public void configure() throws Exception {
// getContext().getProperties().put("CamelJettyTempDir", "target");
getContext().setStreamCaching(true);
getContext().setTracing(true);
from("jetty:///test").process(new Processor() {
// from("servlet:///hello").process(new Processor() {
public void process(Exchange exchange) throws Exception {
String body = exchange.getIn().getBody(String.class);
HttpServletRequest request = exchange.getIn().getBody(
HttpServletRequest.class);
StringBuffer v = new StringBuffer();
// byte[] picture = (request.getParameter("image")).getBytes();
v.append("\n Printing All Request Parameters From HttpSerlvetRequest: \n+"+body +" \n\n");
Enumeration<String> requestParameters = request
.getParameterNames();
while (requestParameters.hasMoreElements()) {
String paramName = (String) requestParameters.nextElement();
v.append("\n Request Paramter Name: " + paramName
+ ", Value - " + request.getParameter(paramName));
}
I had a similar problem and managed to resolve inspired by the answer of brentos. The rest endpoint in my case is defined via xml:
<restContext id="UploaderServices" xmlns="http://camel.apache.org/schema/spring">
<rest path="/uploader">
<post bindingMode="off" uri="/upload" produces="application/json">
<to uri="bean:UploaderService?method=uploadData"/>
</post>
</rest>
</restContext>
I had to use "bindingMode=off" to disable xml/json unmarshalling because the HttpRequest body contains multipart data (json/text+file) and obviously the standard unmarshaling process was unable to process the request because it's expecting a string in the body and not a multipart payload.
The file and other parameters are sent from a front end that uses the file upload angular module: https://github.com/danialfarid/ng-file-upload
To solve CORS problems I had to add a CORSFilter filter in the web.xml like the one here:
public class CORSFilter implements Filter {
#Override
public void doFilter(ServletRequest req, ServletResponse resp, FilterChain chain) throws IOException,
ServletException {
HttpServletResponse httpResp = (HttpServletResponse) resp;
HttpServletRequest httpReq = (HttpServletRequest) req;
httpResp.setHeader("Access-Control-Allow-Methods", "GET, HEAD, POST, PUT, DELETE, TRACE, OPTIONS, CONNECT, PATCH");
httpResp.setHeader("Access-Control-Allow-Origin", "*");
if (httpReq.getMethod().equalsIgnoreCase("OPTIONS")) {
httpResp.setHeader("Access-Control-Allow-Headers",
httpReq.getHeader("Access-Control-Request-Headers"));
}
chain.doFilter(req, resp);
}
#Override
public void init(FilterConfig arg0) throws ServletException {
}
#Override
public void destroy() {
}
}
Also, I had to modify a little bit the unmarshaling part:
public String uploadData(Message exchange) {
String contentType=(String) exchange.getIn().getHeader(Exchange.CONTENT_TYPE);
MediaType mediaType = MediaType.valueOf(contentType); //otherwise the boundary parameter is lost
InputRepresentation representation = new InputRepresentation(exchange
.getBody(InputStream.class), mediaType);
try {
List<FileItem> items = new RestletFileUpload(
new DiskFileItemFactory())
.parseRepresentation(representation);
for (FileItem item : items) {
if (!item.isFormField()) {
InputStream inputStream = item.getInputStream();
// Path destination = Paths.get("MyFile.jpg");
// Files.copy(inputStream, destination,
// StandardCopyOption.REPLACE_EXISTING);
System.out.println("found file in request:" + item);
}else{
System.out.println("found string in request:" + new String(item.get(), "UTF-8"));
}
}
} catch (Exception e) {
e.printStackTrace();
}
return "200";
}
I'm using the Camel REST DSL with Restlet and was able to get file uploads working with the following code.
rest("/images").description("Image Upload Service")
.consumes("multipart/form-data").produces("application/json")
.post().description("Uploads image")
.to("direct:uploadImage");
from("direct:uploadImage")
.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
MediaType mediaType =
exchange.getIn().getHeader(Exchange.CONTENT_TYPE, MediaType.class);
InputRepresentation representation =
new InputRepresentation(
exchange.getIn().getBody(InputStream.class), mediaType);
try {
List<FileItem> items =
new RestletFileUpload(
new DiskFileItemFactory()).parseRepresentation(representation);
for (FileItem item : items) {
if (!item.isFormField()) {
InputStream inputStream = item.getInputStream();
Path destination = Paths.get("MyFile.jpg");
Files.copy(inputStream, destination,
StandardCopyOption.REPLACE_EXISTING);
}
}
} catch (FileUploadException | IOException e) {
e.printStackTrace();
}
}
});
you can do this with restdsl even if you are not using restlet (exemple jetty) for your restdsl component.
you need to turn restdinding of first for that route and reate two classes to handle the multipart that is in your body.
you need two classes :
DWRequestContext
DWFileUpload
and then you use them in your custom processor
here is the code :
DWRequestContext.java
import org.apache.camel.Exchange;
import org.apache.commons.fileupload.RequestContext;
import java.io.IOException;
import java.io.InputStream;
import java.nio.charset.StandardCharsets;
public class DWRequestContext implements RequestContext {
private Exchange exchange;
public DWRequestContext(Exchange exchange) {
this.exchange = exchange;
}
public String getCharacterEncoding() {
return StandardCharsets.UTF_8.toString();
}
//could compute here (we have stream cache enabled)
public int getContentLength() {
return (int) -1;
}
public String getContentType() {
return exchange.getIn().getHeader("Content-Type").toString();
}
public InputStream getInputStream() throws IOException {
return this.exchange.getIn().getBody(InputStream.class);
}
}
DWFileUpload.java
import org.apache.camel.Exchange;
import org.apache.commons.fileupload.FileItem;
import org.apache.commons.fileupload.FileItemFactory;
import org.apache.commons.fileupload.FileUpload;
import org.apache.commons.fileupload.FileUploadException;
import java.util.List;
public class DWFileUpload extends
FileUpload {
public DWFileUpload() {
super();
}
public DWFileUpload(FileItemFactory fileItemFactory) {
super(fileItemFactory);
}
public List<FileItem> parseInputStream(Exchange exchange)
throws FileUploadException {
return parseRequest(new DWRequestContext(exchange));
}
}
you can define your processor like this:
routeDefinition.process(new Processor() {
#Override
public void process(Exchange exchange) throws Exception {
// Create a factory for disk-based file items
DiskFileItemFactory factory = new DiskFileItemFactory();
factory.setRepository(new File(System.getProperty("java.io.tmpdir")));
DWFileUpload upload = new DWFileUpload(factory);
java.util.List<FileItem> items = upload.parseInputStream(exchange);
//here I assume I have only one, but I could split it here somehow and link them to camel properties...
//with this, the first file sended with your multipart replaces the body
// of the exchange for the next processor to handle it
exchange.getIn().setBody(items.get(0).getInputStream());
}
});
I stumbled into the same requirement of having to consume a multipart request (containing file data including binary) through Apache Camel Restlet component.
Even though 2.17.x is out, since my project was part of a wider framework / application, I had to be using version 2.12.4.
Initially, my solution drew a lot from restlet-jdbc example yielded data in exchange that although was successfully retrieving text files but I was unable to retrieve correct binary content.
I attempted to dump the data directly into a file to inspect the content using following code (abridged).
from("restlet:/upload?restletMethod=POST")
.to("direct:save-files");
from("direct:save-files")
.process(new org.apache.camel.Processor(){
public void process(org.apache.camel.Exchange exchange){
/*
* Code to sniff exchange content
*/
}
})
.to("file:///C:/<path to a folder>");
;
I used org.apache.commons.fileupload.MultipartStream from apache fileuplaod library to write following utility class to parse Multipart request from a file. It worked successfully when the output of a mulitpart request from Postman was fed to it. However, failed to parse content of the file created by Camel (even through to eyes content of both files looked similar).
public class MultipartParserFileCreator{
public static final String DELIMITER = "\\r?\\n";
public static void main(String[] args) throws Exception {
// taking it from the content-type in exchange
byte[] boundary = "------5lXVNrZvONBWFXxd".getBytes();
FileInputStream fis = new FileInputStream(new File("<path-to-file>"));
extractFile(fis, boundary);
}
public static void extractFile(InputStream is, byte[] boundary) throws Exception {
MultipartStream multipartStream = new MultipartStream(is, boundary, 1024*4, null);
boolean nextPart = multipartStream.skipPreamble();
while (nextPart) {
String headers = multipartStream.readHeaders();
if(isFileContent(headers)) {
String filename = getFileName(headers);
File file = new File("<dir-where-file-created>"+filename);
if(!file.exists()) {
file.createNewFile();
}
FileOutputStream fos = new FileOutputStream(file);
multipartStream.readBodyData(fos);
fos.flush();
fos.close();
}else {
multipartStream.readBodyData(System.out);
}
nextPart = multipartStream.readBoundary();
}
}
public static String[] getContentDispositionTokens(String headersJoined) {
String[] headers = headersJoined.split(DELIMITER, -1);
for(String header: headers) {
System.out.println("Processing header: "+header);
if(header != null && header.startsWith("Content-Disposition:")) {
return header.split(";");
}
}
throw new RuntimeException(
String.format("[%s] header not found in supplied headers [%s]", "Content-Disposition:", headersJoined));
}
public static boolean isFileContent(String header) {
String[] tokens = getContentDispositionTokens(header);
for (String token : tokens) {
if (token.trim().startsWith("filename")) {
return true;
}
}
return false;
}
public static String getFileName(String header) {
String[] tokens = getContentDispositionTokens(header);
for (String token : tokens) {
if (token.trim().startsWith("filename")) {
String filename = token.substring(token.indexOf("=") + 2, token.length()-1);
System.out.println("fileName is " + filename);
return filename;
}
}
return null;
}
}
On debugging through the Camel code, I noticed that at one stage Camel is converting the entire content into String. After a point I had to stop pursuing this approach as there was very little on net applicable for version 2.12.4 and my work was not going anywhere.
Finally, I resorted to following solution
Write an implementation of HttpServletRequestWrapper to allow
multiple read of input stream. One can get an idea from
How to read request.getInputStream() multiple times
Create a filter that uses the above to wrap HttpServletRequest object, reads and extract the file to a directory Convenient way to parse incoming multipart/form-data parameters in a Servlet and attach the path to the request using request.setAttribute() method. With web.xml, configure this filter on restlet servlet
In the process method of camel route, type cast the
exchange.getIn().getBody() in HttpServletRequest object, extract the
attribute (path) use it to read the file as ByteStreamArray for
further processing
Not the cleanest, but I could achieve the objective.

Solrj client and XML response

I am using solr4.0 in jetty server. I want to query solr using solrj and expecting results to be formatted in XML. So i used HttpSolrServer (CloudSolrServer and LBHttpSolrServer does not provide support for setting parser) and i set parser to Xmlparser. Moreover i am also setting SolrQuery param wt=xml.But i am not able to get results in XML.Here is my test code
package solrjtest;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.util.UUID;
import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.SolrServerException;
import org.apache.solr.client.solrj.impl.CommonsHttpSolrServer;
import org.apache.solr.client.solrj.impl.XMLResponseParser;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocumentList;
class SolrjTest
{
public static void main(String[] args) throws IOException, SolrServerException
{
SolrjTest solrj = new SolrjTest();
solrj.query("hello");
}
public void query(String q) throws IOException, SolrServerException
{
CommonsHttpSolrServer server = null;
String uuid = null;
boolean flag = true;
while (flag == true)
{
uuid = UUID.randomUUID().toString();
File f = new File("D:/SearchResult/" + uuid + ".txt");
if (!f.exists())
{
flag=false;
f.createNewFile();
}
}
try
{
server = new CommonsHttpSolrServer("http://skyfall:8983/solr/documents");
server.setParser(new XMLResponseParser());
}
catch (Exception e)
{
e.printStackTrace();
}
SolrQuery query = new SolrQuery();
query.setQuery(q);
query.setParam("wt", "xml");
FileWriter fw = new FileWriter("D:/SearchResult/" + uuid + ".txt");
try
{
QueryResponse qr = server.query(query);
SolrDocumentList sdl = qr.getResults();
XMLResponseParser r = new XMLResponseParser();
Object[] o = new Object[sdl.size()];
o = sdl.toArray();
for (int i = 0; i < o.length; i++)
{
System.out.println(o[i].toString());
fw.write(o[i].toString() + "\n");
}
fw.flush();
fw.close();
System.out.println("finished");
}
catch (SolrServerException e)
{
e.printStackTrace();
}
}
}
Any idea whats going wrong here ?
With that setup, the Solr server at the machine skyfall does send the response in XML and the CommonsHttpSolrServer wrapper does correctly parse the XML. However, that does not change the internal representation in the QueryResponse, which is just a thin wrapper around the Solr class NamedList.
You can (mis)use the XMLResponseWriter to get an XML representation of the full QueryResponse:
private String toXML(SolrParams request, QueryResponse response) {
XMLResponseWriter xmlWriter = new XMLResponseWriter();
Writer w = new StringWriter();
SolrQueryResponse sResponse = new SolrQueryResponse();
sResponse.setAllValues(response.getResponse());
try {
xmlWriter.write(w, new LocalSolrQueryRequest(null, request), sResponse);
} catch (IOException e) {
throw new RuntimeException("Unable to convert Solr response into XML", e);
}
return w.toString();
}

Resources