I have written Solr Custom Search Component as described here
My goal is to update the query parameters, specifically to remove Unicode Quotes as early as possible in the pipeline.
However, after intercepting the request and editing the parameters, the request does not seem to update.
public void updateSolrRequest(SolrQueryRequest req) {
SolrParams params = req.getParams();
System.out.println( "params = " + req.getParamString());
String newQuery = params.get(CommonParams.Q);
newQuery = newQuery.toString().replaceAll("[A]","XXX");
ModifiableSolrParams newParams = new ModifiableSolrParams(params);
newParams.remove(CommonParams.Q);
newParams.add(CommonParams.Q, newQuery);
// all good to here, the next line should
// overwrite the old params with the new ones
// but it does not
req.setParams(newParams);
System.out.println("newQuery = " + newQuery);
System.out.println("newParams = " + newParams.get(CommonParams.Q));
System.out.println("updated req = " + req.getParamString());
}
Output
params = q=“A+Game+of+Thrones“&defType=dismax&q.alt=thrones&fq=Game&_=1548262845155
newQuery = “XXX Game of Thrones“
newParams = “XXX Game of Thrones“
updated req = q=“A+Game+of+Thrones“&defType=dismax&q.alt=thrones&fq=Game&_=1548262845155
The problem here is, that
public String getParamString() {
return origParams.toString();
}
is actually returning original params, which aren't affected by setParams called
/** Change the parameters for this request. This does not affect
* the original parameters returned by getOriginalParams()
*/
void setParams(SolrParams params);
You should use org.apache.solr.request.SolrQueryRequest#getParams to check your updated parameters.
Related
I am creating an external stage, I want it to be based on 2 URLs.
PROBLEM 1
url1 = s3://bucket1/f1/2022/2/
url2 = s3://bucket1/f3/2022/2/
create or replace stage ext_stage url = ??????????
file_format=data_format
storage_integration=s3_integration;
How can I give 2 URLs in the external stage command? Is it possible?
PROBLEM 2
Also, I need to form the URLs.
I am thinking to use a procedure for it.
CREATE OR REPLACE PROCEDURE get_url()
RETURNS STRING
LANGUAGE SCALA
RUNTIME_VERSION = '2.12'
HANDLER = 'C.run'
PACKAGES = ('com.snowflake:snowpark:latest')
AS
$$
import java.util.Calendar
object C{
def run(session: com.snowflake.snowpark.Session): String = {
try {
val cal = Calendar.getInstance()
val year =cal.get(Calendar.YEAR)
var month =cal.get(Calendar.MONTH) + 1
return "s3://bucket1/folder1/" + year + "/"+ month + "/"
}
catch {
case e: Throwable => println("Not able to for url")
return "Failed"
}
}
}
$$;
create or replace stage ext_stage url = call get_url()
file_format=data_format
storage_integration=s3_integration;
It is failing and I can not call the function. How can I call it?
I want to return a List of "Posts" from an endpoint with optional pagination.
I need 100 results per query.
The Code i have written is as follows, it doesn't seem to work.
I am referring to an example at Objectify Wiki
Another option i know of is using query.offset(100);
But i read somewhere that this just loads the entire table and then ignores the first 100 entries which is not optimal.
I guess this must be a common use case and an optimal solution will be available.
public CollectionResponse<Post> getPosts(#Nullable #Named("cursor") String cursor,User auth) throws OAuthRequestException {
if (auth!=null){
Query<Post> query = ofy().load().type(Post.class).filter("isReviewed", true).order("-timeStamp").limit(100);
if (cursor!=null){
query.startAt(Cursor.fromWebSafeString(cursor));
log.info("Cursor received :" + Cursor.fromWebSafeString(cursor));
} else {
log.info("Cursor received : null");
}
QueryResultIterator<Post> iterator = query.iterator();
for (int i = 1 ; i <=100 ; i++){
if (iterator.hasNext()) iterator.next();
else break;
}
log.info("Cursor generated :" + iterator.getCursor());
return CollectionResponse.<Post>builder().setItems(query.list()).setNextPageToken(iterator.getCursor().toWebSafeString()).build();
} else throw new OAuthRequestException("Login please.");
}
This is a code using Offsets which seems to work fine.
#ApiMethod(
name = "getPosts",
httpMethod = ApiMethod.HttpMethod.GET
)
public CollectionResponse<Post> getPosts(#Nullable #Named("offset") Integer offset,User auth) throws OAuthRequestException {
if (auth!=null){
if (offset==null) offset = 0;
Query<Post> query = ofy().load().type(Post.class).filter("isReviewed", true).order("-timeStamp").offset(offset).limit(LIMIT);
log.info("Offset received :" + offset);
log.info("Offset generated :" + (LIMIT+offset));
return CollectionResponse.<Post>builder().setItems(query.list()).setNextPageToken(String.valueOf(LIMIT + offset)).build();
} else throw new OAuthRequestException("Login please.");
}
Be sure to assign the query:
query = query.startAt(cursor);
Objectify's API uses a functional style. startAt() does not mutate the object.
Try the following:
Remove your for loop -- not sure why it is there. But just iterate through your list and build out the list of items that you want to send back. You should stick to the iterator and not force it for 100 items in a loop.
Next, once you have iterated through it, use the iterator.getStartCursor() as the value of the cursor.
I am trying to use a cursor with Objectify and Google App Engine to return a subset of data and a cursor so that I can retrieve more data when the user is ready. I found an example here that looks exactly like what I need but I don't know how to return the final list plus the cursor. Here is the code I have:
#ApiMethod(name = "listIconThemeCursor") //https://code.google.com/p/objectify-appengine/wiki/Queries#Cursors
public CollectionResponse<IconTheme> listIconThemeCursor(#Named("cursor") String cursorStr) {
Query<IconTheme> query = ofy().load().type(IconTheme.class).limit(10);
if (cursorStr != null ) {
query.startAt(Cursor.fromWebSafeString(cursorStr));
}
List<IconTheme> result = new ArrayList<IconTheme>();
int count = 0;
QueryResultIterator<IconTheme> iterator = query.iterator();
while (iterator.hasNext()) {
IconTheme theme = iterator.next();
result.add(theme);
count++;
}
Cursor cursor = iterator.getCursor();
String encodeCursor = cursor.toWebSafeString();
return serial(tClass, result, encodeCursor);
}
Note that this was modified from a previous endpoint in which I returned the CollectionResponse of ALL the data. My dataset is large enough that this is no longer practical. Basically, I don't know what was in the user's function of 'serial(tClass, result, encodeCursor) that let it get returned to the user.
There is another example here but it doesn't appear to answer my question either.
I don't quite understand what you are asking, but I see one immediate bug in your code:
query.startAt(Cursor.fromWebSafeString(cursorStr));
...should be:
query = query.startAt(Cursor.fromWebSafeString(cursorStr));
Objectify command objects are immutable, functional objects.
After a long slog, I figured out that CollectionResponse has the cursor in it :(
Here is the complete code I used incorporating the comment from stickfigure above:
#ApiMethod(name = "listIconThemeCursor", path="get_cursor")
public CollectionResponse<IconTheme> listIconThemeCursor(#Named("cursor") String cursorStr) {
Query<IconTheme> query = ofy().load().type(IconTheme.class)
.filter("errors <", 10)
.limit(10);
if (cursorStr != null ) {
query = query.startAt(Cursor.fromWebSafeString(cursorStr));
}
List<IconTheme> result = new ArrayList<IconTheme>();
QueryResultIterator<IconTheme> iterator = query.iterator();
while (iterator.hasNext()) {
IconTheme theme = iterator.next();
result.add(theme);
}
Cursor cursor = iterator.getCursor();
CollectionResponse<IconTheme> response = CollectionResponse.<IconTheme> builder()
.setItems(result)
.setNextPageToken(cursor.toWebSafeString())
.build();
return response;
}
I get files from google drive, like that. The result of Folders. ok,
I have List result, so I want to find if result(i) have children with application/vnd.google-apps.folder mime type or not.
in the other words, method which checks: if current folder contains children folder returns true. owherwise returns false;
I use it in google App engine.
how can I do that?
public void getAllFiles(String id, Drive service) throws IOException{
String query="'"+id + "'"+ " in parents and trashed=false and mimeType=application/vnd.google-apps.folder'";
FileList files = service.files().list().setQ(query).execute();
List<File> result = new ArrayList<File>();
Files.List request = service.files().list();
do {
result.addAll(files.getItems());
request.setPageToken(files.getNextPageToken());
} while (request.getPageToken() != null && request.getPageToken().length() > 0);
}
I do not want to create additional requests (new children request to each node) to see if it have children or not. something like that:
for (File file : result) {
query = "'" + file.getId() + "'" + " in parents and trashed=false and mimeType='application/vnd.google-apps.folder'";
FileList fileChild = service.files().list().setQ(query).execute();
List<File> res = new ArrayList<File>();
Files.List req = service.files().list();
do {
res.addAll(fileChild.getItems());
req.setPageToken(fileChild.getNextPageToken());
} while (req.getPageToken() != null && req.getPageToken().length() > 0);
RemoteFolder remFolder = new RemoteFolder();
remFolder.setFile(file);
if(res.size()>0){
remFolder.setChildrenable(true);
log.info("folder named "+file.getTitle() + " have children folders count: "+ res.size());
}
folderList.add(remFolder);
}
I want to optimize my code.
You have to make another request for each node. If you want to avoid too many calls use memcached.
I'm working on a project where i need to post the data i acquire to a Google form and obtain the data from the spreadsheet. I cannot use google apps script and need a method using the direct POST method as i will be doing this function from a GSM module. All the solutions posted previously take into consideration the old structure of the Google form which provides a form key.Like the solution described in this one:
http://www.open-electronics.org/how-send-data-from-arduino-to-google-docs-spreadsheet/
The link to my current form is this.
https://docs.google.com/forms/d/14MkYG3fPNezzUC_nXUsWHlZ5JhplvjyWTAeob7f_W7g/viewform
Any help would be appreciated.
Is it a requirement that a google form be in the middle of this? If it is enough to be able to post your data to a spreadsheet, here's a Google-Apps-Script for one side of the problem: a simple web service that will accept form data as a query string, and write that to your spreadsheet.
This examples assumes a very simple spreadsheet, with three columns, "Timestamp", "col1" and "col2". Edit the code to suit your situation.
You can see the spreadsheet here, and even make a test post.
/**
* doGet() function to add data to a spreadsheet.
*
* Spreadsheet data is provided as a querystring, e.g. ?col1=1&col2='pizza'
*
* From: http://stackoverflow.com/a/18725479/1677912
*
* #param {event} e Event passed to doGet, with querystring
* #returns {String/html} Html to be served
*
* Test URLs (adjust ID as needed):
* https://script.google.com/macros/s/--DEV-SCRIPT-ID--/dev?col1=1&col2='pizza'
* https://script.google.com/macros/s/--PUB-SCRIPT-ID--/exec?col1=1&col2='pizza'
*/
function doGet(e) {
Logger.log( JSON.stringify(e) ); // view parameters
var result = 'Ok'; // assume success
if (e.parameter == undefined) {
result = 'No Parameters';
}
else {
var id = '--SHEET-ID---'; // Spreadsheet id for responses
var sheet = SpreadsheetApp.openById(id).getActiveSheet();
var newRow = sheet.getLastRow() + 1;
var rowData = [];
rowData[0] = new Date(); // Timestamp
for (var param in e.parameter) {
Logger.log('In for loop, param='+param);
var value = stripQuotes(e.parameter[param]);
//Logger.log(param + ':' + e.parameter[param]);
switch (param) {
case 'col1':
rowData[1] = value;
break;
case 'col2':
rowData[2] = value;
break;
default:
result = "unsupported parameter";
}
}
Logger.log(JSON.stringify(rowData));
// Write new row to spreadsheet
var newRange = sheet.getRange(newRow, 1, 1, rowData.length);
newRange.setValues([rowData]);
}
// Return result of operation
return ContentService.createTextOutput(result);
}
/**
* Remove leading and trailing single or double quotes
*/
function stripQuotes( value ) {
return value.replace(/^["']|['"]$/g, "");
}
You can do the sending with the new forms, there is a menu option for it. (Responses->Get prefill url) It gives the url for posting data to a form.
You also asked: "obtain the data from the spreadsheet":There are two ways, google apps script and gdata style "google-spreadsheet-api". But I suggest you use a mix of google apps script and "arduino" style code, as it has better docs and features than gdata style api.
p.s. I created some formulas for creating an "arduino" user interface a while back.