How to sort a resultset by date (unixtime)?
search post {
document post {
field created type long {
indexing: summary | attribute
}
field description type string {
indexing: summary | index
}
}
rank-profile date inherits default {
first-phase {
expression: attribute(created)
}
}
}
curl:
curl -s -H "Content-Type: application/json" --data
'{"yql" : "select * from post where description contains \"computer\";","ranking":"date"}'
http://localhost:8080/search/ | jq .
The resultset is not sorted by 'created'.
And 'relevance' is always zero:
{
"id": "id:post:post::1",
"relevance": 0,
"source": "content",
"fields": {...}
}
For straight forward sorting on long attributes, it would be more efficient to use the sorting/ordering functionality rather than the more powerful, yet more expensive ranking framework.
As mentioned in the sorting documentation, it is recommended to use the built-in unranked ranking profile for queries with sorting/ordering. Also, I'm not sure the ranking alias is allowed when using the JSON query language - I believe you would need to use the full ranking.profile parameter which is nested in JSON.
Your curl would then look something like:
curl -s -H "Content-Type: application/json" --data
'{"yql" : "select * from post where description contains \"computer\" order by created desc;","ranking": { "profile" : "unranked" } }'
http://localhost:8080/search/ | jq .
I am trying to delete index in Solr. When i run XML query it deletes, but if i try JSON query it gives success message but it does not delete and weirdly it increase count by 1 by adding new row:
XML:
<delete><query>id:5a048bb5-f661-4517-a55e-e15e663c2cd7</query></delete>
JSON:
{
"delete": {
"query": "id:5a048bb5-f661-4517-a55e-e15e663c2cd7"
}
}
I have JSONs in the below format in a S3 bucket and I'm trying to extract only the "id", "label" & "value" from the "fields" key using Athena. I tried ARRAY-MAP but wasn't successful. Also, on the "value" field - I want the content to be captured as a simple text ignoring any list / dictionaries in it.
I also don't want to create any Hive schema for these JSONs and looking for a Presto SQL solution if possible.
{
"reports":{
"client":{
"pdf":"https://reports.s3-accelerate.amazonaws.com/looks/123/reports/client.pdf",
"html":"https://api.com/looks/123/reports/client.html"
},
"public":{
"pdf":"https://s3.amazonaws.com/reports.com/looks/123/reports/public.pdf",
"html":"https://api.look.com/looks/123/reports/public.html"
}
},
"actors":{
"looker":{
"firstName":"Rosa",
"lastName":"Mart"
},
"client":{
"email":"XXX.XXX#XXXXXX.com",
"firstName":"XXX",
"lastName":"XXX"
}
},
"_id":"123",
"fields":[
{
"id":"fence_condition_missing_sections",
"context":[
"Fence Condition"
],
"label":"Missing Sections",
"type":"choice",
"value":"None"
},
{
"id":"photos_landscaped_area",
"context":[
"Landscaping Photos"
],
"label":"Landscaped Area",
"type":"photo-with-description",
"value":[
{
"description":"Front",
"photo":"https://reports-wegolook-com.s3-accelerate.amazonaws.com/looks/123/looker/1.jpg"
},
{
"description":"Front entrance ",
"photo":"https://reports-wegolook-com.s3-accelerate.amazonaws.com/looks/123/looker/2.jpg"
}
]
}
],
"jobNumber":"xxx",
"createdAt":"2018-10-11T22:39:37.223Z",
"completedAt":"2018-01-27T20:13:49.937Z",
"inspectedAt":"2018-01-21T23:33:48.718Z",
"type":"ZZZ-commercial",
"name":"Commercial"
}'
expected output:
--------------------------------------------------------------------------------
| ID | LABEL | VALUE |
--------------------------------------------------------------------------------
| photos_landscaped_area | Landscaped Area | [{"description":"Front",...}] |
----------------------------------------------------------------------------
| fence_condition_missing_sections | Missing Sections | None|
----------------------------------------------------------------------------
I'm going to assume your data is in a one-document-per-line format and that you provided a formatted example for readability's sake. If this is incorrect, please see the question Multi-line JSON file querying in hive
.
When the schema of a JSON document is not entirely regular you can create that column as a string column and use the JSON_* functions to extract values out of it.
First you need to create a table for the raw data:
CREATE TABLE data (
fields array<struct<id:string,label:string,value:string>>
)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
LOCATION 's3://…'
(if you're not interested in the other fields in the JSON documents you can just ignore those when creating the table)
Then you create a view that flattens the data:
CREATE VIEW flat_data AS
SELECT
field.id,
field.label,
field.value
FROM data
CROSS JOIN UNNEST(fields) AS f(field)
Selecting from this view should give you the results you are looking for.
I suspect you are also looking for how to extract properties from the values structure, which is what I alluded to above:
SELECT
label,
JSON_EXTRACT(value, '$.photo') AS photo_urls
FROM flat_data
WHERE id = 'photos_landscaped_area'
Look in the Presto documentation for all available JSON functions.
Question: how do I export from mongo such that I can import into SQL Server if I use $unwind?
I need to use $unwind which means I can't use mongoexport.exe. Mongo.exe gives different output for json as shown below. Output I can't load into SQL Server. I would export as csv output, but my data includes commas. I would use $out to first copy my data to a new collection & then use mongoexport, but I'm querying a production server in the cloud where I only have read access.
To illustrate my problem, I created a collection with one record that has a date field "edited_on". You can see here that mongoexport output starts with ["_id:{$oid.... while mongo output starts with {"_id : ObjectID(….
*** MONGOEXPORT
The command:
mongoexport --quiet --host localhost:27017 --db "zzz" -c
"Test_Structures" --fields edited_on --type json --jsonArray --out
C:\export_test.json
The output:
[{"_id":{"$oid":"5aaa1d85b8078250f1000c0e"},"edited_on":{"$date":"2018-03-15T07:15:17.583Z"}}]
I can import this data into SQL with OPENROWSET along with OPENJSON.
Described here: https://www.mssqltips.com/sqlservertip/5295/different-ways-to-import-json-files-into-sql-server/
*** MONGO
The command:
mongo localhost/UW --quiet -eval "db.Test_Structures.aggregate( {
$project: { _id: 1 , edited_on: 1} } )" > C:\aggregate_test.json
The output:
{ "_id" : ObjectId("5aaa1d85b8078250f1000c0e"), "edited_on" :
ISODate("2018-03-15T07:15:17.583Z") }
Declare #JSON varchar(max)
My coworker answered my question. Use replace() to remove the text in the json file that are causing problems as follows.
SELECT #JSON = BulkColumn
FROM OPENROWSET (BULK 'C:\aggregate_test.json', SINGLE_CLOB) as j
SET #JSON = replace(replace(replace(#JSON,'objectid(',''),'isodate(',''),'")','"')
SELECT * FROM OPENJSON (#JSON) With (...)
How do I delete all data from solr by command? We are using solr with lily and hbase.
How can I delete data from both hbase and solr?
http://lucene.apache.org/solr/4_10_0/tutorial.html#Deleting+Data
If you want to clean up Solr index -
you can fire http url -
http://host:port/solr/[core name]/update?stream.body=<delete><query>*:*</query></delete>&commit=true
(replace [core name] with the name of the core you want to delete from). Or use this if posting data xml data:
<delete><query>*:*</query></delete>
Be sure you use commit=true to commit the changes
Don't have much idea with clearing hbase data though.
I've used this request to delete all my records but sometimes it's necessary to commit this.
For that, add &commit=true to your request :
http://host:port/solr/core/update?stream.body=<delete><query>*:*</query></delete>&commit=true
Post json data (e.g. with curl)
curl -X POST -H 'Content-Type: application/json' \
'http://<host>:<port>/solr/<core>/update?commit=true' \
-d '{ "delete": {"query":"*:*"} }'
You can use the following commands to delete.
Use the "match all docs" query in a delete by query command:
'<delete><query>*:*</query></delete>
You must also commit after running the delete so, to empty the index, run the following two commands:
curl http://localhost:8983/solr/update --data '<delete><query>*:*</query></delete>' -H 'Content-type:text/xml; charset=utf-8'
curl http://localhost:8983/solr/update --data '<commit/>' -H 'Content-type:text/xml; charset=utf-8'
Another strategy would be to add two bookmarks in your browser:
http://localhost:8983/solr/update?stream.body=<delete><query>*:*</query></delete>
http://localhost:8983/solr/update?stream.body=<commit/>
Source docs from SOLR:
https://wiki.apache.org/solr/FAQ#How_can_I_delete_all_documents_from_my_index.3F
If you want to delete all of the data in Solr via SolrJ do something like this.
public static void deleteAllSolrData() {
HttpSolrServer solr = new HttpSolrServer("http://localhost:8080/solr/core/");
try {
solr.deleteByQuery("*:*");
} catch (SolrServerException e) {
throw new RuntimeException("Failed to delete data in Solr. "
+ e.getMessage(), e);
} catch (IOException e) {
throw new RuntimeException("Failed to delete data in Solr. "
+ e.getMessage(), e);
}
}
If you want to delete all of the data in HBase do something like this.
public static void deleteHBaseTable(String tableName, Configuration conf) {
HBaseAdmin admin = null;
try {
admin = new HBaseAdmin(conf);
admin.disableTable(tableName);
admin.deleteTable(tableName);
} catch (MasterNotRunningException e) {
throw new RuntimeException("Unable to delete the table " + tableName
+ ". The actual exception is: " + e.getMessage(), e);
} catch (ZooKeeperConnectionException e) {
throw new RuntimeException("Unable to delete the table " + tableName
+ ". The actual exception is: " + e.getMessage(), e);
} catch (IOException e) {
throw new RuntimeException("Unable to delete the table " + tableName
+ ". The actual exception is: " + e.getMessage(), e);
} finally {
close(admin);
}
}
Use the "match all docs" query in a delete by query command: :
You must also commit after running the delete so, to empty the index, run the following two commands:
curl http://localhost:8983/solr/update --data '<delete><query>*:*</query></delete>' -H 'Content-type:text/xml; charset=utf-8'
curl http://localhost:8983/solr/update --data '<commit/>' -H 'Content-type:text/xml; charset=utf-8'
I came here looking to delete all documents from solr instance through .Net framework using SolrNet. Here is how I was able to do it:
Startup.Init<MyEntity>("http://localhost:8081/solr");
ISolrOperations<MyEntity> solr =
ServiceLocator.Current.GetInstance<ISolrOperations<MyEntity>>();
SolrQuery sq = new SolrQuery("*:*");
solr.Delete(sq);
solr.Commit();
This has cleared all the documents. (I am not sure if this could be recovered, I am in learning and testing phase of Solr, so please consider backup before using this code)
From the command line use:
bin/post -c core_name -type text/xml -out yes -d $'<delete><query>*:*</query></delete>'
fire this in the browser
http://localhost:8983/solr/update?stream.body=<delete><query>*:*</query></delete>&commit=true
this commmand will delete all the documents in index in solr
I've used this query to delete all my records.
http://host/solr/core-name/update?stream.body=%3Cdelete%3E%3Cquery%3E*:*%3C/query%3E%3C/delete%3E&commit=true
To delete all documents of a Solr collection, you can use this request:
curl -X POST -H 'Content-Type: application/json' --data-binary '{"delete":{"query":"*:*" }}' http://localhost:8983/solr/my_collection/update?commit=true
It uses the JSON body.
The curl examples above all failed for me when I ran them from a cygwin terminal. There were errors like this when i ran the script example.
curl http://192.168.2.20:7773/solr/CORE1/update --data '<delete><query>*:*</query></delete>' -H 'Content-type:text/xml; charset=utf-8'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst>
</response>
<!--
It looks like it deleted stuff, but it did not go away
maybe because the committing call failed like so
-->
curl http://192.168.1.2:7773/solr/CORE1/update --data-binary '' -H 'Content-type:text/xml; charset=utf-8'
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">400</int><int name="QTime">2</int></lst><lst name="error"><str name="msg">Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]</str><int name="code">400</int></lst>
</response>
I needed to use the delete in a loop on core names to wipe them all out in a project.
This query below worked for me in the Cygwin terminal script.
curl http://192.168.1.2:7773/hpi/CORE1/update?stream.body=<delete><query>*:*</query></delete>&commit=true
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int name="QTime">1</int></lst>
</response>
This one line made the data go away and the change persisted.
I tried the below steps. It works well.
Please make sure the SOLR server it running
Just click the link Delete all SOLR data which will hit and delete all your SOLR indexed datas then you will get the following details on the screen as output.
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">494</int>
</lst>
</response>
if you are not getting the above output then please make sure the following.
I used the default host (localhost) and port (8080) on the above link. please alter the host and port if it is different in your end.
The default core name should be collection / collection1. I used collection1 in the above link. please change it too if your core name is different.
If you need to clean out all data, it might be faster to recreate collection, e.g.
solrctl --zk localhost:2181/solr collection --delete <collectionName>
solrctl --zk localhost:2181/solr collection --create <collectionName> -s 1
I made a JavaScript bookmark which adds the delete link in Solr Admin UI
javascript: (function() {
var str, $a, new_href, href, upd_str = 'update?stream.body=<delete><query>*:*</query></delete>&commit=true';
$a = $('#result a#url');
href = $a.attr('href');
str = href.match('.+solr\/.+\/(.*)')[1];
new_href = href.replace(str, upd_str);
$('#result').prepend('<a id="url_upd" class="address-bar" href="' + new_href + '"><strong>DELETE ALL</strong> ' + new_href + '</a>');
})();
If you're using Cloudera 5.x, Here in this documentation is mentioned that Lily maintains the Real time updations and deletions also.
Configuring the Lily HBase NRT Indexer Service for Use with Cloudera Search
As HBase applies inserts, updates, and deletes to HBase table cells,
the indexer keeps Solr consistent with the HBase table contents, using
standard HBase replication.
Not sure iftruncate 'hTable' is also supported in the same.
Else you create a Trigger or Service to clear up your data from both Solr and HBase on a particular Event or anything.
When clearing out a Solr index, you should also do a commit and optimize after running the delete-all query. Full steps required (curl is all you need): http://www.alphadevx.com/a/365-Clearing-a-Solr-search-index
Solr I am not sure but you can delete all the data from hbase using truncate command like below:
truncate 'table_name'
It will delete all row-keys from hbase table.