hi all ijson newbie I have a very large .json file 168 (GB) I want to get all possible keys, but in the file some values are written as NaN. ijson creates a generator and outputs dictionaries, in My code value. When a specific item is returned, it throws an error. How can you get a string instead of a dictionary instead of value? Tried **parser = ijson.items (input_file, '', multiple_values = True, map_type = str) **, didn't help.
def parse_json(json_filename):
with open('max_data_error.txt', 'w') as outfile:
with open(json_filename, 'r') as input_file:'''
# outfile.write('[ '
parser = ijson.items(input_file, '', multiple_values=True)
cont = 0
max_keys_list = list()
for value in parser:
for i in json.loads(json.dumps(value, ensure_ascii=False, default=str)) :
if i not in max_keys_list:
max_keys_list.append(i)
print(value)
print(max_keys_list)
for keys_item in max_keys_list:
outfile.write(keys_item + '\n')
if __name__ == '__main__':
parse_json('./email/emailrecords.bson.json')
Traceback (most recent call last):
File "panda read.py", line 29, in <module>
parse_json('./email/emailrecords.bson.json')
File "panda read.py", line 17, in parse_json
for value in parser:
ijson.common.IncompleteJSONError: lexical error: invalid char in json text.
litecashwire.com","lastname":NaN,"firstname":"Mia","zip":"87
(right here) ------^
Your file I not valid JSON (NaN is not a valid JSON value); therefore any JSON parsing library will complain about this, one way or another, unless they have an extension to handle this non-standard content.
The ijson FAQ found in the project description has a question about invalid UTF-8 characters and how to deal with them. Those same answers apply here, so I would suggest you go and try one of those.
Need to write a regex to get 3 groups from strings-
<whatever text including new lines optional -group 1>/command <text until \n or </p> is encountered- group 2><whatever text including new lines optional -group 3>
what I tried is-
Pattern pattern1 = Pattern.compile('(.*?)[/]command (.*?)\n?(.*?)');
It should give the following output for string-
some\nthing/command cmdtext/nasdfjaklsdjf\nfgskdlkfg\ndgsdfgsdfgsdfg
group 1 - some\nthing
group 2 - cmdtext
group 3 - asdfjaklsdjf\nfgskdlkfg\ndgsdfgsdfgsdfg
What I am not getting is how to get the occurrence of </p> and .* is not considering the group. Although this is working for me-
String a = '\na\na\n\n\n\n\n\naaa';
Pattern pattern2 = Pattern.compile('\n(?s:.)*');
Matcher mchr = GiphyPattern.matcher(a);
system.assert (mchr.matches());
This regular expression should match what you need:
'([\\s\\S]*)/command (.*?)(?:\n|</p>)([\\s\\S]*)'
You cannot match \n with .* So I am using \\s\\S instead (which is actually \s\S but with Apex escaped backslashes).
I am trying to do a parsing of a long file like this (the output of the command play in Linux):
File :1.mp3
In:0.00% 00:00:00.00 [00:03:14.51] Out:0 [ | ]
In:0.19% 00:00:00.37 [00:03:14.14] Out:16.4k [ | ]
In:0.29% 00:00:00.56 [00:03:13.95] Out:24.6k [======|======]
In:0.33% 00:00:00.65 [00:03:13.86] Out:28.7k [ =====|===== ]
In:0.43% 00:00:00.84 [00:03:13.67] Out:36.9k [ =====|===== ]
In:0.53% 00:00:01.02 [00:03:13.49] Out:45.1k [ -====|===== ]
In:0.62% 00:00:01.21 [00:03:13.30] Out:53.2k [ =====|===== ]
In:0.72% 00:00:01.39 [00:03:13.11] Out:61.4k [-=====|======]
In:0.81% 00:00:01.58 [00:03:12.93] Out:69.6k [-=====|=====-]
In:0.91% 00:00:01.76 [00:03:12.74] Out:77.8k [-=====|=====-]
In:0.96% 00:00:01.86 [00:03:12.65] Out:81.9k [ =====|===== ]
And so on
I would like to parse the percentage number.
How can i do it without saving the file into(because is too large ~ 100KB) a String.
i thought with this regular expression :"In:(\d{1,2}\.\d{2})"
how to do it?
Try this regex:
/^In:([0-9]{1,3}\.[0-9]{1,2})\%/gm
Explanation:
/
^ Matches start of string.
In: Matches "In:".
( ) Groups percentage (excl. sign).
[0-9]{1,3} Matches 1-3 (incl.) numbers.
\. Matches a dot.
[0-9]{1,2} Matches 1-2 (incl.) numbers.
\% Matches a percent sign.
/gm Allows multiple matches and makes ^ match beginning of line (not beginning of string), respectively.
Please help me on the below code :
I have an array with 155 elements and i have a file which has some elements of array inside it , i need all values of the array elements which are found in the file and also i need the array element to be printed as zero if the array element is not found in the file .
Thanks in advance, this is what i have tried.
args=("C9" "DP10" "DP11" "DP20" "DP21" "DP30" "DP31" "DP50" "FR31" "G128" "G402" "G602" "GA" "GI" "GT08" "GT14" "GT17" "GT25" "GT37" "GT67" "H6" "H7" "IL" "IM" "J6" "JD05" "JD09" "JD14" "JD25" "JD37" "K1" "K2" "L100" "L106" "L116" "L150" "L202" "L7" "L8" "L9" "LD11" "LD21" "LE09" "LE26" "LP11" "LP21" "LP31" "LP55" "LQ11" "LQ21" "LQ31" "LS07" "LT09" "LT10" "LT12" "LT15" "LT20" "LT22" "LT24" "LT25" "LT30" "LT38" "LT42" "LT43" "LT44" "LT48" "LT50" "LT59" "LT60" "LT65" "M395" "OV04" "OV07" "OV14" "OV18" "OV23" "OV27" "OV35" "OV39" "OV40" "OV79" "Q15" "Q150" "Q19" "QD11" "QD21" "QD31" "QD65" "QE11" "QE21" "QE31" "QF50" "QM25" "QP10" "QP15" "QP20" "QP30" "QP31" "QP50" "QT25" "QT50" "R39" "R40" "r57" "R9" "rc23" "RC27" "RC39" "rc7" "rc79" "S1" "S101" "S117" "S118" "S13p" "S18" "S202" "S317" "S318" "S319" "S40" "S408" "S67" "S76" "S82" "S99" "SD11" "SD12" "SD14" "SD17" "SD29" "SD3" "SD5" "SD98" "SF20" "SF74" "SR07" "SV19" "SV6p" "T402" "T602" "TG00" "TG17" "TG43" "TG8" "TG92" "WD09" "WD14" "WD17" "WD24" "WD29" "WD37" "WD43" "WWE1" "XR91")
MY CODE :
for loop i have used to traverse the elements search inside a file .
for i in ${args[#]}; do
grep $i file.txt
if [ $? -ne 0 ]; then
echo $i"","""0"
fi
done >> output.txt
TOTAL FILE:
C9,5015319
DP10,36870732
DP11,188
DP20,18728254
DP21,341182
DP30,8415555
DP31,2390000
DP50,12371853
FR31,24541
G128,49780
G402,2000
G602,2000
GA,879888
GT08,1580384
GT17,1968192
GT25,4104
GT37,21550
GT67,24770
H6,660652
IL,137651
JD05,1518400
JD14,325800
JD25,828600
JD37,357100
K1,261549
K2,4715330
L100,284
L116,80000
L7,200847
L8,3158
L9,5054495
LE09,75776
LE26,343410
LP11,1030
LP21,492
LP31,113
LP55,3
LQ11,6776000
LQ21,3543600
LQ31,4525600
LT09,682800
LT12,5715
LT15,568873
LT22,236077
LT24,702800
LT25,4600
LT38,28990
LT65,300125
M395,29600
OV14,462
OV18,86300
OV40,217899
Q150,678
QD11,1000022
QD31,50
QF50,58575
QM25,57900
QP10,1792153
QP15,953400
QP20,770000
QP30,179450
QP31,163223
QP50,8
QT50,66340
R39,62440
R40,18807
r57,3456
rc23,3370
RC27,2809
RC39,2570
rc7,7137
rc79,1296
S1,25007
S117,1000000
S13p,52313
S18,75000
S317,289148
S318,3046
S319,30000
S40,300
S408,4967
S76,28
S82,103238
S99,480
SD11,6719
SD12,23123
SD14,22595
SD17,100000
SD29,252392
SD3,20000
SD5,14090
SD98,653
SF20,1000
SF74,7330
SV19,26461
SV6p,154994
T402,2000
T602,2000
TG17,2031
TG8,2964
TG92,1759
WD17,131194
WD24,94589
WD29,202198
WD37,101794
WD43,112942
WWE1,9600
XR91,70000
EXPECTED OUTPUT :
The output should contain the values which are present in the file for each array element.
If not present the output should contain the array element as zero. For eg:
c9 is not present in the file
output of c9 should be
c9,0
Your approach is not bad. I just would use
^$i,
as a grep-pattern. With your current file data, it's not necessary, but maybe one day your file will contain things like
X,2354
XA,1234
and suddenly your algorithm will fail, if args contain the element X.
Also, the echo statement is unnecessarily complex. I would write it simply as
echo $x,0
You can also simplify the if, by combining it with the grep
if ! grep ^$i, file.txt
but this is mere cosmetics and a matter of taste.
How does one search for specific punctuation in Solr, such as :)? I have tried URL encoding the text but I still get this message:
org.apache.solr.search.SyntaxError: Cannot parse ':': Encountered " ":" ": "" at line 1, column 0.
Was expecting one of:
<NOT> ...
"+" ...
"-" ...
<BAREOPER> ...
"(" ...
"*" ...
<QUOTED> ...
<TERM> ...
<PREFIXTERM> ...
<WILDTERM> ...
<REGEXPTERM> ...
"[" ...
"{" ...
<LPARAMS> ...
<NUMBER> ...
<TERM> ...
"*" ...
Additionally, I need to perform this search on a text field, not on a string field. How should I configure the analyser to save punctuation?
Note that searching google for the subject is impossible due to two prolific Solr contributors with the name "Smiley"!
What configurations you have for the text field?
You should take care the splitting is not happening on the puntuations e.g. if using StandardTokenizerFactory or word delimiter filter.
You can define a custom field with WhitespaceTokenizerFactory or KeywordTokenizerFactory and have further filters like lower case on it.
Also, There are some characters which Solr/Lucene uses for some operation e.g. + - ! ( ) { } [ ] ^ " ~ * ? :
You would need to escape the special characters with backslash. Check Escape Special Characters
instead of :) search for "\:\ )" , both chars :,) have special meaning in SOLR.
for all special operatos you need to escape by prefixing with '\' char .