I am trying to read a text file as a numpy array. For some reason one of the files is read fine, but an error (X = np.array(X, dtype) ValueError: setting an array element with a sequence.) is reported for another.
The code is:
freq_chan = np.loadtxt(os.path.join(dirs,fil), skiprows = 6+int(no_nodes))
The row of the file that is read is:
45.000000000000 1.73145123922036E-002 -2.27352994577858E-004 0.0000000000000 0.0000000000000 0.0000000000000 0.0000000000000
and the row of the file that is not read is:
450.00000000000 1.75123936984107E-003 4.99078580749004E-004 -1.01870220257046E-005 -1.25748632064143E-005 4.53694668200015E-004 1.75279359420616E-003 1.06388230080026E-005 1.25165432922695E-005 -1.26393875391086E-003
What might be the reason for this?
Thanks
I suspect that there is a problem with your delimiter character at least in the first file. try to set the delimiter argument.
Take a look to this explanation
Related
I was looking for a way to convert the following textfile:
7.73E+01
7.29E+01
7.06E+00
6.84E+00
6.70E+00
6.45E+00
6.12E+00
5.93E+00
Into an IDL array
array = [7.73E+01,7.29E+01,7.06E+00,6.84E+00,6.70E+00,6.45E+00,6.12E+00,5.93E+00]
Thanks in advance for the help.
For a simple file like you show, you can just do:
n_lines = file_lines(filename)
array = fltarr(n_lines)
open, lun, filename, /get_lun
readf, lun, array
free_lun, lun
It can get more complicated if the text file is not just a single value per line and every line is part of the data.
How do I completely remove a line in Rust? Not just replace it with an empty line.
In Rust, when you delete a line from a file with the following code as an example:
let mut file: File = File::open("file.txt").unwrap();
let mut buf = String::from("");
file.read_to_string(&mut buf).unwrap(); //Read the file to a buffer
let reader = BufReader::new(&file);
for (index, line) in reader.lines().enumerate() { //Loop through all the lines in the file
if line.as_ref().unwrap().contains("some text") { //If the line contains "some text", execute the block
buf = buf.replace(line.as_ref().unwrap(), ""); //Replace "some text" with nothing
}
}
file.write_all(buf.as_bytes()).unwrap(); //Write the buffer back to the file
file.txt:
random text
random text
random text
some text
random text
random text
When you run the code, file.txt turns into this:
random text
random text
random text
random text
random text
Rather than just
random text
random text
random text
random text
random text
Is there any way to completely remove the line rather than just leaving it blank? Like some sort of special character?
This part is bad-news: buf = buf.replace(line.as_ref().unwrap(), "");
This is doing a search through your entire buffer to find the line contents (without '\n') and replace it with "". To make it behave as you expect you need to add back in the newline. You can just about do this by buf.replace(line.as_ref().unwrap() + "\n", "") The problem is that lines() treats more than "\n" as a newline, it also splits on "\r\n". If you know you're always using "\n" or "\r\n" as newlines you can work around this - if not you'll need something tricker than lines().
However, there is a trickier issue. For larger files, this may end up scanning through the string and resizing it many times, giving an O(N^2) style behaviour rather than the expected O(N). Also, the entire file needs to be read into memory, which can be bad for very large files.
The simplest solution to the O(N^2) and memory issues is to do your processing line-by-line, and
then move your new file into place. It would look something like this.
//Scope to ensure that the files are closed
{
let mut file: File = File::open("file.txt").unwrap();
let mut out_file: File = File::open("file.txt.temp").unwrap();
let reader = BufReader::new(&file);
let writer = BufWriter::new(&out_file);
for (index, line) in reader.lines().enumerate() {
let line = line.as_ref().unwrap();
if !line.contains("some text") {
writeln!(writer, "{}", line);
}
}
}
fs::rename("file.txt.temp", "file.txt").unwrap();
This still does not handle cross-platform newlines correctly, for that you'd need a smarter lines iterator.
Hmm could try removing the new line char in the previous line
I'm trying to get a random line from a file:
extern crate rand;
use rand::Rng;
use std::{
fs::File,
io::{prelude::*, BufReader},
};
const FILENAME: &str = "/etc/hosts";
fn find_word() -> String {
let f = File::open(FILENAME).expect(&format!("(;_;) file not found: {}", FILENAME));
let f = BufReader::new(f);
let lines: Vec<_> = f.lines().collect();
let n = rand::thread_rng().gen_range(0, lines.len());
let line = lines
.get(n)
.expect(&format!("(;_;) Couldn't get {}th line", n))
.unwrap_or(String::from(""));
line
}
This code doesn't work:
error[E0507]: cannot move out of borrowed content
--> src/main.rs:18:16
|
18 | let line = lines
| ________________^
19 | | .get(n)
20 | | .expect(&format!("(;_;) Couldn't get {}th line", n))
| |____________________________________________________________^ cannot move out of borrowed content
I tried adding .clone() before .expect(...) and before .unwrap_or(...) but it gave the same error.
Is there a better way to get a random line from a file that doesn't involve collecting the whole file in a Vec?
Use IteratorRandom::choose to randomly sample from an iterator using reservoir sampling. This will scan through the entire file once, creating Strings for each line, but it will not create a giant vector for every line:
use rand::seq::IteratorRandom; // 0.7.3
use std::{
fs::File,
io::{BufRead, BufReader},
};
const FILENAME: &str = "/etc/hosts";
fn find_word() -> String {
let f = File::open(FILENAME)
.unwrap_or_else(|e| panic!("(;_;) file not found: {}: {}", FILENAME, e));
let f = BufReader::new(f);
let lines = f.lines().map(|l| l.expect("Couldn't read line"));
lines
.choose(&mut rand::thread_rng())
.expect("File had no lines")
}
Your original problem is that:
slice::get returns an optional reference into the vector.
You can either clone this or take ownership of the value:
let line = lines[n].cloned()
let line = lines.swap_remove(n)
Both of these panic if n is out-of-bounds, which is reasonable here as you know that you are in bounds.
BufRead::lines returns io::Result<String>, so you have to handle that error case.
Additionally, don't use format! with expect:
expect(&format!("..."))
This will unconditionally allocate memory. When there's no failure, that allocation is wasted. Use unwrap_or_else as shown.
Is there a better way to get a random line from a file that doesn't involve collecting the whole file in a Vec?
You will always need to read the whole file, if only to know the number of lines. However, you don't need to store everything in memory, you can read lines one by one and discard them as you go so that you only keep one in the end. Here is how it goes:
Read and store the first line;
Read the second line, draw a random choice and either:
keep the first line with a probability of 50%,
or discard the first line and store the second line with a probability of 50%,
Keep reading lines from the file and for line number n, draw a random choice and:
keep the currently stored line with a probability of (n-1)/n,
or replace the currently stored line with the current line with a probability of 1/n.
Note that this is more or less what sample_iter does, except that sample_iter is more generic since it can work on any iterator and it can pick samples of any size (eg. it can choose k items randomly).
I have details like below in an array. There will be plenty of testbed details in actual case. I want to grep a particular testbed(TESTBED = vApp_eprapot_icr) and an infomation like below should get copied to another array. How can I do it using perl ? End of Testbed info can be understood by a closing flower bracket }.
TESTBED = vApp_eprapot_icr {
DEVICE = vApp_eprapot_icr-ipos1
DEVICE = vApp_eprapot_icr-ipos2
DEVICE = vApp_eprapot_icr-ipos3
DEVICE = vApp_eprapot_icr-ipos5
CARDS=1GIGE,ETHFAST
CARDS=3GIGE,ETHFAST
CARDS=10PGIGE,ETHFAST
CARDS=20PGIGE,ETHFAST
CARDS=40PGIGE,ETHFAST
CARDS=ETHFAST,ETHFAST
CARDS=10GIGE,ETHFAST
CARDS=ETH,ETHFAST
CARDS=10P10GIGE,ETHFAST
CARDS=PPA2GIGE,ETHFAST
CARDS=ETH,ETHFAST,ETHGIGE
}
I will make it simpler, please see the below array
#array("
student=Amit {
Age=20
sex=male
rollno=201
}
student=Akshaya {
Age=24
phone:88665544
sex=female
rollno=407
}
student=Akash {
Age=23
sex=male
rollno=356
address=na
phone=88456789
}
");
Consider an array like this. Where such entries are plenty. I need to grep, for an example student=Akshaya's data. from the opening '{' to closing '}' all info should get copied to another array. This is what I'm looking for.
while (<>) {
print if /TESTBED = vApp_eprapot_icr/../\}/;
}
as a sidenote <> will capture the filename you use on cmdline. So if the data is stored in a file you will run from commandline
perl scriptname.pl filename.txt
Ok. We finally have enough information to come up with an answer. Or, at least, to produce two answers which will work on slightly different versions of your input file.
In a comment you say that you are creating your array like this:
#array = `cat $file`;
That's not a very good idea for a couple of reasons. Firstly, why run an external command like cat when Perl will read the file for you. And secondly, this gives you one element in your array for each line in your input file. Things become far easier if you arrange it so that each of your TESTBED = foo { ... } records is a single array element.
Let's get rid of the cat first. The easiest way to read a single file into an array is to use the file input operator - <>. That will read data from the file whose name is given on the command line. So if you call your program filter_records, you can call it like this:
$ ./filter_records your_input_data.txt
And then read it into an array like this:
#array = <>;
That's good, but we still have each line of the input file in its own array element. How we fix that depends on the exact format of your input file. It's easiest if there's a blank line between each record in the input file, so it looks like this:
student=Amit {
Age=20
sex=male
rollno=201
}
student=Akshaya {
Age=24
phone:88665544
sex=female
rollno=407
}
student=Akash {
Age=23
sex=male
rollno=356
address=na
phone=88456789
}
Perl has a special variable called $/ which controls how it reads records from input files. If we set it to be an empty string then Perl goes into "paragraph" mode and it uses blank lines to delimit records. So we can write code like this:
{
local $/ = '';
#array = <>;
}
Note that it's always a good idea to localise changes to Perl's special variables, which is why I have enclosed the whole thing in a naked block.
If there are no blank lines, then things get slightly harder. We'll read the whole file in and then split it.
Here's our example file with no blank lines:
student=Amit {
Age=20
sex=male
rollno=201
}
student=Akshaya {
Age=24
phone:88665544
sex=female
rollno=407
}
student=Akash {
Age=23
sex=male
rollno=356
address=na
phone=88456789
}
And here's the code we use to read that data into an array.
{
local $/;
$data = <>;
}
#array = split /(?<=^})\n/m, $data;
This time, we've set $/ to undef which means that all of the data has been read from the file. We then split the data wherever we find a newline that is preceded by a } on a line by itself.
Whichever of the two solutions above that we use, we end up with an array which (for our sample data) has three elements - one for each of the records in our data file. It's then simple to use Perl's grep to filter that array in various ways:
# All students whose names start with 'Ak'
#filtered_array = grep { /student=Ak/ } #array;
If you use similar techniques on your original data file, then you can get the records that you are interested in with code like this:
#filtered_array = grep { /TESTBED = vApp_eprapot_icr/ } #array;
Got it.
while 1:
line = sub.readline().split()
if line == []:
new = main
break
else:
new = main.replace(line[0],line[1])
main = new
This seem to work for me. Thanks for the help =).
Try another loop and in that loop index the word swap that you need to occur:
I assume each line of sub.txt has the subs that you want. Read all the lines of sub.txt, storing each line in an indexable array. Set up a loop around your main code, and in that loop index through the array referencing, sequentially the line of sub.txt that you want each time.
As pointed out by Cameron, this method will overwrite the output file every time thus recording only your last change.
The error in this block is :
while True:
word = substitute.readline().split()
print(word)
if word == []: // ---Indentation ---
break
else:
new = (main_story.read().replace(word[0],word[1]))
new_story.write(new)
You need to read the complete file at once, make changes and write to file.
Or you could read from the first file, and then do subsequent read/writes on the new file.