Coverting text file to array in IDL - arrays

I was looking for a way to convert the following textfile:
7.73E+01
7.29E+01
7.06E+00
6.84E+00
6.70E+00
6.45E+00
6.12E+00
5.93E+00
Into an IDL array
array = [7.73E+01,7.29E+01,7.06E+00,6.84E+00,6.70E+00,6.45E+00,6.12E+00,5.93E+00]
Thanks in advance for the help.

For a simple file like you show, you can just do:
n_lines = file_lines(filename)
array = fltarr(n_lines)
open, lun, filename, /get_lun
readf, lun, array
free_lun, lun
It can get more complicated if the text file is not just a single value per line and every line is part of the data.

Related

Is there a way to make python print to file for every iteration of a for loop instead of storing all in the buffer?

I am looping over a very large document to try and lemmatise it.
Unfortunately python does not seem to print to file for every line but run through the whole document before printing, which given the size of my file exceeds the memory...
Before I chunk my document into more bite-sized chunks I wondered if there was a way to force python to print to file for every line.
So far my code reads:
import spacy
nlp = spacy.load('de_core_news_lg')
fin = "input.txt"
fout = "output.txt"
#%%
with open(fin) as f:
corpus = f.readlines()
corpus_lemma = []
for word in corpus:
result = ' '.join([token.lemma_ for token in nlp(word)])
corpus_lemma.append(result)
with open(fout, 'w') as g:
for item in corpus_lemma:
g.write(f'{item}')
To give credits for the code, it was kindly suggested here: Ho to do lemmatization on German text?
As described in: How to read a large file - line by line?
If you do your lemmatisation inside the with block, Python will handle reading line by line using buffered I/O.
In your case, it would look like:
import spacy
nlp = spacy.load('de_core_news_lg')
fin = "input.txt"
fout = "output.txt"
#%%
corpus_lemma = []
with open(fin) as f:
for line in f:
result = " ".join(token.lemma_ for token in nlp(line))
corpus_lemma.append(result)
with open(fout) as g:
for item in corpus_lemma:
g.write(f"{item}")

How to completely remove a line from a file?

How do I completely remove a line in Rust? Not just replace it with an empty line.
In Rust, when you delete a line from a file with the following code as an example:
let mut file: File = File::open("file.txt").unwrap();
let mut buf = String::from("");
file.read_to_string(&mut buf).unwrap(); //Read the file to a buffer
let reader = BufReader::new(&file);
for (index, line) in reader.lines().enumerate() { //Loop through all the lines in the file
if line.as_ref().unwrap().contains("some text") { //If the line contains "some text", execute the block
buf = buf.replace(line.as_ref().unwrap(), ""); //Replace "some text" with nothing
}
}
file.write_all(buf.as_bytes()).unwrap(); //Write the buffer back to the file
file.txt:
random text
random text
random text
some text
random text
random text
When you run the code, file.txt turns into this:
random text
random text
random text
random text
random text
Rather than just
random text
random text
random text
random text
random text
Is there any way to completely remove the line rather than just leaving it blank? Like some sort of special character?
This part is bad-news: buf = buf.replace(line.as_ref().unwrap(), "");
This is doing a search through your entire buffer to find the line contents (without '\n') and replace it with "". To make it behave as you expect you need to add back in the newline. You can just about do this by buf.replace(line.as_ref().unwrap() + "\n", "") The problem is that lines() treats more than "\n" as a newline, it also splits on "\r\n". If you know you're always using "\n" or "\r\n" as newlines you can work around this - if not you'll need something tricker than lines().
However, there is a trickier issue. For larger files, this may end up scanning through the string and resizing it many times, giving an O(N^2) style behaviour rather than the expected O(N). Also, the entire file needs to be read into memory, which can be bad for very large files.
The simplest solution to the O(N^2) and memory issues is to do your processing line-by-line, and
then move your new file into place. It would look something like this.
//Scope to ensure that the files are closed
{
let mut file: File = File::open("file.txt").unwrap();
let mut out_file: File = File::open("file.txt.temp").unwrap();
let reader = BufReader::new(&file);
let writer = BufWriter::new(&out_file);
for (index, line) in reader.lines().enumerate() {
let line = line.as_ref().unwrap();
if !line.contains("some text") {
writeln!(writer, "{}", line);
}
}
}
fs::rename("file.txt.temp", "file.txt").unwrap();
This still does not handle cross-platform newlines correctly, for that you'd need a smarter lines iterator.
Hmm could try removing the new line char in the previous line

Matlab: copy array in file txt

I want to copy elements of an array in a file txt. I read the first 50 samples of a wav file in array a and I want to copy a in a file txt. This is my code:
[s,fs]=wavread('file.wav');
for k=1:50
a=s(k)
end
fid = fopen('file.txt','wt');
fprintf(fid,'%f\n',a);
fclose(fid);
With this code in file txt there is only the last element of a, not all samples.
fprintf can process vector input. So you can simply replace the loop by a = s(1:50);:
[s,fs]=wavread('file.wav');
a = s(1:50);
fid = fopen('file.txt','wt');
fprintf(fid,'%f\n',a);
fclose(fid);

Read text file as numpy array using np.loadtxt

I am trying to read a text file as a numpy array. For some reason one of the files is read fine, but an error (X = np.array(X, dtype) ValueError: setting an array element with a sequence.) is reported for another.
The code is:
freq_chan = np.loadtxt(os.path.join(dirs,fil), skiprows = 6+int(no_nodes))
The row of the file that is read is:
45.000000000000 1.73145123922036E-002 -2.27352994577858E-004 0.0000000000000 0.0000000000000 0.0000000000000 0.0000000000000
and the row of the file that is not read is:
450.00000000000 1.75123936984107E-003 4.99078580749004E-004 -1.01870220257046E-005 -1.25748632064143E-005 4.53694668200015E-004 1.75279359420616E-003 1.06388230080026E-005 1.25165432922695E-005 -1.26393875391086E-003
What might be the reason for this?
Thanks
I suspect that there is a problem with your delimiter character at least in the first file. try to set the delimiter argument.
Take a look to this explanation

Save a string vector as csv in matlab

I have the following string array in matlab built the following way:
labels=textread(nome_tecnicas_base, '%s');
for i=1:size(labels)
temp_vector=cell(1,10);
[temp_vector{1:10}]=deal(labels{i});
final_vector=horzcat(final_vector,temp_vector);
end
I want to save this vector with each string value separated with commas (e.g., csv files) in a text file. I tried in several ways, but when I try to read it with, for example, the textread function i have the following error:
a=textread('labels-cpen-R.txt')
Error using dataread
Trouble reading number from file (row 1, field 1) ==> dct,dct,dct,dct,dct,dct,dct,dct,dct,dct,hierar
this is how my file was saved
dct,dct,dct,dct,dct,dct,dct,dct,dct,dct,hierarch-sift,hierarch-sift,hierarch-sift,hierarch-sift,hierarch-sift,hierarch-sift,hierarch-sift,hierarch sift,hierarch-sift,hierarch
sift,zernike,zernike,zernike,zernike,zernike,zernike,zernike,zernike,zernike,zernike,zernike2,zernike2,zernike2,zernike2,zernike2,zernike2,zernike2,zernike2,zernike2,zernike2,kpca,kpca,kpca,kpca,kpca,kpca,kpca,kpca,kpca,kpca,sift,sift,sift,sift,sift,sift,sift,sift,sift,sift,surf,surf,surf,surf,surf,surf,surf,surf,surf,surf,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bayesianfusion0,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,bks-fusion,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting4,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,fusionvoting6,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,multiscale_voting,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_rf_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_lvt,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,bks_svr_otsu,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_rf_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt,multiscale_bks_svr_lvt
How can I save this vector and how can I read this file properly?
try textscan for reading and fprintf for writing
from the matlab documentation:
fileID = fopen('data.csv');
C = textscan(fileID,'%f %f %f %f %u8 %f',...
'Delimiter',',','EmptyValue',-Inf);
so in your case:
textscan(fileID,'%s', 'Delimiter', ',')
edit: for writing data to a file, you can use fprintf with a file identifier:
fileID= fopen('data.csv', 'w') ;
fprintf(fileID, '%s,', data{1,1:end-1}) ;
fprintf(fileID, '%s\n', data{1,end}) ;
fclose(fileID)

Resources