I am trying to read data (which is actually an array) in Lisp from a text file.
I tried to use with-open-file and read-line stuff but could not achieve my goal. What I am looking for is equivalent to doing data=load('filename.txt') in MATLAB, so that I get an array called data which has loaded the whole information in filename.txt.
The text file will be in a format like
1.0 2.0 3.0 ...
1.5 2.5 3.5 ...
2.0 3.0 4.0 ...
.....
The size may also vary. Thanks a lot in advance.
The basic way to do that is to use with-open-file for getting the input stream, read-line in a loop to get the lines, split-sequence (from the library of the same name) to split it into fields, and parse-number (from the library of the same name) to transform the strings into numbers. All libraries mentioned are available from Quicklisp.
EDIT: Just to get you started, this is a simple version without validation:
(defun load-array-from-file (filename)
(with-open-file (in filename
:direction :input)
(let* ((data-lol (loop :for line := (read-line in nil)
:while line
:collect (mapcar #'parse-number:parse-number
(cl-ppcre:split "\\s+" line))))
(rows (length data-lol))
(columns (length (first data-lol))))
(make-array (list rows columns)
:initial-contents data-lol))))
You should add some checks and think about what you want to get in case they are not fulfilled:
Are the rows all the same length?
Are all fields valid numbers?
Assuming your file follows the formatting pattern you gave in your question: a sequence of numbers separated with white spaces, this is a quick snippet that should do what you want.
(defun read-array (filename)
(with-open-file (in filename)
(loop for num = (read in nil)
until (null num)
collect num)))
Another approach is to leverage the lisp reader to parse the data in the text file. To do this, I'd probably convert the entire file into a string first, and then call
(eval (read-from-string (format nil "~a~a~a" "(initial wrapper code " str ")")))
For example, if you wanted to read in a data file that is all numbers, delimited by whitespace/newlines, into a list, the previous command would look like:
(eval (read-from-string (format nil "~a~a~a" "(list " str ")")))
I followed Svante's advice. I just needed a single column in the text file, this is what I am using for this purpose.
(defun load_data (arr column filename)
(setf lnt (first (array-dimensions arr)))
(with-open-file (str (format nil "~A.txt" filename) :direction :input)
(loop :for i :from 0 :to (1- lnt) :do
(setf (aref arr i 0) (read-from-string (nth (1- column) (split-sequence:SPLIT-SEQUENCE #\Space (read-line str))))))))
Thank you all for your help.
Related
I want to write a Common Lisp list in a .lisp file. If the file does not exist, it will be created and add the element.
If the file already exists, it will re-write the file appending new content to the list.
This implementation partially works:
(defun append-to-list-in-file (filename new-item &aux contents) ;;;;
(setq contents (list)) ;; default in case reading fails
(ignore-errors
(with-open-file (str filename :direction :input)
(setq contents (read str))))
(setq contents (nconc contents (list new-item)))
(with-open-file (str filename :direction :output :if-exists :overwrite)
(write contents :stream str)))
If I do:
(append-to-list-in-file "/home/pedro/miscellaneous/misc/tests-output/CL.lisp" 4)
It works. The code creates the file AND puts 4 inside of it as '(4). However, if I run the code again with a new element using the file that was just created:
(append-to-list-in-file "/home/pedro/miscellaneous/misc/tests-output/CL.lisp" 5)
It throws an error:
Error opening #P"/home/pedro/miscellaneous/misc/tests-output/CL.lisp"
[Condition of type SB-EXT:FILE-EXISTS]
I was expecting: '(4 5)
What do I need to change?
Probably a good idea to create the file. Otherwise you can't overwrite it.
... :if-does-not-exist :create :if-exists :overwrite ...
I realise this is a pretty basic question, but I'm just starting out in CL and I was wondering how to take input from standard input like:
1 2 3 4 5
And store it in an array.
I tried this:
(setq array (read-line))
Then checking the type gives cons.
I also tried constructing an array first like this:
(setf array (make-array n :element-type 'number))
Where n is the number of values I'll enter as input, but I'm lost after this. Do I need to use a loop or is there a way to do this without one?
Thanks.
You need to do these steps:
read the line
split into numbers
parse the numbers
It could look like this:
(defun read-array (stream)
(let* ((line (read-line stream))
(items (split-sequence #\Space line))
(numbers (map 'vector #'parse-integer items)))
numbers))
(Split-sequence is from the library of the same name.)
This is just the basic implementation, you likely want to sanitize your input, and split on any run of whitespace.
I advise against using read for reading user input, in any way, because the reader can do much more and you need to be very careful with user input.
The predefined function read-line returns a string (see the manual).
A simple way of obtaining from that string an array (assuming that the numbers are on a single line) is for instance to manipulate the returned string by adding the necessary syntax for reading it as a literal array through the function read-from-string. Here is a simple function adapted from one presented for lists in the excellent On Lisp book by Paul Graham:
CL-USER> (defun readarray (&rest args)
(values (read-from-string
(concatenate 'string "#("
(apply #'read-line args)
")"))))
READARRAY
CL-USER> (readarray)
1 2 3 4
#(1 2 3 4)
CL-USER> (type-of *)
(SIMPLE-VECTOR 4)
Of course if the numbers are on multiple lines some kind of iteration is required.
One can read from a string, by using a stream. Then one calls read as long as there are numbers, collects it into a list and converts the list to a vector.
CL-USER 36 > (coerce (with-input-from-string (stream "1 2 3 4 5")
(loop for n = (read stream nil nil)
while (numberp n)
collect n))
'vector)
#(1 2 3 4 5)
or: one creates a vector, which can grow - in Common Lisp the vector should be adjustable (when the size is unknown) and have a fill pointer. Then read from the string stream and push the numbers onto the vector.
CL-USER 40 > (let ((vector (make-array 0 :adjustable t :fill-pointer t)))
(with-input-from-string (stream "1 2 3 4 5")
(loop for n = (read stream nil nil)
while (numberp n)
do (vector-push-extend n vector)))
vector)
#(1 2 3 4 5)
The syntax for a literal array in Common Lisp like the one you're describing is #(1 2 3 4 5). You can simply type that instead of 1 2 3 4 5, and read it in:
CL-USER> (read)
; type "#(1 2 3 4 5)" (no quotes)
#(1 2 3 4 5) ; return value
CL-USER> (let ((array (read)))
(type-of array))
; type "#(1 2 3)" (no quotes)
(SIMPLE-VECTOR 3)
Given the path of a directory, how can I return the path of the newest file in that directory?
Using built-in APIs can achieve as well:
(defun latest-file (path)
"Get latest file (including directory) in PATH."
(car (directory-files path 'full nil #'file-newer-than-file-p)))
(latest-file "~/.emacs.d") ;; => "/Users/xcy/.emacs.d/var"
If you also needs files under sub-directory, use directory-files-recursively rather than directory-files. If you want to exclude directories, filter the file/directory list first by using file-directory-p.
Using f:
(defun aj-fetch-latest (path)
(let ((e (f-entries path)))
(car (sort e (lambda (a b)
(not (time-less-p (aj-mtime a)
(aj-mtime b))))))))
(defun aj-mtime (f) (let ((attrs (file-attributes f))) (nth 5 attrs)))
If you want to do it without dependencies
directory-files-and-attributes gives us a list of files and directories with attributes
the 4th arg NOSORT is a bool and does not take a function as the top answer suggests
(directory-files-and-attributes DIRECTORY &optional FULL MATCH NOSORT
ID-FORMAT)
If NOSORT is non-nil, the list is not sorted--its order is unpredictable.
NOSORT is useful if you plan to sort the result yourself.
attributes are in the format of file-attributes - which gives us the modification time and whether the item is a file or directory
t for directory, string (name linked to) for symbolic link, or nil.\
Last modification time, likewise. This is the time of the last
(car
(seq-find
'(lambda (x) (not (nth 1 x))) ; non-directory
(sort
(directory-files-and-attributes path 'full nil t)
'(lambda (x y) (time-less-p (nth 5 y) (nth 5 x)))))) ; last modified first: y < x
I'm trying to write reader for big files, based on iterations in Clojure. But how I can return line by line strings in Clojure? I want to make something like that:
(println (do_something(readFile (:file opts))) ; process and print first line
(println (do_something(readFile (:file opts))) ; process and print second line
Code:
(ns testapp.core
(:gen-class)
(:require [clojure.tools.cli :refer [cli]])
(:require [clojure.java.io]))
(defn readFile [file, cnt]
; Iterate over opened file (read line by line)
(with-open [rdr (clojure.java.io/reader file)]
(let [seq (line-seq rdr)]
; how return only one line there? and after, when needed, take next line?
)))
(defn -main [& args]
; Main function for project
(let [[opts args banner]
(cli args
["-h" "--help" "Print this help" :default false :flag true]
["-f" "--file" "REQUIRED: File with data"]
["-c" "--clusters" "Count of clusters" :default 3]
["-g" "--hamming" "Use Hamming algorithm"]
["-e" "--evklid" "Use Evklid algorithm"]
)]
; Print help, when no typed args
(when (:help opts)
(println banner)
(System/exit 0))
; Or process args and start work
(if (and (:file opts) (or (:hamming opts) (:evklid opts)))
(do
; Use Hamming algorithm
(if (:hamming opts)
(do
(println (readFile (:file opts))
(println (readFile (:file opts))
)
;(count (readFile (:file opts)))
; Use Evklid algorithm
(println "Evklid")))
(println "Please, type path for file and algorithm!"))))
May be i'm not understanding right what do you mean by "return line by line", but i'll suggest you to write function, which accepts file and processing function, then prints result of processing fuction for every line of your big file. Or, evem more general way, let's accept processing function and output function (println by default), so if we want not just print, but send it over network, save someplace, send to another thread, etc:
(defn process-file-by-lines
"Process file reading it line-by-line"
([file]
(process-file-by-lines file identity))
([file process-fn]
(process-file-by-lines file process-fn println))
([file process-fn output-fn]
(with-open [rdr (clojure.java.io/reader file)]
(doseq [line (line-seq rdr)]
(output-fn
(process-fn line))))))
So
(process-file-by-lines "/tmp/tmp.txt") ;; Will just print file line by ine
(process-file-by-lines "/tmp/tmp.txt"
reverse) ;; Will print each line reversed
Try doseq:
(defn readFile [file]
(with-open [rdr (clojure.java.io/reader file)]
(doseq [line (line-seq rdr)]
(println line))))
You can also try to read lazily from the reader, which is not the same as the lazy list of strings returned by line-seq. The details are discussed in this answer to a very similar question, but the gist of it is here:
(defn lazy-file-lines [file]
(letfn [(helper [rdr]
(lazy-seq
(if-let [line (.readLine rdr)]
(cons line (helper rdr))
(do (.close rdr) nil))))]
(helper (clojure.java.io/reader file))))
You can then map over the lines which will only be read as far as necessary. As discussed in more details in the linked answer, the downside is that if you don't read till the end of the file, the (.close rdr) will never be run, potentially causing issues with resources.
Would anyone have a suggestion how to go about having for:
\begin{array}{cc}
Lorem & Ipsum \\
More & Stuff \\
\end{array}
Where adding or removing a c, l or r in the part after array would add or remove the & from all lines in the array environment.
Basically the same trick could then be applied to matrices or table environments.
At the least I'd be interested in how others go about this "easy-to-go-wrong", "hard-to-efficiently-alter" task.
I usually generate the tables from a different format (tab separated values or org-mode tables) in which such operations are simpler.
This is not exactly the answer, but this is how I was doing it:
Align on &, for example: C-x . &.
Select the entire column I need using regular selection commands.
Cut a rectangular area by using C-x r k.
This is not super automatic, but given some exercise isn't really a hurdle, except, perhaps, if you have to re-format some old document and make a lot of changes all at once.
EDIT
(defun latex-merge-next-column (start end column)
"Works on selected region, removes COLUMN'th ampersand
in every line in the selected region"
(interactive "r\nnColumn to merge: ")
(labels ((%nth-index-of
(line)
(let ((i -1) (times 0))
(while (and (< times column) i)
(setq i (position ?\& line :start (1+ i))
times (1+ times))) i)))
(let ((region (split-string (buffer-substring start end) "\n"))
amp-pos
replacement)
(dolist (line region)
(setq amp-pos (%nth-index-of line)
replacement
(cons (if amp-pos
(concat (subseq line 0 amp-pos)
(subseq line (1+ amp-pos)))
line) replacement)))
(kill-region start end)
(insert (mapconcat #'identity (reverse replacement) "\n")))))
This would work on the selected region and remove the n'th ampersand in every line. You could bind it to some key that is comfortable for you, say:
(global-set-key (kbd "C-c C-n") 'latex-merge-next-column)
Then C-c C-n 2 would remove every second ampersand in the selected lines.
As suggested you can make a YASnippet that according to the amount of letters in the second argument array automatically adds the appropriate amount of &s to the first row of the array:
# -*- mode: snippet -*-
# name: array
# key: arr
# expand-env: ((yas/indent-line 'fixed))
# --
\begin{array}{${1:cc}}$0
${1:$
(let ((row ""))
(dotimes (i (- (string-width yas/text) 1) row)
(setq row (concat row "& "))))
}\\\\
\end{array}
The manual exemplifies this technique. The line with (yas/indent-line 'fixed) is to avoid AUCTeX indenting the row. The reason for placing the exit point of the snippet ($0) at the end of the declaration of the array rather than at the beginning of the first row is that when placed at the beginning of the first row the exit point does not behave as expected.
The following snippet will also add as many rows as there are columns:
# -*- mode: snippet -*-
# name: array
# key: arr
# expand-env: ((yas/indent-line 'fixed))
# --
\begin{array}{${1:cc}}$0
${1:$
(let ((row "") (allrows ""))
(dotimes (i (- (string-width yas/text) 1))
(setq row (concat row "& ")))
(dotimes (i (string-width yas/text) allrows)
(setq allrows (concat allrows row "\\\\\\\\\n"))))
}\end{array}
A problem with this snippet is that it adds \\ even if there only one column but such arrays may be rare.
There seems to be problems with adding lisp comments to embedded lisp code in snippets so I simply add a commented version of only the lisp code to explain it:
;; Make an empty row with as many columns as symbols in $1 (the $1 in
;; the snippet which is what yas/text refer to)
(let ((row "") (allrows ""))
;; Make an empty row with as many columns as symbols in $1
(dotimes (i (- (string-width yas/text) 1))
(setq row (concat row "& ")))
;; Make as many rows as symbols in $1
(dotimes (i (string-width yas/text) allrows)
(setq allrows (concat allrows row "\\\\\\\\\n"))))
Building on the solution by #wvxvw, how about just using M-x align-current in the tabular/matrix/array environment and then manipulating using the block selection/insertion commands? This seems to work intelligently with escaped ampersands. I find it useful to disable wrapping during this operation. I don't find this hard to edit at all, as relatively regular re-alignment makes everything quite readable.