What I want to do:
take the output of (drakma:http-request "someURL.jpg") and write it to a file. The output consists of a byte array.
What I came up with:
(with-open-file (my-stream "thing.jpg"
:direction :output
:element-type 'binary
:if-does-not-exist :create
:if-exists :supersede)
(let ((content (drakma:http-request "someURL.jpg")))
(loop for i across content do
(write-byte i my-stream))))
The error I get:
debugger invoked on a SIMPLE-ERROR in thread #<THREAD "initial thread" RUNNING
{1002978FB1}>:
could not find any output routine for :FULL buffered BINARY
I use SBCL 1.0.49.
If you want any other info or find my question unclear then please, ask away :).
EDIT:
The problem is the :element-type, it's supposed to be 'unsigned-byte.
You can find information on the different possibilities of :element-type over at the Common Lisp Hyper Spec under open . Other than that everything is correct.
PS. I couldn't respond as an answer because my reputation is too low.
As Xach said, you're better off using '(unsigned-byte 8) (or make a type-definition , e.g. (deftype binary () '(unsigned-byte 8))).
In addition, you can probably replace your loop over the array and writing byte by byte with a call to write-sequence
Related
I have a Fortran code that gets integers from users, sorts them in descending order with bubble sort, then writes it into a file. However, in the beginning, instead of getting the integers from users, I have to get them from a file. How can I do that? Could you please help me? Thanks.
PROGRAM project
IMPLICIT NONE
INTEGER array(1000),t,p,c
PRINT*,"Enter 1000 element array"
READ*,array
c=1
OPEN(UNIT=25,FILE="sorted.txt")
DO p=1,999
DO c=1,999
IF (array(c)>array(c+1)) then
t=array(c)
array(c)=array(c+1)
array(c+1)=t
ENDIF
ENDDO
ENDDO
WRITE(98,*) array
CLOSE(98)
PRINT*,array(2:999)
END PROGRAM
Before I start, you open the file to write the output to as unit 25:
OPEN(UNIT=25,FILE="sorted.txt")
But you write and close unit 98:
WRITE(98,*) array
CLOSE(98)
Is this an error in your question, otherwise you might want to investigate.
That said, if you are running the program from a shell like bash, the easiest way is to use the output redirecting:
$ cat unsorted.txt > sort.exe
That way you don't have to change anything in your code.
If you don't want to do that, you need to open and read from the file in your program:
open(unit=24, file='unsorted.txt', action='READ', &
status='OLD', form="FORMATTED")
read(24, *) array
close(24)
This assumes that the file with the unsorted data contains exactly 1000 integers and nothing else.
I am new to clojure. I am having trouble as in how to print a particular string after removing special characters and reading a clojure file line-by-line.
I have a file named example.clj which looks like:
(defn patient
"Some string"
(measures
patient/symptom-fever
patient/symptom-cold
.
.
.
.
patient/symptom-bp))
From the above file i have removed all the special characters and reading the file line-by-line with below clojure code.
(defn loadcljfile []
(def newstring(clojure.string/replace (slurp "C:/Users/Desktop/example.clj") #"[\[/.(:)\]-]" " "))
(with-open [rdr (reader newstring)]
(doseq [line (line-seq rdr)]
(println line))))
(loadcljfile)
Now not getting how to print patient i.e my defn name and all the measures.
Can anyone help me?
Thanks
when defining things with names inside a function it's much safer and easier to use let rather than def. def defines it for the whole namespace (usually the whole file) and let gives something a name just until it's closing ).
I'm this case it makes a lot of sense to let Clojure read the expression and turn it into a data structure for you. you can do this with the load-file function if your file is a function you wish to run or use tools.reader if you just want to have it at a data structure.
if you go the load-file approach you can print it with the clojure.repl/source function and if you go the tools.reader path then you can just print it.
I checked the post here on how to loop over a file. I checked several places online trying to understand what is going on in that code snipped posted here again:
(defun get-file (filename)
(with-open-file (stream filename)
(loop for line = (read-line stream nil)
while line
collect line)))
This syntax seems to be so arcane, and the references are not trivial to understand. According to the reference here " the fourth of seven for/as syntaxes", the syntax is:
for var [type-spec] = expr1 [then expr2]
And if there is no expr2, then expr1 is used.
What is confusing me:
I am lost at the lines without knowing what to ask other than how does it work:
while line
collect line
I wouldn't know how to substitute that with other code. It does not look like a list at all. My intuition would tell me that it should've looked something like:
(while line (collect line))
Also, isn't that also same as:
while line collect line
And what is the expr1:
(read-line stream nil)
while line
collect line)
or
while line
collect line
What if I had much more code instead of collect line? Will there be no lists? I don't see the structure.
I know these are more than 1 question, but probably there is something big I am missing, preventing me from asking a good question.
It's a loop. Stuff which runs multiple times.
Basically the loop has two parts:
part one defines a variable line and what happens to it: on each loop iteration line is set to the result of evaluating the expression (read-line stream nil). This means that lines get read one by one.
the second part are the clauses which are run on each iteration. This part says: as long as the variable line is not nil collect the value of line into a list.
Thus each line is read and as long there is a line, collect it into a list. If the end of the input stream (which is a file stream) is reached, the (read-line stream nil) form returns nil, the while sees that line is nil. nil is false and thus the loop is terminated. Then the so far collected lines will be returned as the result.
The whole loop form returns the collected list of lines.
Here loop is convenient because:
it conditionally collects a value
we don't need to give the result list a variable name, because there is only one accumulation going on: the single collect ... clause.
the result list is returned in the correct order: first collected item is the first element in the returned list
Background
Generally LOOP is a macro, which is based on the similar FOR macro from Interlisp of the early 70s. There the idea was introduced to have program constructs which could be used like normal english: for item in list collect item. This was called conversational Lisp. Similar ideas are for example used in Applescript, a simple scripting language from Apple. This style was never very popular in Lisp (though there are other macros using it), but the loop macro remained, because it turned out to be useful.
My intuition would tell me that it should've looked something like:
(while line (collect line))
So you might like Shinmera's FOR macro: https://github.com/Shinmera/for
The minimum one need to know is for… over, where over is generic, it works with many data types.
(for:for ((a over '(1 2 3))
(b over #(a b c))
(c over (for:for ((table as (make-hash-table)))
(setf (gethash (random 10) table) (random 10)) (repeat 3)))
(d over *package*)
(e over *random-state*)
(f over (directory (merge-pathnames "*.*" (user-homedir-pathname))))
(g over (make-string-input-stream "Hi!")))
(print (list a b c d e f g)))
I was wondering how bad would be the impact in the performance of a program migrated to shell script from C.
I have intensive I/O operations.
For example, in C, I have a loop reading from a filesystem file and writing into another one. I'm taking parts of each line without any consistent relation. I'm doing this using pointers. A really simple program.
In the Shell script, to move through a line, I'm using ${var:(char):(num_bytes)}. After I finish processing each line I just concatenate it to another file.
"$out" >> "$filename"
The program does something like:
while read line; do
out="$out${line:10:16}.${line:45:2}"
out="$out${line:106:61}"
out="$out${line:189:3}"
out="$out${line:215:15}"
...
echo "$out" >> "outFileName"
done < "$fileName"
The problem is, C takes like half a minute to process a 400MB file and the shell script takes 15 minutes.
I don't know if I'm doing something wrong or not using the right operator in the shell script.
Edit: I cannot use awk since there is not a pattern to process the line
I tried commenting the "echo $out" >> "$outFileName" but it doesn't gets much better. I think the problem is the ${line:106:61} operation. Any suggestions?
Thanks for your help.
I suspect, based on your description, that you're spawning off new processes in your shell script. If that's the case, then that's where your time is going. It takes a lot of OS resource to fork/exec a new process.
As donitor and Dietrich sugested, I did a little research about the AWK language and, again, as they said, it was a total success. here is a little example of the AWK program:
#!/bin/awk -f
{
option=substr($0, 5, 9);
if (option=="SOMETHING"){
type=substr($0, 80, 1)
if (type=="A"){
type="01";
}else if (type=="B"){
type="02";
}else if (type=="C"){
type="03";
}
print substr($0, 7, 3) substr($0, 49, 8) substr($0, 86, 8) type\
substr($0, 568, 30) >> ARGV[2]
}
}
And it works like a charm. It takes barely 1 minute to process a 500mb file
What's wrong with the C program? Is it broken? Too hard to maintain? Too inflexible? You are more of a Shell than a C expert?
If it ain't broke, don't fix it.
A look at Perl might be an option, too. Easier than C to modify and still speedy I/O; and it's much harder to create useless forks in Perl than in the shell.
If you told us exactly what the C program does, maybe there's a simple and faster-than-light solution with sed, grep, awk or other gizmos in the Unix tool box. In other words, tell us what you actually want to achieve, don't ask us to solve some random problem you ran into while pursuing what you think is a step towards your actual goal.
Alright, one problem with your shell script is the repeated open in echo "$out" >> "outFileName". Use this instead:
while read line; do
echo "${line:10:16}.${line:45:2}${line:106:61}${line:189:3}${line:215:15}..."
done < "$fileName" > "$outFileName"
As an alternative, simply use the cut utility (but note that it doesn't insert the dot after the first part):
cut -c 10-26,45-46,106-166 "$fileName" > "$outFileName"
You get the idea?
What is the best way to read a very large file (like a text file having 100 000 names one on each line) into a list (lazily - loading it as needed) in clojure?
Basically I need to do all sorts of string searches on these items (I do it with grep and reg ex in shell scripts now).
I tried adding '( at the beginning and ) at the end but apparently this method (loading a static?/constant list, has a size limitation for some reason.
There are various ways of doing this, depending on exactly what you want.
If you have a function that you want to apply to each line in a file, you can use code similar to Abhinav's answer:
(with-open [rdr ...]
(doall (map function (line-seq rdr))))
This has the advantage that the file is opened, processed, and closed as quickly as possible, but forces the entire file to be consumed at once.
If you want to delay processing of the file you might be tempted to return the lines, but this won't work:
(map function ; broken!!!
(with-open [rdr ...]
(line-seq rdr)))
because the file is closed when with-open returns, which is before you lazily process the file.
One way around this is to pull the entire file into memory with slurp:
(map function (slurp filename))
That has an obvious disadvantage - memory use - but guarantees that you don't leave the file open.
An alternative is to leave the file open until you get to the end of the read, while generating a lazy sequence:
(ns ...
(:use clojure.test))
(defn stream-consumer [stream]
(println "read" (count stream) "lines"))
(defn broken-open [file]
(with-open [rdr (clojure.java.io/reader file)]
(line-seq rdr)))
(defn lazy-open [file]
(defn helper [rdr]
(lazy-seq
(if-let [line (.readLine rdr)]
(cons line (helper rdr))
(do (.close rdr) (println "closed") nil))))
(lazy-seq
(do (println "opening")
(helper (clojure.java.io/reader file)))))
(deftest test-open
(try
(stream-consumer (broken-open "/etc/passwd"))
(catch RuntimeException e
(println "caught " e)))
(let [stream (lazy-open "/etc/passwd")]
(println "have stream")
(stream-consumer stream)))
(run-tests)
Which prints:
caught #<RuntimeException java.lang.RuntimeException: java.io.IOException: Stream closed>
have stream
opening
closed
read 29 lines
Showing that the file wasn't even opened until it was needed.
This last approach has the advantage that you can process the stream of data "elsewhere" without keeping everything in memory, but it also has an important disadvantage - the file is not closed until the end of the stream is read. If you are not careful you may open many files in parallel, or even forget to close them (by not reading the stream completely).
The best choice depends on the circumstances - it's a trade-off between lazy evaluation and limited system resources.
PS: Is lazy-open defined somewhere in the libraries? I arrived at this question trying to find such a function and ended up writing my own, as above.
Andrew's solution worked well for me, but nested defns are not so idiomatic, and you don't need to do lazy-seq twice: here is an updated version without the extra prints and using letfn:
(defn lazy-file-lines [file]
(letfn [(helper [rdr]
(lazy-seq
(if-let [line (.readLine rdr)]
(cons line (helper rdr))
(do (.close rdr) nil))))]
(helper (clojure.java.io/reader file))))
(count (lazy-file-lines "/tmp/massive-file.txt"))
;=> <a large integer>
You need to use line-seq. An example from clojuredocs:
;; Count lines of a file (loses head):
user=> (with-open [rdr (clojure.java.io/reader "/etc/passwd")]
(count (line-seq rdr)))
But with a lazy list of strings, you cannot do those operations efficiently which require the whole list to be present, like sorting. If you can implement your operations as filter or map then you can consume the list lazily. Otherwise it'll be better to use an embedded database.
Also note that you should not hold on to the head of the list, otherwise the whole list will be loaded in memory.
Furthermore, if you need to do more than one operation, you'll need to read the file again and again. Be warned, laziness can make things difficult sometimes.
see my answer here
(ns user
(:require [clojure.core.async :as async :refer :all
:exclude [map into reduce merge partition partition-by take]]))
(defn read-dir [dir]
(let [directory (clojure.java.io/file dir)
files (filter #(.isFile %) (file-seq directory))
ch (chan)]
(go
(doseq [file files]
(with-open [rdr (clojure.java.io/reader file)]
(doseq [line (line-seq rdr)]
(>! ch line))))
(close! ch))
ch))
so:
(def aa "D:\\Users\\input")
(let [ch (read-dir aa)]
(loop []
(when-let [line (<!! ch )]
(println line)
(recur))))
You might find the iota library useful for working with very large files in Clojure. I use iota sequences all the time when I am applying reducers to large amounts of input, and iota/vec provides random access to files larger than memory by indexing them.