How to retrieve file path from an open IOStream in Julia - file

I open some file:
myfilepath = "/home/colin/somefile.csv"
fid = open(myfilepath, "r")
I would like to retrieve the file path from fid. I can use:
fid.name
and this will return something like:
"<file /home/colin/somefile.csv>
However, as near as I can tell, this isn't documented, so I've got no guarantee this solution is future-proof.
So, my question is, is there a safe method to retrieve the file-path of an open IOStream from the IOStream itself?

Currently no such function exists in base Julia. This can be determined by looking at the complete list of functions defined for the type IOStream:
julia> methodswith(IOStream)
[1] crc32c(io::IOStream) in CRC32c at /Applications/Julia-1.6.app/Contents/Resources/julia/share/julia/stdlib/v1.6/CRC32c/src/CRC32c.jl:50
[2] crc32c(io::IOStream, crc::UInt32) in CRC32c at /Applications/Julia-1.6.app/Contents/Resources/julia/share/julia/stdlib/v1.6/CRC32c/src/CRC32c.jl:50
[3] bytesavailable(s::IOStream) in Base at iostream.jl:377
[4] close(s::IOStream) in Base at iostream.jl:61
[5] eof(s::IOStream) in Base at iostream.jl:232
[6] fd(s::IOStream) in Base at iostream.jl:55
[7] filesize(s::IOStream) in Base at iostream.jl:222
[8] flush(s::IOStream) in Base at iostream.jl:66
[9] isopen(s::IOStream) in Base at iostream.jl:59
[10] isreadable(s::IOStream) in Base at iostream.jl:75
[11] iswritable(s::IOStream) in Base at iostream.jl:73
[12] peek(s::IOStream, ::Type{UInt8}) in Base at iostream.jl:568
[13] position(s::IOStream) in Base at iostream.jl:216
[14] read(s::IOStream) in Base at iostream.jl:516
[15] read(s::IOStream, ::Type{UInt8}) in Base at iostream.jl:396
[16] read(s::IOStream, T::Union{Type{Int16}, Type{Int32}, Type{Int64}, Type{UInt16}, Type{UInt32}, Type{UInt64}}) in Base at iostream.jl:405
[17] read(s::IOStream, ::Type{Float16}) in Base at iostream.jl:417
[18] read(s::IOStream, ::Type{Float32}) in Base at iostream.jl:418
[19] read(s::IOStream, ::Type{Float64}) in Base at iostream.jl:419
[20] read(s::IOStream, nb::Integer; all) in Base at iostream.jl:557
[21] readavailable(s::IOStream) in Base at iostream.jl:379
[22] readbytes!(s::IOStream, b::Union{SubArray{UInt8, var"#s828", var"#s827", I, true} where {var"#s828", var"#s827"<:(Array{UInt8, N} where N), I<:Union{Tuple{Vararg{Real, N} where N}, Tuple{AbstractUnitRange, Vararg{Any, N} where N}}}, Array{UInt8, N} where N}) in Base at iostream.jl:509
[23] readbytes!(s::IOStream, b::Union{SubArray{UInt8, var"#s826", var"#s825", I, true} where {var"#s826", var"#s825"<:(Array{UInt8, N} where N), I<:Union{Tuple{Vararg{Real, N} where N}, Tuple{AbstractUnitRange, Vararg{Any, N} where N}}}, Array{UInt8, N} where N}, nb; all) in Base at iostream.jl:509
[24] readline(s::IOStream; keep) in Base at iostream.jl:444
[25] readuntil(s::IOStream, delim::UInt8; keep) in Base at iostream.jl:435
[26] redirect_stderr(handle::Union{IOStream, Base.LibuvStream}) in Base at stream.jl:1148
[27] redirect_stdin(handle::Union{IOStream, Base.LibuvStream}) in Base at stream.jl:1148
[28] redirect_stdout(handle::Union{IOStream, Base.LibuvStream}) in Base at stream.jl:1148
[29] seek(s::IOStream, n::Integer) in Base at iostream.jl:127
[30] seekend(s::IOStream) in Base at iostream.jl:161
[31] show(io::IO, s::IOStream) in Base at iostream.jl:35
[32] skip(s::IOStream, delta::Integer) in Base at iostream.jl:184
[33] stat(s::IOStream) in Base at iostream.jl:57
[34] take!(s::IOStream) in Base at iostream.jl:432
[35] truncate(s::IOStream, n::Integer) in Base at iostream.jl:106
[36] unsafe_read(s::IOStream, p::Ptr{UInt8}, nb::UInt64) in Base at iostream.jl:422
[37] unsafe_write(s::IOStream, p::Ptr{UInt8}, nb::UInt64) in Base at iostream.jl:371
[38] write(s::IOStream, b::UInt8) in Base at iostream.jl:366
The closest of any of these to what you are looking for is stat, which can give you the inode of the underlying file, but not the file path per se.
help?> stat
search: stat lstat #static startswith stacktrace StackTraces #fastmath ignorestatus InvalidStateException abstract AbstractSet AbstractChar AbstractDict abstract type AbstractFloat AbstractArray AbstractRange AbstractMatch
stat(file)
Returns a structure whose fields contain information about the file. The fields of the structure are:
Name Description
––––––– ––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––––
size The size (in bytes) of the file
device ID of the device that contains the file
inode The inode number of the file
mode The protection mode of the file
nlink The number of hard links to the file
uid The user id of the owner of the file
gid The group id of the file owner
rdev If this file refers to a device, the ID of the device it refers to
blksize The file-system preferred block size for the file
blocks The number of such blocks allocated
mtime Unix timestamp of when the file was last modified
ctime Unix timestamp of when the file was created

Related

`unsafe_convert` fails for array views with `Vector{U}` for indices instead of `UnitRange{U}` in Julia

I noticed the following error when handling views with indices given by a vector of indices instead of a UnitRange.
c = rand(10)
c1 = view(c, [1, 3])
c2 = view(c1, 2:2)
Base.unsafe_convert(Ptr{Float64}, c2)
now returns
ERROR: conversion to pointer not defined for SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false}
Stacktrace:
[1] error(s::String)
# Base ./error.jl:33
[2] unsafe_convert(#unused#::Type{Ptr{Float64}}, a::SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false})
# Base ./pointer.jl:67
[3] top-level scope
# REPL[6]:1
They both result in the same error for me (i.e., taking pointers is just explicitely disallowed for non-contiguous views -- indicated by the false type parameter):
julia> Base.unsafe_convert(Ptr{Float64}, c1)
ERROR: conversion to pointer not defined for SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false}
Stacktrace:
[1] error(s::String)
# Base ./error.jl:33
[2] unsafe_convert(#unused#::Type{Ptr{Float64}}, a::SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false})
# Base ./pointer.jl:67
[3] top-level scope
# REPL[8]:1
julia> Base.unsafe_convert(Ptr{Float64}, c2)
ERROR: conversion to pointer not defined for SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false}
Stacktrace:
[1] error(s::String)
# Base ./error.jl:33
[2] unsafe_convert(#unused#::Type{Ptr{Float64}}, a::SubArray{Float64, 1, Vector{Float64}, Tuple{Vector{Int64}}, false})
# Base ./pointer.jl:67
[3] top-level scope
# REPL[9]:1
Which makes sense, as a pointer to a non-contiguous chunk is not really meaningful (the information about the selected indices is lost).
You can get a pointer for a range, though:
julia> Base.unsafe_convert(Ptr{Float64}, view(c, 2:2))
Ptr{Float64} #0x00007f1feaa58b88

ERROR: LoadError: MethodError: no method matching Array(::Type{Int8}, ::Int64, ::Int64, ::Int64)

I am using Julia v0.7.0 to upgrade a package written for Julia v0.5.0. I am stuck at the following error:
ERROR: LoadError: MethodError: no method matching Array(::Type{Int8}, ::Int64, ::Int64, ::Int64)
Closest candidates are:
Array(!Matched::LinearAlgebra.UniformScaling, ::Integer, ::Integer) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/LinearAlgebra/src/uniformscaling.jl:329
Array(::Any) where T<:AbstractArray at abstractarray.jl:22
Stacktrace:
[1] CellularAutomata.CA2d(::Array{Int64,1}, ::Array{Int64,1}, ::Array{Int64,2}, ::Int64, ::Int64, ::Int64) at /home/jafar_isbarov/.julia/dev/CellularAutomata/src/2dim.jl:22
[2] CellularAutomata.CA2d(::Array{Int64,1}, ::Array{Int64,1}, ::Array{Int64,2}, ::Int64) at /home/jafar_isbarov/.julia/dev/CellularAutomata/src/2dim.jl:21
[3] turbine_test() at /home/jafar_isbarov/Documents/projects/CellularAutomata.jl/test/conway_test.jl:57
[4] top-level scope at none:0
[5] include at ./boot.jl:317 [inlined]
[6] include_relative(::Module, ::String) at ./loading.jl:1038
[7] include(::Module, ::String) at ./sysimg.jl:29
[8] exec_options(::Base.JLOptions) at ./client.jl:239
[9] _start() at ./client.jl:432
in expression starting at /home/jafar_isbarov/Documents/projects/CellularAutomata.jl/test/runtests.jl:3
Here is the line that causes it:
cells = Array{Int8}(h, w, gen)
And here is the entire block:
mutable struct CA2d
#User given values
k::Int #Number of states
r::Int #r-nearest neigbors
#Internal values
cells::Array{Int8, 3}
function CA2d(B::Array{Int,1},
S::Array{Int,1},
init::Array{Int,2},
gen::Int,
k::Int=2,
r::Int=1)
h, w = size(init)
cells = Array{Int8}(h, w, gen) #Syntax A(T, dims) is deprecated
cells[:, :, 1] = Array{Int8}(init[:, :])
for g = 2:gen
for i = 1:h, j = 1:w
cc = -cells[i, j, g-1]
for p = (i-r):(i+r), q = (j-r):(j+r)
#Cyclic boundary conditions
if p < 1; p = h-p; end
if p > h; p = p-h; end
if q < 1; q = w-q; end
if q > w; q = q-w; end
cc += cells[p, q, g-1]
end
cells[i, j, g] = eval_rule(cc, cells[i, j, g-1], B, S)
end #hw ij
end #gen
new(k, r, cells)
end
end
I have checked and know that h, w, and gen are indeed integers.
You can find the entire repository here.
Edit:
When I replaced cells = Array{Int8}(h, w, gen) with cells = Array{Int8}(undef, h, w, gen), I got the following error:
ERROR: LoadError: MethodError: no method matching Array(::Type{Int8}, ::Int64, ::Int64, ::Int64)
Closest candidates are:
Array(!Matched::LinearAlgebra.UniformScaling, ::Integer, ::Integer) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v0.7/LinearAlgebra/src/uniformscaling.jl:329
Array(::Any) where T<:AbstractArray at abstractarray.jl:22
Stacktrace:
[1] CellularAutomata.CA2d(::Array{Int64,1}, ::Array{Int64,1}, ::Array{Int64,2}, ::Int64, ::Int64, ::Int64) at /home/jafar_isbarov/.julia/dev/CellularAutomata/src/2dim.jl:22
[2] CellularAutomata.CA2d(::Array{Int64,1}, ::Array{Int64,1}, ::Array{Int64,2}, ::Int64) at /home/jafar_isbarov/.julia/dev/CellularAutomata/src/2dim.jl:21
[3] turbine_test() at /home/jafar_isbarov/Documents/projects/CellularAutomata.jl/test/conway_test.jl:57
[4] top-level scope at none:0
[5] include at ./boot.jl:317 [inlined]
[6] include_relative(::Module, ::String) at ./loading.jl:1038
[7] include(::Module, ::String) at ./sysimg.jl:29
[8] exec_options(::Base.JLOptions) at ./client.jl:239
[9] _start() at ./client.jl:432
in expression starting at /home/jafar_isbarov/Documents/projects/CellularAutomata.jl/test/runtests.jl:3
to make an un-initialized array of eltype Int8:
Array{Int8}(undef, h, w, gen)

"ERROR: LoadError: BoundsError: attempt to access 0-element Array{Candidate,1} at index [1]" when trying to fill the empty array of a composite type

I get an error when trying to populate an empty array with input from a user.
const max = 9 # a maximal number of candidates
# Let us define a composite type for the candidates in our elections
mutable struct Candidate
name::String
votes::Int8
end
candidates = Candidate[]
i = 1
while i < max
println("Name of the candidate: ?")
candidates[i].name = readline();
println("Votes on him: ?")
candidates[i].votes = parse(Int8, readline(), base=10);
println("Thank you, let us move to the next candidate.")
global i = i +1
end
After ("Name of the candidate: ?") being displayed, I get the following error:
ERROR: LoadError: BoundsError: attempt to access 0-element Array{Candidate,1} at index [1]
Stacktrace:
[1] getindex(::Array{Candidate,1}, ::Int64) at ./array.jl:731
[2] top-level scope at /home/jerzy/ComputerScience/SoftwareDevelopment/MySoftware/MyJulia/plurality.jl:18 [inlined]
[3] top-level scope at ./none:0
[4] include at ./boot.jl:317 [inlined]
[5] include_relative(::Module, ::String) at ./loading.jl:1044
[6] include(::Module, ::String) at ./sysimg.jl:29
[7] exec_options(::Base.JLOptions) at ./client.jl:266
[8] _start() at ./client.jl:425
in expression starting at /home/jerzy/ComputerScience/SoftwareDevelopment/MySoftware/MyJulia/plurality.jl:16
Alternatively, using
candidates = Array{Candidate}(undef, 0)
instead of
candidates = Candidate[]
results in:
ERROR: LoadError: UndefRefError: access to undefined reference
Apologies for being such a newbie. I relied on what I read in this wikibook. Would you refer me to some more reading?
You are almost correct, issue is that the length of your array is 0 (you can verify it with length(candidates)), so Julia complains when you are trying to set non-zero indexed elements of an array with candidates[i]. If you do not know length of your array in advance, then you should use push! function.
const max_candidates = 9 # a maximal number of candidates
while i < max_candidates
println("Name of the candidate: ?")
name = readline();
println("Votes on him: ?")
votes = parse(Int, readline());
push!(candidates, Candidate(name, votes))
println("Thank you, let us move to the next candidate.")
global i = i + 1
end
I've changed here max to max_candidates because max interferes with Base max function.
If you know number of candidates, you can use candidates = Vector{Candidate}(undef, max_candidates) form of initialization, take notice of max_candidates instead of 0, because you should allocate vector of necessary length.
candidates = Vector{Candidate}(undef, max_candidates)
for i in 1:max_candidates
println("Name of the candidate: ?")
name = readline();
println("Votes on him: ?")
votes = parse(Int, readline());
candidates[i] = Candidate(name, votes)
println("Thank you, let us move to the next candidate.")
end
Take notice, that I changed while to for it may be or may be not useful in your case, but at least that lets you remove global i = i + 1 line.
If the last version suits you, then you could probably remove mutable from struct definition, it is usually better for performance.

lldb: Displaying array as hex

In lldb is it possible to display a uint8_t array of bytes in a concatenated hex?
For instance the default behaviour for displaying the uint256 type described here is
{
[14] = '['
[15] = '\xbf'
[16] = '('
[17] = '\xc3'
[18] = 'O'
[19] = ':'
[20] = '^'
[21] = '3'
[22] = '*'
[23] = '\x1f'
[24] = '\xc7'
[25] = '\xb2'
[26] = '\xb7'
[27] = '<'
[28] = '\xf1'
[29] = '\x88'
[30] = '\x91'
[31] = '\x0f'
}
by running type summary add --summary-string "${var.data[0-31]%x}" uint256 I can display it like this:
[0x06,0x22,0x6e,0x46,0x11,0x1a,0x0b,0x59,0xca,0xaf,0x12,0x60,0x43,0xeb,0x5b,0xbf,0x28,0xc3,0x4f,0x3a,0x5e,0x33,0x2a,0x1f,0xc7,0xb2,0xb7,0x3c,0xf1,0x88,0x91,0x0f]
but I wanted something like this:
0x06226e46111a0b59caaf126043eb5bbf28c34f3a5e332a1fc7b2b73cf188910f
I can't think of a way to do this with the summary string formatters. We do some special purpose magic like this for char[] but that's behind the %s specifier which you don't want.
However, it is fairly easy to write a python summary to do this. Here's a simple example for the type uint8_t [32]:
def unvector_32 (valobj,internal_dict):
str = "0x"
for i in range(0,31):
str += ("%x"%(valobj.GetChildAtIndex(i).GetValueAsUnsigned())).zfill(2)
return str
def __lldb_init_module(debugger, internal_dict):
debugger.HandleCommand('type category define -e my_formatters')
debugger.HandleCommand('type summary add -w my_formatters -F %s.unvector_32 "uint8_t [32]" -v'%(__name__))
Put that in a .py file somewhere (I put all these in ~/.lldb) and do:
(lldb) com scr imp ~/.lldb/my_formatters.py
either in your lldb session or in your ~/.lldbinit. Then you will see:
(lldb) fr v data
(uint8_t [32]) data = 0x01020304050600000000000000000000000000000000000000000000000000
(lldb)
I think that's roughly the output you wanted. To apply this to the uint256 class you can change the match type to "uint256" when you add the summary, and have the summary function first get the data member using GetChildMemberWithName("data") then print the individual vector elements. You could also use a regex match against uint([0-9]+) and then get the number of elements in data with the GetNumChildren call if you want the formatter to be more general...
BTW, I find it handy to put all my formatters in a category so I can list them easily (type summary list -w my_formatters) and turn them all off easily if I've messed something up...
More on the SB API's is here:
http://lldb.llvm.org/python_reference/index.html
You can change formate of lldb debugger to another format
type format add -f bytes uint8_t
Bytes format to use
uiint8_t type to apply format on
For more details Link :- https://lldb.llvm.org/use/variable.html

Reducer for non-parallel for loops/multiline comprehensions

Julia has a parallel macro for for loops, which allows things like:
s = #sync #parallel vcat for i in 1:9
k = iseven(i) ? i÷2 : 3i+1
k^2
end
and since the reducer specified is vcat, we get back an array of numbers.
Is it possible to do something like this with a normal for loop (without having to explicitly initialize and push! into the array)?
Since I'm only looking to reduce using vcat, another way to ask this question is: is there a neat readable multiline form of array comprehensions? It's possible to stretch to usual comprehension syntax like this:
s = [
(k = iseven(i) ? i÷2 : 3i+1;
k^2)
for i in 1:9
]
but that seems messy and less readable compared to the #parallel vcat for syntax. Is there a better way of doing multiline comprehensions?
Extending on #Gnimuc's answer, I think mapreduce plus do-syntax is pretty nice:
julia> mapreduce(vcat, 1:9) do i
k = iseven(i) ? i÷2 : 3i+1
k^2
end
9-element Array{Int64,1}:
16
1
100
4
256
9
484
16
784
The short answer is to write multiline functions(or do-blocks as #phg reminds) with a single line array comprehension or map/mapreduce:
s = [
(k = iseven(i) ? i÷2 : 3i+1;
k^2)
for i in 1:9
]
This example is pure comprehension, no reducer is involved. Array comprehension is usually written in one line, for example, s = [iseven(i) ? i÷2 : 3i+1 |> x->x^2 for i in 1:9]. As #phg suggested, multi-line functions can be enclosed in a do-block:
julia> map(1:9) do x
k = iseven(x) ? x÷2 : 3x+1
k^2
end
However, no reducer such as vcat is needed in this case, but if the output of f in the above example is a vector:
julia> function f(x)
k = iseven(x) ? x÷2 : 3x+1
[k^2]
end
f (generic function with 1 method)
julia> s = [f(i) for i in 1:9]
9-element Array{Array{Int64,1},1}:
[16]
[1]
[100]
[4]
[256]
[9]
[484]
[16]
[784]
array comprehension will give you an array of vectors. This time you need to use mapreduce instead:
julia> mapreduce(f, vcat, 1:9)
9-element Array{Int64,1}:
16
1
100
4
256
9
484
16
784

Resources