There is an array, how correctly to deduce in csv a file?
arr1 = [["A","B"], ["C","D"], ["E","F"], ["G","H"]]
Expected result in csv:
A,B
C,D
E,F
G,H
I do so:
out_file = File.open('file.csv', 'w')
arr1.each_index do |inx|
arr1[inx].each do |val|
out_file.puts val
end
end
But, Prints all in one column:
A
B
C
D
..
If you output to the console through p val, then in each value is / r:
"A\r"
"B\r"
"C\r"
"D\r"
What do I do wrong?
Edit:
result csv Excel
result csv Vim
You are not writing to file.
require 'csv'
CSV.open('file.csv', 'w') do |csv|
arr1.each { |ar| csv << ar }
end
If all you want is to simply print out the CSV string, then you can do it like this:
csv_string = CSV.generate { |csv| array2d.each { |row| csv << row }
Here's a worked example:
> # array2d contains the raw data
> csv_string = CSV.generate { |csv| array2d.each { |row| csv << row } }
> puts csv_string
5014,"John O""Neill",4295,1,Finance Plus
314,"Thomas, Duncan",436,2,Finance Plus
1930,Fraser Smith,436,12,Finance Plus
5057,Fred McDonald,436,12,Finance Plus
Note that it handles double-quotes and commas inside strings.
See: https://ruby-doc.org/stdlib-2.6.1/libdoc/csv/rdoc/CSV.html
It works,
require 'csv'
CSV.open('file.csv', 'w') do |csv|
arr1.each { |ar| csv << ar }
end
But it was necessary to finish in front of it:
arr1.each_index do |inx|
arr1[inx].each do |val|
val.chop!
end
end
To delete a line break \r
and it works:
File.write('file.csv', [["A","B"], ["C","D"], ["E","F"], ["G","H"]].map { |e| e.join(",") }.join($/))
Related
This question already has answers here:
Strange, unexpected behavior (disappearing/changing values) when using Hash default value, e.g. Hash.new([])
(4 answers)
Closed 2 years ago.
Given this CSV file:
date,name,st,code,num
2020-03-25,AB,53,2585,130
2020-03-26,AB,53,3208,151
2020-03-26,BA,35,136,1
2020-03-27,BA,35,191,1
I want to create the following hash with the given data:
{:AB=>[["2020-03-25", "2585"], ["2020-03-26", "3208"]], :BA=>[["2020-03-26", "136"], ["2020-03-27", "191"]]}
I tried this:
require 'csv'
h=Hash.new([])
CSV.foreach('file.csv', headers: true) do |row|
h[row['st']] << [[row['date'], row['code']]]
end
but all I get is an empty hash h.
Let's first create the CSV file.
str =<<~_
date,name,st,code,num
2020-03-25,AB,53,2585,130
2020-03-26,AB,53,3208,151
2020-03-26,BA,35,136,1
2020-03-27,BA,35,191,1
_
FName = 't'
File.write(FName, str)
#=> 120
Now we can simply read the file line-by-line, using CSV::foreach, which, without a block, returns an enumerator, and build the hash as we go along.
require 'csv'
CSV.foreach(FName, headers: true).
with_object(Hash.new { |h,k| h[k] = [] }) do |row,h|
h[row['name'].to_sym] << [row['date'], row['code']]
end
#=> {:AB=>[["2020-03-25", "2585"], ["2020-03-26", "3208"]],
# :BA=>[["2020-03-26", "136"], ["2020-03-27", "191"]]}
I've used the method Hash::new with a block to create a hash h such that if h does not have a key k, h[k] causes h[k] #=> []. That way, h[k] << 123, when h has no key k results in h[k] #=> [123].
Alternatively, one could write:
CSV.foreach(FName, headers: true).with_object({}) do |row,h|
(h[row['name'].to_sym] ||= []) << [row['date'], row['code']]
end
One could also use a converter to convert the values of name to symbols, but some might see that as over-kill here:
CSV.foreach(FName, headers: true,
converters: [->(v) { v.match?(/\p{Alpha}+/) ? v.to_sym : v }] ).
with_object(Hash.new { |h,k| h[k] = [] }) do |row,h|
h[row['name']] << [row['date'], row['code']]
end
There is no need to read a CSV file as a text file or whatever, you can use the CSV file as you intended and address the actual issues at hand.
There are three issues here:
This won't work:
h = Hash.new([])
use this instead:
h = Hash.new {|h, k| h[k] = [] }
See "Strange, unexpected behavior (disappearing/changing values) when using Hash default value, e.g. Hash.new([])" as #jack commented.
You need headers: true because the first row is a headers row in your case.
You are only pushing to the values array. You need to overwrite it like:
h[row['name']] = h[row['name']] << [row['date'], row['code']]
This will work for you:
require 'csv'
h = Hash.new { |h, k| h[k] = [] }
CSV.foreach('file.csv', headers: true) do |row|
h[row['name']] = h[row['name']] << [row['date'], row['code']]
end
h.transform_keys(&:to_sym)
#=> {:AB=>[["2020-03-25", "2585"], ["2020-03-26", "3208"]], :BA=>[["2020-03-26", "136"], ["2020-03-27", "191"]]}
I'm currently writing some code and am using pandas to export all of the data into csv files. My program runs multiple iterations until it has gone through all of the necessary files. Pandas is re-writing one file each iteration but when it moves onto the next file I need it to reset all of the data (I think).
Structure is roughly:
While loop>a few variables are named>program runs>dataframe=(pandas.DataFrame(averagepercentagelist,index=namelist,columns=header))
This part works with no problem for one file. When moving onto the next file, all of the arrays I use are reset and this I think is why pandas gives the error Shape of passed values is (1,1), indices imply (3,1).
Please let me know if I need to explain it better.
EDIT:
While True:
try:
averagepercentagelist=[]
namelist=[]
columns=[]
for row in database:
averagepercentagelist=["12","23"]
namelist=["Name0","Name1"]
columns=["Average percentage"]
dataframe=(pandas.DataFrame(averagepercentagelist,index=namelist,columns=header))
except Exception as e:
print e
break
SNIPPET:
dataframe= (pandas.DataFrame(averagepercentagelist,index=namelist,columns=header))
currentcalculatedatafrane = 'averages' + currentcalculate
dataframeexportpath = os.path.join(ROOT_PATH,'Averages',currentcalculatedatafrane)
dataframe.to_csv(dataframeexportpath)
FULL PROGRAM SO FAR:
import csv
import os
import re
import pandas
import tkinter as tk
from tkinter import messagebox
from os.path import isfile, join
from os import listdir
import time
ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
indexforcalcu=0
line_count=0
testlist=[]
namelist=[]
header=['Average Percentage']
def clearvariables():
indexforcalcu=0
testlist=[]
def findaverageofstudent(findaveragenumber,numoftests):
total=0
findaveragenumber = findaveragenumber/numoftests
findaveragenumber = round(findaveragenumber, 1)
return findaveragenumber
def removecharacters(nameforfunc):
nameforfunc=str(nameforfunc)
elem=re.sub("[{'}]", "",nameforfunc)
return elem
def getallclasses():
onlyfiles = [f for f in listdir(ROOT_PATH) if isfile(join(ROOT_PATH, f))]
onlyfiles.remove("averagecalculatorv2.py")
return onlyfiles
def findaveragefunc():
indexforcalcu=-1
while True:
try:
totaltests=0
line_count=0
averagepercentagelist=[]
indexforcalcu=indexforcalcu+1
allclasses=getallclasses()
currentcalculate=allclasses[indexforcalcu]
classpath = os.path.join(ROOT_PATH, currentcalculate)
with open(classpath) as csv_file:
classscoredb = csv.reader(csv_file, delimiter=',')
for i, row in enumerate(classscoredb):
if line_count == 0:
while True:
try:
totaltests=totaltests+1
rowreader= {row[totaltests]}
except:
totaltests=totaltests-1
line_count = line_count + 1
break
else:
calculating_column_location=1
total=0
while True:
try:
total = total + int(row[calculating_column_location])
calculating_column_location = calculating_column_location + 1
except:
break
i=str(i)
name=row[0]
cleanname=removecharacters(nameforfunc=name)
namelist.append(cleanname)
findaveragenumbercal=findaverageofstudent(findaveragenumber=total,numoftests=totaltests)
averagepercentagelist.append(findaveragenumbercal)
line_count = line_count + 1
dataframe= (pandas.DataFrame(averagepercentagelist,index=namelist,columns=header))
currentcalculatedatafrane = 'averages' + i + currentcalculate
dataframeexportpath = os.path.join(ROOT_PATH,'Averages',currentcalculatedatafrane)
dataframe.to_csv(dataframeexportpath)
i=int(i)
except Exception as e:
print("ERROR!!!!!!!!!!!!!!!!!!!!!!!!!!!!\n\n",e)
break
def makenewclass():
global newclassname
getclassname=str(newclassname.get())
if getclassname == "":
messagebox.showerror("Error","The class name you have entered is invalid.")
else:
classname = getclassname + ".csv"
with open(classname, mode='w') as employee_file:
classwriter = csv.writer(employee_file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)
classwriter.writerow(["Name","Test 1"])
root=tk.Tk()
root.title("Test result average finder")
findaveragebutton=tk.Button(root,text="Find Averages",command=findaveragefunc())
findaveragebutton.grid(row=2,column=2,padx=(10, 10),pady=(0,10))
classnamelabel=tk.Label(root, text="Class name:")
classnamelabel.grid(row=1, column=0,padx=(10,0),pady=(10,10))
newclassname = tk.Entry(root)
newclassname.grid(row=1,column=1,padx=(10, 10))
newclassbutton=tk.Button(root,text="Create new class",command=makenewclass)
newclassbutton.grid(row=1,column=2,padx=(0, 10),pady=(10,10))
root.mainloop()
Thanks in advance,
Sean
Use:
import glob, os
import pandas as pd
ROOT_PATH = os.path.dirname(os.path.abspath(__file__))
#extract all csv files to list
files = glob.glob(f'{ROOT_PATH}/*.csv')
print (files)
#create new folder if necessary
new = os.path.join(ROOT_PATH,'Averages')
if not os.path.exists(new):
os.makedirs(new)
#loop each file
for f in files:
#create DataFrame and convert first column to index
df = pd.read_csv(f, index_col=[0])
#count average in each row, rond and create one colum DataFrame
avg = df.mean(axis=1).round(1).to_frame('Average Percentage')
#remove index name if nncessary
avg.index.name = None
print (avg)
#create new path
head, tail = os.path.split(f)
path = os.path.join(head, 'Averages', tail)
print (path)
#write DataFrame to csv
avg.to_csv(path)
I have collected some data and saved it to an array of hashes in the form of:
info = [
{'Name' => 'Alex', 'Tel' => 999, 'E-mail' => "bla#bla.com"},
{'Name' => 'Ali', 'Tel' => 995, 'E-mail' => "ali#bla.com"}
# ...
]
But not all information is always there:
{'Name' => 'GuyWithNoTelephone', 'E-mail' => "poor#bla.com"}
I want to turn this information into a CSV file. Here is what I tried:
def to_csv (info)
CSV.open("sm-data.csv", "wb") do |csv|
csv << ["Name", "Tel", "E-mail"]
info.each do |person|
csv << person.values
When I try this, the format of the table is not correct, and say, if a telephone number is missing, then the e-mail of that person appears at telephone column.
How can I selectively write this info to the CSV file, i.e how can I tell to skip columns if a value is missing?
But sadly when I try this, the format of the table is not correct, and say, if a telephone number is missing, then the e-mail of that person appears at telephone column.
That's because you are omitting the telephone number in that case, providing just 2 of the 3 values. To fix this, you simply have to provide all 3 values, even if they don't exist in the hash:
csv << ['Name', 'Tel', 'E-mail']
info.each do |person|
csv << [person['Name'], person['Tel'], person['E-Mail']]
end
or via Hash#values_at:
csv << ['Name', 'Tel', 'E-mail']
info.each do |person|
csv << person.values_at('Name', 'Tel', 'E-Mail')
end
For:
{'Name' => 'GuyWithNoTelephone', 'E-mail' => "poor#bla.com"}
this results in:
csv << ['GuyWithNoTelephone', nil, 'poor#bla.com']
which generates this output: (note the two commas, denoting an empty field in-between)
"GuyWithNoTelephone,,poor#bla.com\n"
Try this,
def to_csv(csv_filename="sm-data.csv")
# Get all unique keys into an array:
keys = info.map(&:keys).inject(&:|)
CSV.open(csv_filename, "wb") do |csv|
csv << keys
info.each do |hash|
# fetch values at keys location, inserting null if not found.
csv << hash.values_at(*keys)
end
end
end
Simple as this:
path = "data/backtest-results.csv"
CSV.open(path, "wb") do |csv|
csv << ["Asset", "Strategy", "Profit"]
result_array.each do |p|
csv << p.map { |key, value| value }
end
end
use "(File.file?(path) ? "ab" : "wb")" rather than "wb" if you want to continue adding as the new data comes
I Need to parse a String into Array of Arrays. String is splited in Arrays by "\n" char. And each of that Array is splited by "," or ";" signs.
Example: 4;5\n6,7 -----> [[4,5][6,7]]
import qualified Text.Parsec as P (char,runP,noneOf,many,(<|>),eof)
import Text.ParserCombinators.Parsec
import Text.Parsec.String
import Text.Parsec.Char
import Text.PrettyPrint.HughesPJ
import Data.Maybe
newtype CSV = CSV [Row] deriving (Show,Eq)
type Row = [String]
parseCSV :: Parser CSV
parseCSV = do
return $ CSV (endBy (sepBy (many (noneOf ",\n")) (Text.Parsec.Char.char ',')) (Text.Parsec.Char.char '\n'))
runParsec :: Parser a -> String -> Maybe a
runParsec parser input = case P.runP parser () "" input of
Left _ -> Nothing
Right a -> Just a
But when I try to run the Code I get error because of wrong data types
Here is a solution that parses runParsec parseCSV "4;5\n6,7\n" into Just (CSV [["4","5"],["6","7"]]).
parseCSV :: Parser CSV
parseCSV = CSV <$> csv
where
csv = row `endBy` char '\n'
row = many (noneOf ",;\n") `sepBy` oneOf ",;"
I have a CSV file with mixed records (both double and string) and a header line of strings in the first row.
I have written the following code to import this csv file into a cell array in MatLAB
%Determine Formatspec for CSV file
fid = fopen('URM_sample.csv')
C = textscan(fid, '%s', 'Delimiter', '\n')
str = C{1}{2,:}
nstr = textscan(str,'%s','delimiter',',')
fspec = ''
for i = 1:length(nstr{1}) % Check each string
if isempty(str2num(nstr{1}{i})) % This string is text
fspec = [fspec ' %s'];
else % This string is a number
fspec = [fspec ' %f'];
end
end
B = textscan(fid,fspec,'HeaderLines',1,'Delimiter',',')
Unfortunatley when I open B all I get is an empty 1x24 cell array, where the original csv is 201x24 (contains 201 records).... Any idea as to what I am missing here? Thanks
I believe you are using the 'delimiter' parameter incorrectly, and also specifying formatSpec incorrectly. Try
fid = fopen('URM_sample.csv')
nstr = textscan(fid,'%s %s ... %s','delimiter',',',
'HeaderLines', 1, 'EndOfLine', '\n')
I believe you need to repeat %s 24 times in the formatSpec parameter.