so I'm trying to write a script to import text files, based on the file names, and the tab names.
The txt files are game server log extracts, 3 different types of logs.
I have managed to get the script to import the text based on the file names (types), but having issues running some simple formatting at the end of a loop before moving to the next type or log.
Basically, the script imports all the text from the log files parses it as a CSV and dumps it to the sheet. The sheet removes duplicates, splits the text into columns etc, they should move on to the next type of log (and switch tab) - repeat.
I'm still picking scripting up so please be gentle....! Any help is greatly appreciated.
function ImportAll() {
var folderID = "my google drive foler"
var folder = DriveApp.getFolderById(folderID);
var ss, sRow, lRow, fRow, csvData, logFile, file, sheet =
SpreadsheetApp.getActive();
var data = [];
var logType = ["admin_", "kill_", "login_"]
var allSheets = ['Admin', 'Kill', 'Login']
for (var k = 0; k < logType.length; k++) {
var files = folder.searchFiles('title contains "' + logType[k] + '"');
while (files.hasNext()) {
file = files.next();
data.push(file.getName())
};
ss = sheet.getSheetByName(allsheets[k])
sRow = ss.getDataRange().getLastRow() + 1;
//Browser.msgBox(data[0])
for (var i = 0; i < data.length; i++) {
logFile = DriveApp.getFilesByName(data[i]).next();
csvData = Utilities.parseCsv(logFile.getBlob().getDataAsString());
fRow = ss.getDataRange().getLastRow() + 1;
ss.getRange(fRow, 1, csvData.length, csvData[0].length).setValues(csvData);
};
lRow = ss.getDataRange().getLastRow();
ss.getRange('A' + sRow + ':A' + lRow).removeDuplicates().activate();
ss.getRange('A' + sRow + ':A' + lRow).splitTextToColumns('-');
ss.getRange('B' + sRow + ':B' + lRow).splitTextToColumns(':');
ss.getRange('B:B').setNumberFormat('HH:mm:ss');
ss.getFilter().remove();
ss.getRange('A1:E' + lRow).createFilter();
var rng = sheet.getRange('D2:D' + lRow)
var rngV = rng.getValues();
var String = "";
for(var i=0;i<rngV.length;i++)
{
String = rngV[i].toString().replace(s, '')
rngV[i] = String // is working
}
rng.setValues(rngV) // NOT WORKING!!!!!!
//sheet.appendRow(data); //throws [L]JavaLang#****
}
};
}
I keep getting thrown "TypeError: Cannot call method "getDataRange" of null." errors, and I've tried a heap of different things to no avail.
In my test script I try to process 100 lines data from csv data feed via following statement:
private val scn = scenario("test_scn").feed(insertFeeder, 100).exec(httpReq)
But I always get an error:
[ERROR] HttpRequestAction - 'httpRequest-1' failed to execute: No attribute named 'name' is defined
Could you please help me to find out the root cause? thank you.
Here is the script:
private val insertFeeder = csv("test_data.csv").queue
private val csvHeader = GeneralUtil.readFirstLine(""test_data.csv"")
private val httpConf = http .baseURL("http://serviceURL") .disableFollowRedirect .disableWarmUp .shareConnections
private var httpReq = http("insert_request") .post("/insert")
for (i <- 0 to 99) {
val paramsInArray = csvHeader.split(",")
for (param <- paramsInArray) {
if (param.equalsIgnoreCase("name")) {
httpReq = httpReq.formParam(("name" + "[" + i +"]").el[String] , "${name}")
}
if (param.equalsIgnoreCase("url")) {
httpReq = httpReq.formParam(("url" + "[" + i +"]").el[String] , "${url}")
}
if (!param.equalsIgnoreCase("name") && !param.equalsIgnoreCase("url")) {
val firstArg = param + "[" + i + "]"
val secondArg = "${" + param + "}"
httpReq = httpReq.formParam(firstArg, secondArg)
}
}
}
private val scn = scenario("test_scn") .feed(insertFeeder, 100) .exec(httpReq)
setUp( scn.inject( constantUsersPerSec(1) during (1200 seconds) ).protocols(httpConf) ).assertions(global.failedRequests.count.lte(5))
And the data in test_data.csv is:
name,url,price,size,gender
image_1,http://image_1_url,100,xl,male
image_2,http://image_2_url,90,m,female
image_3,http://image_3_url,10,s,female
...
image_2000,http://image_2000_url,200,xl,male
By the way, if I process only 1 line, it works well.
Read the document again, and fixed the issue. If feed multiple records all at once, the attribute names will be suffixed from 1.
https://gatling.io/docs/current/session/feeder/#csv-feeders
def createlist(json, filename, arraygroup):
filehdl = open(filename, "wb")
for key, value in rulegroup.items():
filehdl.write('#' + value + "\n")
for list in json:
if list['arraygroup'] == value:
filehdl.write(list['name'] + " " + list['surname'] + "\n")
filehdl.close()
#depart1
testname testsurname
testname1 testsurname2
testname3 testsurname3
#depart2
#depart3
Json has 5 names, and 2 names should be places in #depart2, #depart3 base on their correct department.
Hi i have here a code that will create a file and separate the names in json file base on there group, but the 2nd forloop (with json) was not resetting its index, so after 1st loop of the forloop list was stuck.
tnx for answer.^^,
I already resolve the problem by inputing json obj to array.
jsonlist = []
for obj in json:
jsonlist.append(obj)
then change the 2nd loop with jsonlist
for list in jsonlist:
if list['arraygroup'] == value:
filehdl.write(list['name'] + " " + list['surname'] + "\n")
I am trying to read one text file and convert the contents of that file to pig latin on a new file. Here is what I have:
def pl_line(word):
statement = input('enter a string: ')
words = statement.split()
for word in words:
if len(word) <= 1:
print(word + 'ay')
else:
print(word[1:] + word[0] + 'ay')
def pl_file(old_file, new_file):
old_file = input('enter the file you want to read from: ')
new_file = input('enter the file you would like to write to: ')
write_to = open(new_file, 'w')
read_from = open(old_file, 'r')
lines = read_from.readlines()
for line in lines():
line = pl_line(line.strip('\n'))
write_to.write(line + '\n')
read_from.close()
write_to.close()
However, when I run this, I get this error message:
TypeError: 'list' object is not callable
Any ideas of how to improve my code?
Here are some improvements to the actual converter:
_VOWELS = 'aeiou'
_VOWELS_Y = _VOWELS + 'y'
_SILENT_H_WORDS = "hour honest honor heir herb".split()
def igpay_atinlay(word:str, with_vowel:str='yay'):
is_title = False
if word.title() == word:
is_title = True
word = word.lower()
# Default case, strangely, is 'in-yay'
result = word + with_vowel
if not word[0] in _VOWELS and not word in _SILENT_H_WORDS:
for pos in range(1, len(word)):
if word[pos] in _VOWELS:
result = word[pos:] + word[0:pos] + 'ay'
break
if is_title:
result = result.title()
return result
def line_to_pl(line:str, with_vowel:str='yay'):
new_line = ''
start = None
for pos in range(0, len(line)):
if line[pos].isalpha() or line[pos] == "'" or line[pos] == "-":
if start is None:
start = pos
else:
if start is not None:
new_line += igpay_atinlay(line[start:pos], with_vowel=with_vowel)
start = None
new_line += line[pos]
if start is not None:
new_line += igpay_atinlay(line[start:pos], with_vowel=with_vowel)
start = None
return new_line
tests = """
Now is the time for all good men to come to the aid of their party!
Onward, Christian soldiers!
A horse! My kingdom for a horse!
Ng!
Run away!
This is it.
Help, I need somebody.
Oh, my!
Dr. Livingston, I presume?
"""
for t in tests.split("\n"):
if t:
print(t)
print(line_to_pl(t))
You very likely mixed up the assignments to read_fromand write_to, so you're unintentionally trying to read from a file opened only for write access.
My customer is sending TDM/TDX files captured in National Instruments Diadem, which I haven't got. I'm looking for a way to convert the files into .CSV, XLS or .MAT files for analysis in Matlab (without using Diadem or Diadem DLLs!)
The format consists of a well structured XML file (.TDM) and a binary (.TDX), with the .TDM defining how fields are packed as bits in the binary TDX. I'd like to read the files (for use in Matlab and other environments). Does anyone have a general purpose tool or conversion script in for instance Python or Perl (not using the NI DLL's) or directly in Matlab?
I've looked into buying the tool, but didn't like it for anything other than one-time conversion to a compatible file format.
Thanks!
I know this is a little late, but I have a simple library to read TDM/TDX files in Python. It works by parsing the TDM file to figure out the data type, then using NumPy.memmap to open the TDX file. It can then be used like a standard NumPy array. The code is pretty simple, so you could probably implement something similar in Matlab.
Here's the link: https://bitbucket.org/joshayers/tdm_loader
Hope that helps.
Maybe a little too late, but I think there is a simple way to get the data from TDM files: NI provides plug-ins for reading TDM files into Excel and OpenOffice Calc. Having the data in one of these programs you could use the CSV export. Search google for "tdm excel" or "tdm openoffice".
Hope this helps...
Gemue
The following script can convert all variables into 'variable' struct.
CurrDirectory = '...//'; % Path to current directory
fileNametdx = '.../utility/'; % Path to TDX file
%%
% Data type conversion
Dtype.eInt8Usi='int8';
Dtype.eInt16Usi='int16';
Dtype.eInt32Usi='int32';
Dtype.eInt64Usi='int64';
Dtype.eUInt8Usi='uint8';
Dtype.eUInt16Usi='uint16';
Dtype.eUInt32Usi='uint32';
Dtype.eUInt64Usi='uint64';
Dtype.eFloat32Usi='single';
Dtype.eFloat64Usi='double';
%% Read .tdx file Name
wb=waitbar(0,'Reading *.tdx Files');
fileNameTDM = strrep(fileNametdx,'.tdx','.TDM');
%% Read .TDM
tdm=xml2struct(fileNameTDM);
for i=1:numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel)
waitbar((1/numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel))*i,wb,['File ' fileNametdx ' conversion started']);
s1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.local_columns.Text),'"');
usi1=s1(2);
% if condition match untill we get usi2
for j=1:numel(tdm.usi_colon_tdm.usi_colon_data.localcolumn)
usi2=string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.Attributes.id);
if usi1==usi2
%take new usi
s2=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.values.Text),'"');
new_usi1=s2(2);
w1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.datatype.Text),'_');
str_1=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence'));
str_2=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.Attributes.id'));
str_3=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.Attributes.external'));
str_4=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values'));
for k=1:numel(eval(str_1))
new_usi2=string(eval(str_2));
if new_usi1==new_usi2
if isfield(eval(str_4), 'Attributes')
inc_value1=string(eval(str_3));
for m=1:numel(tdm.usi_colon_tdm.usi_colon_include.file.block)
inc_value2=string(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.id);
if inc_value1==inc_value2
% offset=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset)/8);
length = round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.length));
offset1=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset));
value_type = tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.valueType;
m = memmapfile(fullfile(CurrDirectory,fileNametdx),'Offset',offset1,'Format',{Dtype.(value_type) [length 1] 'dat'},'Writable',true,'Repeat',1);
dat=m.Data.dat ;
end
end
else
str_5=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.',char(fieldnames(tdm.usi_colon_tdm.usi_colon_data.string_sequence{1, k}.values))));
dat=eval(str_5)';
end
name_variable = string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.name.Text);
varname = genvarname(char(name_variable));
variable.(varname) = dat;
end
end
end
end
end
waitbar(1,wb,[fileNametdx ' conversion completed']);
pause(1)
close(wb)
delete(fullfile(CurrDirectory,fileNametdx),fullfile(CurrDirectory,fileNameTDM));
%Output Variable is Struct
clearvars -except variable
This script requires following XML parser
function [ s ] = xml2struct( file )
%Convert xml file into a MATLAB structure
% [ s ] = xml2struct( file )
%
% A file containing:
% <XMLname attrib1="Some value">
% <Element>Some text</Element>
% <DifferentElement attrib2="2">Some more text</Element>
% <DifferentElement attrib3="2" attrib4="1">Even more text</DifferentElement>
% </XMLname>
%
% Will produce:
% s.XMLname.Attributes.attrib1 = "Some value";
% s.XMLname.Element.Text = "Some text";
% s.XMLname.DifferentElement{1}.Attributes.attrib2 = "2";
% s.XMLname.DifferentElement{1}.Text = "Some more text";
% s.XMLname.DifferentElement{2}.Attributes.attrib3 = "2";
% s.XMLname.DifferentElement{2}.Attributes.attrib4 = "1";
% s.XMLname.DifferentElement{2}.Text = "Even more text";
%
% Please note that the following characters are substituted
% '-' by '_dash_', ':' by '_colon_' and '.' by '_dot_'
%
% Written by W. Falkena, ASTI, TUDelft, 21-08-2010
% Attribute parsing speed increased by 40% by A. Wanner, 14-6-2011
% Added CDATA support by I. Smirnov, 20-3-2012
%
% Modified by X. Mo, University of Wisconsin, 12-5-2012
if (nargin < 1)
clc;
help xml2struct
return
end
if isa(file, 'org.apache.xerces.dom.DeferredDocumentImpl') || isa(file, 'org.apache.xerces.dom.DeferredElementImpl')
% input is a java xml object
xDoc = file;
else
%check for existance
if (exist(file,'file') == 0)
%Perhaps the xml extension was omitted from the file name. Add the
%extension and try again.
if (isempty(strfind(file,'.xml')))
file = [file '.xml'];
end
if (exist(file,'file') == 0)
error(['The file ' file ' could not be found']);
end
end
%read the xml file
xDoc = xmlread(file);
end
%parse xDoc into a MATLAB structure
s = parseChildNodes(xDoc);
end
% ----- Subfunction parseChildNodes -----
function [children,ptext,textflag] = parseChildNodes(theNode)
% Recurse over node children.
children = struct;
ptext = struct; textflag = 'Text';
if hasChildNodes(theNode)
childNodes = getChildNodes(theNode);
numChildNodes = getLength(childNodes);
for count = 1:numChildNodes
theChild = item(childNodes,count-1);
[text,name,attr,childs,textflag] = getNodeData(theChild);
if (~strcmp(name,'#text') && ~strcmp(name,'#comment') && ~strcmp(name,'#cdata_dash_section'))
%XML allows the same elements to be defined multiple times,
%put each in a different cell
if (isfield(children,name))
if (~iscell(children.(name)))
%put existsing element into cell format
children.(name) = {children.(name)};
end
index = length(children.(name))+1;
%add new element
children.(name){index} = childs;
if(~isempty(fieldnames(text)))
children.(name){index} = text;
end
if(~isempty(attr))
children.(name){index}.('Attributes') = attr;
end
else
%add previously unknown (new) element to the structure
children.(name) = childs;
if(~isempty(text) && ~isempty(fieldnames(text)))
children.(name) = text;
end
if(~isempty(attr))
children.(name).('Attributes') = attr;
end
end
else
ptextflag = 'Text';
if (strcmp(name, '#cdata_dash_section'))
ptextflag = 'CDATA';
elseif (strcmp(name, '#comment'))
ptextflag = 'Comment';
end
%this is the text in an element (i.e., the parentNode)
if (~isempty(regexprep(text.(textflag),'[\s]*','')))
if (~isfield(ptext,ptextflag) || isempty(ptext.(ptextflag)))
ptext.(ptextflag) = text.(textflag);
else
%what to do when element data is as follows:
%<element>Text <!--Comment--> More text</element>
%put the text in different cells:
% if (~iscell(ptext)) ptext = {ptext}; end
% ptext{length(ptext)+1} = text;
%just append the text
ptext.(ptextflag) = [ptext.(ptextflag) text.(textflag)];
end
end
end
end
end
end
% ----- Subfunction getNodeData -----
function [text,name,attr,childs,textflag] = getNodeData(theNode)
% Create structure of node info.
%make sure name is allowed as structure name
name = toCharArray(getNodeName(theNode))';
name = strrep(name, '-', '_dash_');
name = strrep(name, ':', '_colon_');
name = strrep(name, '.', '_dot_');
attr = parseAttributes(theNode);
if (isempty(fieldnames(attr)))
attr = [];
end
%parse child nodes
[childs,text,textflag] = parseChildNodes(theNode);
if (isempty(fieldnames(childs)) && isempty(fieldnames(text)))
%get the data of any childless nodes
% faster than if any(strcmp(methods(theNode), 'getData'))
% no need to try-catch (?)
% faster than text = char(getData(theNode));
text.(textflag) = toCharArray(getTextContent(theNode))';
end
end
% ----- Subfunction parseAttributes -----
function attributes = parseAttributes(theNode)
% Create attributes structure.
attributes = struct;
if hasAttributes(theNode)
theAttributes = getAttributes(theNode);
numAttributes = getLength(theAttributes);
for count = 1:numAttributes
%attrib = item(theAttributes,count-1);
%attr_name = regexprep(char(getName(attrib)),'[-:.]','_');
%attributes.(attr_name) = char(getValue(attrib));
%Suggestion of Adrian Wanner
str = toCharArray(toString(item(theAttributes,count-1)))';
k = strfind(str,'=');
attr_name = str(1:(k(1)-1));
attr_name = strrep(attr_name, '-', '_dash_');
attr_name = strrep(attr_name, ':', '_colon_');
attr_name = strrep(attr_name, '.', '_dot_');
attributes.(attr_name) = str((k(1)+2):(end-1));
end
end
end