Google Script import txt files from Google Drive - arrays

so I'm trying to write a script to import text files, based on the file names, and the tab names.
The txt files are game server log extracts, 3 different types of logs.
I have managed to get the script to import the text based on the file names (types), but having issues running some simple formatting at the end of a loop before moving to the next type or log.
Basically, the script imports all the text from the log files parses it as a CSV and dumps it to the sheet. The sheet removes duplicates, splits the text into columns etc, they should move on to the next type of log (and switch tab) - repeat.
I'm still picking scripting up so please be gentle....! Any help is greatly appreciated.
function ImportAll() {
var folderID = "my google drive foler"
var folder = DriveApp.getFolderById(folderID);
var ss, sRow, lRow, fRow, csvData, logFile, file, sheet =
SpreadsheetApp.getActive();
var data = [];
var logType = ["admin_", "kill_", "login_"]
var allSheets = ['Admin', 'Kill', 'Login']
for (var k = 0; k < logType.length; k++) {
var files = folder.searchFiles('title contains "' + logType[k] + '"');
while (files.hasNext()) {
file = files.next();
data.push(file.getName())
};
ss = sheet.getSheetByName(allsheets[k])
sRow = ss.getDataRange().getLastRow() + 1;
//Browser.msgBox(data[0])
for (var i = 0; i < data.length; i++) {
logFile = DriveApp.getFilesByName(data[i]).next();
csvData = Utilities.parseCsv(logFile.getBlob().getDataAsString());
fRow = ss.getDataRange().getLastRow() + 1;
ss.getRange(fRow, 1, csvData.length, csvData[0].length).setValues(csvData);
};
lRow = ss.getDataRange().getLastRow();
ss.getRange('A' + sRow + ':A' + lRow).removeDuplicates().activate();
ss.getRange('A' + sRow + ':A' + lRow).splitTextToColumns('-');
ss.getRange('B' + sRow + ':B' + lRow).splitTextToColumns(':');
ss.getRange('B:B').setNumberFormat('HH:mm:ss');
ss.getFilter().remove();
ss.getRange('A1:E' + lRow).createFilter();
var rng = sheet.getRange('D2:D' + lRow)
var rngV = rng.getValues();
var String = "";
for(var i=0;i<rngV.length;i++)
{
String = rngV[i].toString().replace(s, '')
rngV[i] = String // is working
}
rng.setValues(rngV) // NOT WORKING!!!!!!
//sheet.appendRow(data); //throws [L]JavaLang#****
}
};
}
I keep getting thrown "TypeError: Cannot call method "getDataRange" of null." errors, and I've tried a heap of different things to no avail.

Related

How to write a 2D array to a single column in Google Sheets, using Google Apps Script?

I'm trying to write the array to a single column, but despite everything seems right to me, it keeps throwing me an error:
Here's the piece of code:
function getGFTickersData() {
var ss = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("LIST OF STOCKS");
var tickerRng = ss.getRange(2, 1, ss.getLastRow(), 1).getValues();
//var TDSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("TickersData");
var TDSheet = SpreadsheetApp.getActiveSpreadsheet().getSheetByName("Sheet10");
var tickerArr = [];
for (var b = 0; b < tickerRng.length; b++) {
var tickerToArr = [tickerRng[b]];
if (tickerToArr != '') {
var gFinFormula = "=query(googlefinance(" + '"' + tickerToArr + '"' + ",'all shares'!A4,'all shares'!D3,'all shares'!D4,'all shares'!D5)," + '"' + "select *" + '"' + ",1)";
var repeated = [].concat(... new Array(105).fill(tickerToArr))
tickerArr.push(repeated)
}
}
Logger.log(tickerArr[0]);
TDSheet.getRange(TDSheet.getLastRow() + 1, 1, tickerArr.length, 1).setValues(tickerArr);
}
Appreciate any pointers!
From your following replying,
The array is composed of about 200k element, each showing in the array 105 times. I want to write all of them, one on the top of the other in a single column.
How about the following modification?
From:
tickerArr.push(repeated)
To:
tickerArr = tickerArr.concat(repeated);
References:
push()
concat()

Alway get an error "No attribute named 'name' is defined" if try to process multiple lines from csv data feed

In my test script I try to process 100 lines data from csv data feed via following statement:
private val scn = scenario("test_scn").feed(insertFeeder, 100).exec(httpReq)
But I always get an error:
[ERROR] HttpRequestAction - 'httpRequest-1' failed to execute: No attribute named 'name' is defined
Could you please help me to find out the root cause? thank you.
Here is the script:
private val insertFeeder = csv("test_data.csv").queue
private val csvHeader = GeneralUtil.readFirstLine(""test_data.csv"")
private val httpConf = http .baseURL("http://serviceURL") .disableFollowRedirect .disableWarmUp .shareConnections
private var httpReq = http("insert_request") .post("/insert")
for (i <- 0 to 99) {
val paramsInArray = csvHeader.split(",")
for (param <- paramsInArray) {
if (param.equalsIgnoreCase("name")) {
httpReq = httpReq.formParam(("name" + "[" + i +"]").el[String] , "${name}")
}
if (param.equalsIgnoreCase("url")) {
httpReq = httpReq.formParam(("url" + "[" + i +"]").el[String] , "${url}")
}
if (!param.equalsIgnoreCase("name") && !param.equalsIgnoreCase("url")) {
val firstArg = param + "[" + i + "]"
val secondArg = "${" + param + "}"
httpReq = httpReq.formParam(firstArg, secondArg)
}
}
}
private val scn = scenario("test_scn") .feed(insertFeeder, 100) .exec(httpReq)
setUp( scn.inject( constantUsersPerSec(1) during (1200 seconds) ).protocols(httpConf) ).assertions(global.failedRequests.count.lte(5))
And the data in test_data.csv is:
name,url,price,size,gender
image_1,http://image_1_url,100,xl,male
image_2,http://image_2_url,90,m,female
image_3,http://image_3_url,10,s,female
...
image_2000,http://image_2000_url,200,xl,male
By the way, if I process only 1 line, it works well.
Read the document again, and fixed the issue. If feed multiple records all at once, the attribute names will be suffixed from 1.
https://gatling.io/docs/current/session/feeder/#csv-feeders

How to specify where to save string in file

How to save string to file to specify place ? I use path << 'string' to save, but it give it on end of file. In destination to xml(path) file have </databaseChangeLog>. I want to save to file before that word occurs.
There is java solution click, but it is static line. My file will be dynamic, I don't know with line it will be.
def add_to_version() {
def path = new File('C:/groovy/version-1.xml')
def branchId = "Promt"
def lineCount = 0
def count = path.eachLine { line ->
if(line.contains('<include file="' + branchId + '/' + branchId + '.xml" ')){
wordCount++
}else if(lineCount == 1 ){
println "package is there"
}
}
if(lineCount == 0){
path << '<include file="' + branchId + '/' + branchId + '.xml" ' + 'relativeToChangelogFile="true"/>'
}
}
code above do that :
and I want to get xml like that :
you can use xml parser like this:
def add_to_version(String branchId) {
def path = new File('C:/groovy/version-1.xml')
def xml = new XmlParser().parse(path)
xml.appendNode("include", [
file:"${branchId}/${branchId}.xml",
relativeToChangelogFile:"true"
])
groovy.xml.XmlUtil.serialize(xml, path.newOutputStream())
}
this variant will not keep the xml formatting and comments
however xml will be valid

AS2 Extract top 5 from array

Good evening everyone!
Anyone knows how to extract the top 5 numbers (the highest) from an array?
I have an XML with names, surnames and score and want to extract the top 5 to create a chart.
As for now, I am getting all the info in this way:
num = xmlData.childNodes.length;
for (var i = 0; i <= num - 1; i++)
{
names[i] = this.childNodes[i].childNodes[6].firstChild.nodeValue;
surnames[i] = this.childNodes[i].childNodes[5].firstChild.nodeValue;
points[i] = this.childNodes[i].childNodes[0].firstChild.nodeValue;
pointsint[i] = parseInt(speeds[i]);
trace(i + "." + pointsint[i] + " (" + names[i] + " " + surnames[i] + ")");
}
Thank you for any help!
Organize data as generic object entries and Array.sortOn(...) (http://help.adobe.com/en_US/as2/reference/flashlite/WS5b3ccc516d4fbf351e63e3d118ccf9c47f-7ea5.html). I didn't test it but I think the idea is clear:
var num = xmlData.childNodes.length;
var entries = [];
for (var i = 0; i <= num - 1; i++)
{
var anEntry = {};
var aNode = xmlData.childNodes[i];
anEntry['name'] = aNode.childNodes[6].firstChild.nodeValue;
anEntry['surname'] = aNode.childNodes[5].firstChild.nodeValue;
var aSpeed = aNode.childNodes[0].firstChild.nodeValue;
anEntry['speed'] = parseInt(aSpeed);
entries.push(anEntry);
trace(i + "." + anEntry['speed'] + " (" + anEntry['name'] + " " + anEntry['surname'] + ")");
}
entries.sortOn("speed", Array.DESCENDING | Array.NUMERIC);
// Now first five elements of entries Array is what you want.

Reading TDM (Diadem) files from script

My customer is sending TDM/TDX files captured in National Instruments Diadem, which I haven't got. I'm looking for a way to convert the files into .CSV, XLS or .MAT files for analysis in Matlab (without using Diadem or Diadem DLLs!)
The format consists of a well structured XML file (.TDM) and a binary (.TDX), with the .TDM defining how fields are packed as bits in the binary TDX. I'd like to read the files (for use in Matlab and other environments). Does anyone have a general purpose tool or conversion script in for instance Python or Perl (not using the NI DLL's) or directly in Matlab?
I've looked into buying the tool, but didn't like it for anything other than one-time conversion to a compatible file format.
Thanks!
I know this is a little late, but I have a simple library to read TDM/TDX files in Python. It works by parsing the TDM file to figure out the data type, then using NumPy.memmap to open the TDX file. It can then be used like a standard NumPy array. The code is pretty simple, so you could probably implement something similar in Matlab.
Here's the link: https://bitbucket.org/joshayers/tdm_loader
Hope that helps.
Maybe a little too late, but I think there is a simple way to get the data from TDM files: NI provides plug-ins for reading TDM files into Excel and OpenOffice Calc. Having the data in one of these programs you could use the CSV export. Search google for "tdm excel" or "tdm openoffice".
Hope this helps...
Gemue
The following script can convert all variables into 'variable' struct.
CurrDirectory = '...//'; % Path to current directory
fileNametdx = '.../utility/'; % Path to TDX file
%%
% Data type conversion
Dtype.eInt8Usi='int8';
Dtype.eInt16Usi='int16';
Dtype.eInt32Usi='int32';
Dtype.eInt64Usi='int64';
Dtype.eUInt8Usi='uint8';
Dtype.eUInt16Usi='uint16';
Dtype.eUInt32Usi='uint32';
Dtype.eUInt64Usi='uint64';
Dtype.eFloat32Usi='single';
Dtype.eFloat64Usi='double';
%% Read .tdx file Name
wb=waitbar(0,'Reading *.tdx Files');
fileNameTDM = strrep(fileNametdx,'.tdx','.TDM');
%% Read .TDM
tdm=xml2struct(fileNameTDM);
for i=1:numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel)
waitbar((1/numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel))*i,wb,['File ' fileNametdx ' conversion started']);
s1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.local_columns.Text),'"');
usi1=s1(2);
% if condition match untill we get usi2
for j=1:numel(tdm.usi_colon_tdm.usi_colon_data.localcolumn)
usi2=string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.Attributes.id);
if usi1==usi2
%take new usi
s2=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.values.Text),'"');
new_usi1=s2(2);
w1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.datatype.Text),'_');
str_1=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence'));
str_2=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.Attributes.id'));
str_3=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.Attributes.external'));
str_4=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values'));
for k=1:numel(eval(str_1))
new_usi2=string(eval(str_2));
if new_usi1==new_usi2
if isfield(eval(str_4), 'Attributes')
inc_value1=string(eval(str_3));
for m=1:numel(tdm.usi_colon_tdm.usi_colon_include.file.block)
inc_value2=string(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.id);
if inc_value1==inc_value2
% offset=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset)/8);
length = round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.length));
offset1=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset));
value_type = tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.valueType;
m = memmapfile(fullfile(CurrDirectory,fileNametdx),'Offset',offset1,'Format',{Dtype.(value_type) [length 1] 'dat'},'Writable',true,'Repeat',1);
dat=m.Data.dat ;
end
end
else
str_5=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.',char(fieldnames(tdm.usi_colon_tdm.usi_colon_data.string_sequence{1, k}.values))));
dat=eval(str_5)';
end
name_variable = string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.name.Text);
varname = genvarname(char(name_variable));
variable.(varname) = dat;
end
end
end
end
end
waitbar(1,wb,[fileNametdx ' conversion completed']);
pause(1)
close(wb)
delete(fullfile(CurrDirectory,fileNametdx),fullfile(CurrDirectory,fileNameTDM));
%Output Variable is Struct
clearvars -except variable
This script requires following XML parser
function [ s ] = xml2struct( file )
%Convert xml file into a MATLAB structure
% [ s ] = xml2struct( file )
%
% A file containing:
% <XMLname attrib1="Some value">
% <Element>Some text</Element>
% <DifferentElement attrib2="2">Some more text</Element>
% <DifferentElement attrib3="2" attrib4="1">Even more text</DifferentElement>
% </XMLname>
%
% Will produce:
% s.XMLname.Attributes.attrib1 = "Some value";
% s.XMLname.Element.Text = "Some text";
% s.XMLname.DifferentElement{1}.Attributes.attrib2 = "2";
% s.XMLname.DifferentElement{1}.Text = "Some more text";
% s.XMLname.DifferentElement{2}.Attributes.attrib3 = "2";
% s.XMLname.DifferentElement{2}.Attributes.attrib4 = "1";
% s.XMLname.DifferentElement{2}.Text = "Even more text";
%
% Please note that the following characters are substituted
% '-' by '_dash_', ':' by '_colon_' and '.' by '_dot_'
%
% Written by W. Falkena, ASTI, TUDelft, 21-08-2010
% Attribute parsing speed increased by 40% by A. Wanner, 14-6-2011
% Added CDATA support by I. Smirnov, 20-3-2012
%
% Modified by X. Mo, University of Wisconsin, 12-5-2012
if (nargin < 1)
clc;
help xml2struct
return
end
if isa(file, 'org.apache.xerces.dom.DeferredDocumentImpl') || isa(file, 'org.apache.xerces.dom.DeferredElementImpl')
% input is a java xml object
xDoc = file;
else
%check for existance
if (exist(file,'file') == 0)
%Perhaps the xml extension was omitted from the file name. Add the
%extension and try again.
if (isempty(strfind(file,'.xml')))
file = [file '.xml'];
end
if (exist(file,'file') == 0)
error(['The file ' file ' could not be found']);
end
end
%read the xml file
xDoc = xmlread(file);
end
%parse xDoc into a MATLAB structure
s = parseChildNodes(xDoc);
end
% ----- Subfunction parseChildNodes -----
function [children,ptext,textflag] = parseChildNodes(theNode)
% Recurse over node children.
children = struct;
ptext = struct; textflag = 'Text';
if hasChildNodes(theNode)
childNodes = getChildNodes(theNode);
numChildNodes = getLength(childNodes);
for count = 1:numChildNodes
theChild = item(childNodes,count-1);
[text,name,attr,childs,textflag] = getNodeData(theChild);
if (~strcmp(name,'#text') && ~strcmp(name,'#comment') && ~strcmp(name,'#cdata_dash_section'))
%XML allows the same elements to be defined multiple times,
%put each in a different cell
if (isfield(children,name))
if (~iscell(children.(name)))
%put existsing element into cell format
children.(name) = {children.(name)};
end
index = length(children.(name))+1;
%add new element
children.(name){index} = childs;
if(~isempty(fieldnames(text)))
children.(name){index} = text;
end
if(~isempty(attr))
children.(name){index}.('Attributes') = attr;
end
else
%add previously unknown (new) element to the structure
children.(name) = childs;
if(~isempty(text) && ~isempty(fieldnames(text)))
children.(name) = text;
end
if(~isempty(attr))
children.(name).('Attributes') = attr;
end
end
else
ptextflag = 'Text';
if (strcmp(name, '#cdata_dash_section'))
ptextflag = 'CDATA';
elseif (strcmp(name, '#comment'))
ptextflag = 'Comment';
end
%this is the text in an element (i.e., the parentNode)
if (~isempty(regexprep(text.(textflag),'[\s]*','')))
if (~isfield(ptext,ptextflag) || isempty(ptext.(ptextflag)))
ptext.(ptextflag) = text.(textflag);
else
%what to do when element data is as follows:
%<element>Text <!--Comment--> More text</element>
%put the text in different cells:
% if (~iscell(ptext)) ptext = {ptext}; end
% ptext{length(ptext)+1} = text;
%just append the text
ptext.(ptextflag) = [ptext.(ptextflag) text.(textflag)];
end
end
end
end
end
end
% ----- Subfunction getNodeData -----
function [text,name,attr,childs,textflag] = getNodeData(theNode)
% Create structure of node info.
%make sure name is allowed as structure name
name = toCharArray(getNodeName(theNode))';
name = strrep(name, '-', '_dash_');
name = strrep(name, ':', '_colon_');
name = strrep(name, '.', '_dot_');
attr = parseAttributes(theNode);
if (isempty(fieldnames(attr)))
attr = [];
end
%parse child nodes
[childs,text,textflag] = parseChildNodes(theNode);
if (isempty(fieldnames(childs)) && isempty(fieldnames(text)))
%get the data of any childless nodes
% faster than if any(strcmp(methods(theNode), 'getData'))
% no need to try-catch (?)
% faster than text = char(getData(theNode));
text.(textflag) = toCharArray(getTextContent(theNode))';
end
end
% ----- Subfunction parseAttributes -----
function attributes = parseAttributes(theNode)
% Create attributes structure.
attributes = struct;
if hasAttributes(theNode)
theAttributes = getAttributes(theNode);
numAttributes = getLength(theAttributes);
for count = 1:numAttributes
%attrib = item(theAttributes,count-1);
%attr_name = regexprep(char(getName(attrib)),'[-:.]','_');
%attributes.(attr_name) = char(getValue(attrib));
%Suggestion of Adrian Wanner
str = toCharArray(toString(item(theAttributes,count-1)))';
k = strfind(str,'=');
attr_name = str(1:(k(1)-1));
attr_name = strrep(attr_name, '-', '_dash_');
attr_name = strrep(attr_name, ':', '_colon_');
attr_name = strrep(attr_name, '.', '_dot_');
attributes.(attr_name) = str((k(1)+2):(end-1));
end
end
end

Resources