I have two sets of arrays stored in a file and I need to extract values one by one and compare them. I am using this code but does look like I am doing correctly.
# First Dataset
File.foreach(file_set_a) do |data_a|
data_array_a = data_a.split("\t")
#file_name_a = data_array_a[0]
#file_ext_a = data_array_a[1]
# Second Dataset
File.foreach(file_set_b) do |data_b|
data_array_b = data_b.split("\t")
#file_name_b = data_array_b[0]
#file_ext_b = data_array_b[1]
#Compare
#file_name_a == #file_name_b
end
end
The problem is, I cannot go back and extract the next values in the set A when I enter the set B. Any suggestions?
First, convert those 2 files into two separated data arrays
lines_array_a = File.readlines(file_set_a)
lines_array_b = File.readlines(file_set_b)
I am assuming both of the array size will be same. Now run a loop and get the items from both array to compare them.
for i in 0..(lines_array_a.count - 1) do
data_array_a = lines_array_a[i].split("\t")
#file_name_a = data_array_a[0]
#file_ext_a = data_array_a[1]
data_array_b = lines_array_b[i].split("\t")
#file_name_b = data_array_b[0]
#file_ext_b = data_array_b[1]
#file_name_a == #file_name_b
end
I have a simple bit of code written in Octave, all is does is open 3 files (one at a time) and scans each line for one of two strings, if it finds the matching string it extracts the number which follows the string and appends that number to an array.
clear
names = dir('*txt');
rec = [];
emit = [];
for i = 1:length(names)
name = names(i).name;
fid = fopen(name);
while !(feof(fid))
tline = fgets(fid);
if !(isempty(regexp(tline,'reception: ')))
offset = regexp(tline,'reception: ');
val = str2double(strtok(tline(offset+11:end)));
rec(end+1) = val;
elseif !(isempty(regexp(tline,'emission: ')))
offset = regexp(tline,'emission: ');
val = str2double(strtok(tline(offset+10:end)));
emit(end+1) = val;
end
end
length(rec)
end
fn = strcat(name,'REC.mat');
save('-mat',fn,'rec');
fn = strcat(name,'EMIT.mat');
save('-mat',fn,'emit');
I expect the "length(rec) to be a cumulative value, but it's not - it just prints the number of "reception:" lines in each file, each time a new file is loaded its like the arrays "rec" and "emit" are being cleared, why does this happen? Is it to do with the file index? How can this be affecting the arrays?!
I tried this with a simple test program and it works exactly as I expect:
clear
test = [];
fin = 5;
for i = 1:3
j = 0;
while (j<fin)
j = j+1;
test(end+1) = j;
end
fin = fin+5;
length(test)
end
length(test) prints a cumulative value (as it should). What's going on?
NOTE:
I can easily solve the problem by adding at the beginning:
totalRec = [];
and after each iteration of the for loop:
totalRec = [totalRec,rec]
But I'm interested in why it's happening.
EDIT:
An example of the content of the data files is as follows:-
2017.07.13 10:31:04:346301 FESA.USR.ABT_SocketServer.RealTime.CustEvtSrcProducer src/ABT_SocketServer/RealTime/CustEvtSrcProducer.cpp:168 INFO Control_data_reception: 10936 us
2017.07.13 10:31:04:363889 FESA.USR.ABT_SocketServer.RealTime.RT_NewCycleReady src/ABT_SocketServer/RealTime/RT_NewCycleReady.cpp:52 INFO RT_NewCycleReady execute -> Start
2017.07.13 10:31:05:050910 FESA.USR.ABT_SocketServer.RealTime.RT_ELFTReady src/ABT_SocketServer/RealTime/RT_ELFTReady.cpp:49 INFO RT_ELFTReady execute -> Start
2017.07.13 10:31:05:057764 FESA.USR.ABT_SocketServer.RealTime.CustEvtSrcProducer src/ABT_SocketServer/RealTime/CustEvtSrcProducer.cpp:112 INFO acquisition_data_emission: 6469 us
2017.07.13 10:31:05:474323 FESA.USR.ABT_SocketServer.RealTime.RT_endCycle src/ABT_SocketServer/RealTime/RT_endCycle.cpp:49 INFO RT_endCycle execute -> Start
2017.07.13 10:31:05:504081 FESA.USR.ABT_SocketServer.RealTime.RT_NewCycle src/ABT_SocketServer/RealTime/RT_NewCycle.cpp:59 INFO RT_NewCycle execute -> Start
2017.07.13 10:31:05:544370 FESA.USR.ABT_SocketServer.RealTime.CustEvtSrcProducer src/ABT_SocketServer/RealTime/CustEvtSrcProducer.cpp:168 INFO Control_data_reception: 9424 us
i want to exclude the zeros that results from file's self comparisons and keep all the values other than that in the final array. what i am doing is using ~= for file comparison but not succeeded yet. Also tried strcmp. my code is as follows.
dirName= 'C:\Users\PCPC\U1'; %# folder path
files = dir( fullfile(dirName,'*.HWR') ); %# list all *.HWR files
files = {files.name}; %'# file names
supfinal = [];
final = [];
dist = [];
for i=1:length(files)
fname = fullfile(dirName,files{i});
c = dlmread(fname, '', 0,0);
c1=c(:,1);
for j=1:length(files)
fname = fullfile(dirName,files{j});
d = dlmread(fname, '', 0,0);
d1=d(:,1);
**if (j ~= i)**
dist = dtw(c1,d1); %# SOme Calculation
final = [final; dist];
end
end
**supfinal = [supfinal; final];**
end
i want from my code to give me results other than self comparison in the supfinal array. what's wrong i am doing. How to correct?
The code I'm using imports data from multiple files and saves them into an array of cells, the code is as follows:
[FileName,PathName,FilterIndex] = uigetfile('*.txt*','MultiSelect','on');
numfiles = size(FileName,2);
FileData= cell(1,numfiles);
for ii = 1:numfiles
FileName{ii};
A=[];
entirefile =fullfile(PathName,FileName{ii});
fid = fopen(entirefile);
tline = fgets(fid);
while ischar(tline)
parts = textscan(tline, '%f;');
if numel(parts{1}) > 0
A = [ A ; parts{:}' ];
end
tline = fgets(fid);
end
fclose(fid);
FileData{ii} = A;
A = FileData{ii};
X = A(:,1);
Y = A(:,5);
DataToUse = [X,Y];
end
Now my issue is I want to use the first DataToUse created by the loop, which will be data from the first file, seperatley to the other files but I can not issolate it. I have tried DataToUse(1), DataToUse(1,1) and DataToUse(:,[1,2]) but none are working for me. An example of the type of data would be:
DataToUse=
0.0762 0.0271
0.0763 0.2671
0.0764 0.4079
0.0765 0.0510
0.0766 0.0087
0.0767 0.0099
0.0768 0.0067
0.0769 0.0047
0.0770 0.0047
0.0771 0.0349
0.0772 0.2094
0.0773 0.2740
0.0774 0.0294
0.0775 0.0100
0.0776 0.0159
I have different numbers of this kind of data depending on how many files are selected but I would like to only use the first initially and use the others later. Anybody know how I can go about doing this? Many thanks in advance
The solution is to use cell arrays, like so:
DataToUse{ii} = [X, Y]
To get the desired output put this after your for-loop:
firstLoopXY = DataToUse{1}
Enjoy!
My customer is sending TDM/TDX files captured in National Instruments Diadem, which I haven't got. I'm looking for a way to convert the files into .CSV, XLS or .MAT files for analysis in Matlab (without using Diadem or Diadem DLLs!)
The format consists of a well structured XML file (.TDM) and a binary (.TDX), with the .TDM defining how fields are packed as bits in the binary TDX. I'd like to read the files (for use in Matlab and other environments). Does anyone have a general purpose tool or conversion script in for instance Python or Perl (not using the NI DLL's) or directly in Matlab?
I've looked into buying the tool, but didn't like it for anything other than one-time conversion to a compatible file format.
Thanks!
I know this is a little late, but I have a simple library to read TDM/TDX files in Python. It works by parsing the TDM file to figure out the data type, then using NumPy.memmap to open the TDX file. It can then be used like a standard NumPy array. The code is pretty simple, so you could probably implement something similar in Matlab.
Here's the link: https://bitbucket.org/joshayers/tdm_loader
Hope that helps.
Maybe a little too late, but I think there is a simple way to get the data from TDM files: NI provides plug-ins for reading TDM files into Excel and OpenOffice Calc. Having the data in one of these programs you could use the CSV export. Search google for "tdm excel" or "tdm openoffice".
Hope this helps...
Gemue
The following script can convert all variables into 'variable' struct.
CurrDirectory = '...//'; % Path to current directory
fileNametdx = '.../utility/'; % Path to TDX file
%%
% Data type conversion
Dtype.eInt8Usi='int8';
Dtype.eInt16Usi='int16';
Dtype.eInt32Usi='int32';
Dtype.eInt64Usi='int64';
Dtype.eUInt8Usi='uint8';
Dtype.eUInt16Usi='uint16';
Dtype.eUInt32Usi='uint32';
Dtype.eUInt64Usi='uint64';
Dtype.eFloat32Usi='single';
Dtype.eFloat64Usi='double';
%% Read .tdx file Name
wb=waitbar(0,'Reading *.tdx Files');
fileNameTDM = strrep(fileNametdx,'.tdx','.TDM');
%% Read .TDM
tdm=xml2struct(fileNameTDM);
for i=1:numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel)
waitbar((1/numel(tdm.usi_colon_tdm.usi_colon_data.tdm_channel))*i,wb,['File ' fileNametdx ' conversion started']);
s1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.local_columns.Text),'"');
usi1=s1(2);
% if condition match untill we get usi2
for j=1:numel(tdm.usi_colon_tdm.usi_colon_data.localcolumn)
usi2=string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.Attributes.id);
if usi1==usi2
%take new usi
s2=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.localcolumn{1, j}.values.Text),'"');
new_usi1=s2(2);
w1=strsplit(string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.datatype.Text),'_');
str_1=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence'));
str_2=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.Attributes.id'));
str_3=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.Attributes.external'));
str_4=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values'));
for k=1:numel(eval(str_1))
new_usi2=string(eval(str_2));
if new_usi1==new_usi2
if isfield(eval(str_4), 'Attributes')
inc_value1=string(eval(str_3));
for m=1:numel(tdm.usi_colon_tdm.usi_colon_include.file.block)
inc_value2=string(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.id);
if inc_value1==inc_value2
% offset=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset)/8);
length = round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.length));
offset1=round(str2num(tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.byteOffset));
value_type = tdm.usi_colon_tdm.usi_colon_include.file.block{1, m}.Attributes.valueType;
m = memmapfile(fullfile(CurrDirectory,fileNametdx),'Offset',offset1,'Format',{Dtype.(value_type) [length 1] 'dat'},'Writable',true,'Repeat',1);
dat=m.Data.dat ;
end
end
else
str_5=char(strcat('tdm.usi_colon_tdm.usi_colon_data.',lower(w1(2)),'_sequence{1, k}.values.',char(fieldnames(tdm.usi_colon_tdm.usi_colon_data.string_sequence{1, k}.values))));
dat=eval(str_5)';
end
name_variable = string(tdm.usi_colon_tdm.usi_colon_data.tdm_channel{1, i}.name.Text);
varname = genvarname(char(name_variable));
variable.(varname) = dat;
end
end
end
end
end
waitbar(1,wb,[fileNametdx ' conversion completed']);
pause(1)
close(wb)
delete(fullfile(CurrDirectory,fileNametdx),fullfile(CurrDirectory,fileNameTDM));
%Output Variable is Struct
clearvars -except variable
This script requires following XML parser
function [ s ] = xml2struct( file )
%Convert xml file into a MATLAB structure
% [ s ] = xml2struct( file )
%
% A file containing:
% <XMLname attrib1="Some value">
% <Element>Some text</Element>
% <DifferentElement attrib2="2">Some more text</Element>
% <DifferentElement attrib3="2" attrib4="1">Even more text</DifferentElement>
% </XMLname>
%
% Will produce:
% s.XMLname.Attributes.attrib1 = "Some value";
% s.XMLname.Element.Text = "Some text";
% s.XMLname.DifferentElement{1}.Attributes.attrib2 = "2";
% s.XMLname.DifferentElement{1}.Text = "Some more text";
% s.XMLname.DifferentElement{2}.Attributes.attrib3 = "2";
% s.XMLname.DifferentElement{2}.Attributes.attrib4 = "1";
% s.XMLname.DifferentElement{2}.Text = "Even more text";
%
% Please note that the following characters are substituted
% '-' by '_dash_', ':' by '_colon_' and '.' by '_dot_'
%
% Written by W. Falkena, ASTI, TUDelft, 21-08-2010
% Attribute parsing speed increased by 40% by A. Wanner, 14-6-2011
% Added CDATA support by I. Smirnov, 20-3-2012
%
% Modified by X. Mo, University of Wisconsin, 12-5-2012
if (nargin < 1)
clc;
help xml2struct
return
end
if isa(file, 'org.apache.xerces.dom.DeferredDocumentImpl') || isa(file, 'org.apache.xerces.dom.DeferredElementImpl')
% input is a java xml object
xDoc = file;
else
%check for existance
if (exist(file,'file') == 0)
%Perhaps the xml extension was omitted from the file name. Add the
%extension and try again.
if (isempty(strfind(file,'.xml')))
file = [file '.xml'];
end
if (exist(file,'file') == 0)
error(['The file ' file ' could not be found']);
end
end
%read the xml file
xDoc = xmlread(file);
end
%parse xDoc into a MATLAB structure
s = parseChildNodes(xDoc);
end
% ----- Subfunction parseChildNodes -----
function [children,ptext,textflag] = parseChildNodes(theNode)
% Recurse over node children.
children = struct;
ptext = struct; textflag = 'Text';
if hasChildNodes(theNode)
childNodes = getChildNodes(theNode);
numChildNodes = getLength(childNodes);
for count = 1:numChildNodes
theChild = item(childNodes,count-1);
[text,name,attr,childs,textflag] = getNodeData(theChild);
if (~strcmp(name,'#text') && ~strcmp(name,'#comment') && ~strcmp(name,'#cdata_dash_section'))
%XML allows the same elements to be defined multiple times,
%put each in a different cell
if (isfield(children,name))
if (~iscell(children.(name)))
%put existsing element into cell format
children.(name) = {children.(name)};
end
index = length(children.(name))+1;
%add new element
children.(name){index} = childs;
if(~isempty(fieldnames(text)))
children.(name){index} = text;
end
if(~isempty(attr))
children.(name){index}.('Attributes') = attr;
end
else
%add previously unknown (new) element to the structure
children.(name) = childs;
if(~isempty(text) && ~isempty(fieldnames(text)))
children.(name) = text;
end
if(~isempty(attr))
children.(name).('Attributes') = attr;
end
end
else
ptextflag = 'Text';
if (strcmp(name, '#cdata_dash_section'))
ptextflag = 'CDATA';
elseif (strcmp(name, '#comment'))
ptextflag = 'Comment';
end
%this is the text in an element (i.e., the parentNode)
if (~isempty(regexprep(text.(textflag),'[\s]*','')))
if (~isfield(ptext,ptextflag) || isempty(ptext.(ptextflag)))
ptext.(ptextflag) = text.(textflag);
else
%what to do when element data is as follows:
%<element>Text <!--Comment--> More text</element>
%put the text in different cells:
% if (~iscell(ptext)) ptext = {ptext}; end
% ptext{length(ptext)+1} = text;
%just append the text
ptext.(ptextflag) = [ptext.(ptextflag) text.(textflag)];
end
end
end
end
end
end
% ----- Subfunction getNodeData -----
function [text,name,attr,childs,textflag] = getNodeData(theNode)
% Create structure of node info.
%make sure name is allowed as structure name
name = toCharArray(getNodeName(theNode))';
name = strrep(name, '-', '_dash_');
name = strrep(name, ':', '_colon_');
name = strrep(name, '.', '_dot_');
attr = parseAttributes(theNode);
if (isempty(fieldnames(attr)))
attr = [];
end
%parse child nodes
[childs,text,textflag] = parseChildNodes(theNode);
if (isempty(fieldnames(childs)) && isempty(fieldnames(text)))
%get the data of any childless nodes
% faster than if any(strcmp(methods(theNode), 'getData'))
% no need to try-catch (?)
% faster than text = char(getData(theNode));
text.(textflag) = toCharArray(getTextContent(theNode))';
end
end
% ----- Subfunction parseAttributes -----
function attributes = parseAttributes(theNode)
% Create attributes structure.
attributes = struct;
if hasAttributes(theNode)
theAttributes = getAttributes(theNode);
numAttributes = getLength(theAttributes);
for count = 1:numAttributes
%attrib = item(theAttributes,count-1);
%attr_name = regexprep(char(getName(attrib)),'[-:.]','_');
%attributes.(attr_name) = char(getValue(attrib));
%Suggestion of Adrian Wanner
str = toCharArray(toString(item(theAttributes,count-1)))';
k = strfind(str,'=');
attr_name = str(1:(k(1)-1));
attr_name = strrep(attr_name, '-', '_dash_');
attr_name = strrep(attr_name, ':', '_colon_');
attr_name = strrep(attr_name, '.', '_dot_');
attributes.(attr_name) = str((k(1)+2):(end-1));
end
end
end