I know that I am pretty confused about arrays and strings and have tried a bunch of things but I am still stumped. I have groups of data that I am pulling into various arrays. For example I have site locations coming from one source. Numerous cores can be at a single location. The cores can have multiple depths. So I am pulling all this data together in various ways and pushing it out into a single excel file for each core. I create a filename based on location id and core name and year the core was sampled. So it might look like ‘ID_14_CORE_Bu-2-MT-1991.xlsx’ and I am storing it to use with a xlswrite statement in a variable called “filename.” This is all working fine.
But now I want to keep track of what files I have created and when I created them in another EXCEL file. So I was trying to store the location, filename and the date it was processed into some sort of array so that I can use the xlswrite statement to push it all out after I have processed all the locations/cores/layers that might occur in the original input files.
As I start the program and look in the original input files I can figure out how many cores I have so I wanted to create some sort of array to hold the location, filename and the date together. I have tried to use a cell array (a = cell(numcores,3)) but that does not seem to work. I think I am understanding that the filename is actually a string array so each of the letters is trying to be assigned to a separate cell instead of just the cell in the second column.
I also have had problems trying to push the three values out to the summary EXCEL file as each core is being processed but MATLAB tends to treat single dimensional arrays as a row rather than a column so I am kind of confused there.
Below is what I want an array to end up like…but since I am developing the filename on the fly this seems to be more challenging.
ArraytoExcel = [“14”, “ID_14_CORE_Bu-2-MT-1991.xlsx”,”1/1/2018”;
“14”, “ID_14_CORE_Bu-3-MT-1991.xlsx”,”1/1/2018”;
“13”, “ID_13_CORE_Tail_33-1992.xlsx”,”1/1/2018”;]
Maybe I am just going about this the wrong way. Any suggestions would help.
Your question is a little confusing but I think you want to do something like the following. The variables inside of my example are static but from your question it sounds like you already have these figured out somehow.
numcores = 5; %.. Or however, you determine what you are procesing
ArraytoExcel = cell(numcores ,3);
for ii = 1:numcores
%These 3 things will need to determined by you in the loop
% and not be static like in this example.
coreID = '14';
filename = 'ID_14_CORE_Bu-2-MT-1991.xlsx'; %
dataProc = datestr(now,'mm/dd/yyyy');
ArraytoExcel(ii,:) = {coreID,filename,dataProc};
end
xlswrite('YourOutput.xls',ArraytoExcel)
Related
When I append arrays in google spreadsheets, all of the resulting elements are not rendered in cells. For example, if I enter the formula:
={{1,2,3}, {4,5,6}}
the values rendered in spreadsheet cells are 1,4,5,6. Any ideas about why this is happening, or alternatives? My broader problem is to accumulate rows from separate sheets into another sheet - I can do that via
={ImportRange(...), ImportRange(...)}
but the same problem is apparent (missing the second element and beyond from the first array).
Edit (2 Oct 2014)
I just happened upon this when someone upvoted. The information below is obsolete in the newest version of Sheets - you can now (have been able to for a few months) concatenate arrays inside embedded arrays. All the examples that I provided below will work, including the one I said "shouldn't work".
Embedded arrays in Google Sheets
An array of values may be populated by a single function using an embedded array. Each element in the embedded array (and this may be point of conjecture; it is more or less just my opinion) represents the value that will be populated in contiguous cells in the sheet. Semi-colons are row delimiters; commas (or backslashes in locales that use a comma for a decimal separator) are column delimiters. So this will successfully create a two-row, three-column array (all of the following examples assume a locale supporting comma column delimiters):
={1,2,3;4,5,6}
Embedded arrays within embedded arrays
As each element in an embedded array represents a cell in the spreadsheet, I think it is reasonable to assume that one should be able to populate a cell with another embedded array, as long as it does not overwrite other elements in the outer embedded array. So IMO something like this should (see point 3) be successful:
={{1;2;3},{4;5;6}}
However something like this shouldn't work (again IMO), as the second and third elements of the first embedded array would be "overwriting" the second embedded array:
={{1,2,3},{4,5,6}}
There is a bug associated with the first embedded array inside an embedded array
As +Jason pointed out, something like ={{1;2;3},{4;5;6},{7;8;9}} doesn't work in that the first embedded array only populates one element (but every other column is populated correctly). It is also interesting that that one element is auto-converted to a text string. This is (unfortunately) a long standing bug in Google Sheets. The same thing occurs when you attempt to invoke the SPLIT() function on an array (every element in the array is split successfully except for the first one).
I don't think embedded arrays within embedded arrays will help with your broader problem anyway
Embedded arrays can't really be used to append one array on to end of another anyway (due to the "overwriting" effect), and there is no native function that can do it directly. The VMERGE function which you can obtain via the Script gallery (credit to +ahab) will work out of the box:
=VMERGE(ImportRange(...);ImportRange(...);...)
or you can use native functions to do some string manipulation to achieve this. For example, for one-dimensional arrays:
=ArrayFormula(TRANSPOSE(SPLIT(CONCATENATE(ImportRange("key1";"A1:A10")&CHAR(9);ImportRange("key2";"A1:A10")&CHAR(9));CHAR(9))))
but as well as being clunky and not very readable, this type of formula can be very expensive performance-wise for large data sets (I would tend to recommend the VMERGE custom function option in preference).
It is possible to make a union in Google Spreadsheet very easily. For example:
={'Sheet1'!A2:A;'Sheet2'!A2:A;'Sheet3'!A2:A}
See more info in Google Docs Help: Using arrays in Google Sheets
Assuming you have 3 arrays A2:B7, D4:E12, and F2:G230 with the same number of columns but different lengths (often the case if you have the same table of data split into different tabs for each period), I think the easiest way is something like this:
=TRANSPOSE({TRANSPOSE(A2:B7), TRANSPOSE(D4:E12), TRANSPOSE(F2:G230)})
I am trying to create a new reference containing another reference as in ${var${randnum}}.
Ultimately, I want to create a variable which refers to a two times two randomized set of variables.
As the above approach did not work, I developed it further with below result.
In the calculate field I write
concat('$','{','trust',${rand_no2},'_' ,${rand_no3_1},'}')
Which should result in
${trust1_1}
and respective combinations.
Without line 11 (name=ref2) the file compiles and I can start it in ODK Collect (v.2.4) on my phone. When I reach line 10 (in ODK Collect), however, I receive the message:
"Error Occured
Dependency cycle in s; recursion limit exceeded!!"
(I included line 11 to show what I want to do in the end.)
I am writing the file in Excel and compile it with ODK xlsform offline. (For testing I transfer it via cable to my phone.)
The xls file for reproduction can be found here:
https://forum.getodk.org/t/concatenate-references-to-create-new-reference-var-randnum/34968
Thank you very much in advance!
You're mixing up some things related to the ${q} syntax, question names and question values.
Note that ODK Collect does not actually understand the ${q} syntax (which is XLSForm-only). It's helpful to look at the actual form format that ODK collect understands which is called XForm, an XML format that XLSForm converts into. However, even if ODK Collect understood the ${q} syntax, your approach still wouldn't work since you're creating a string value for the ref question (using concat). This wouldn't magically be evaluated as a reference / formula. You cannot dynamically create a reference or formula.
At the moment (until ODK supports something like the local-name() function), maybe the best approach is to use position and put the calculated values inside a group. Something like //group/calc[number(${pos})] perhaps. Note that positions are 1-based (so the first item is position 1) and casting the position to a number or integer is required.
I am new to Python and am over complicating the coding on a project so I am starting with much smaller data sets in order to learn the process. My boss is having me compare two CSV files. The first CSV only contains the data 1,2,3,4,5,6 all in a single column. He wants me to set this CSV file as an array so I can compare the second CSV against it. The second CSV contains the data 3,5,6 all in a single column. The code should result in a print out of 1,2,4 as it is the only data not found in both CSV files.
I originally tried to write a code to import both CSV files and compare data without setting it as an array but this did not work so the first CSV file needs to be set as an array. The problem is I am not sure exactly how to do this with an array. This is what I have so far, any help anyone could give me would be greatly appreciated. I have been working on this project for a week now and am at a total loss, even with this simplified form.
import csv
temp_list = []
with open('1.csv','rb') as csvfile:
reader = csv.reader(csvfile, delimiter=',')
for row in reader:
In terms of psuedo-code, what you need to do here is import both csv files into two separate arrays, Array A and Array B for example.
Now what you need to do is compare each index position in one array, to each index position in the other array.
You need to create a nested loop, where the outer loop will choose an index position in A and then inner loop chooses a position in B.
After you check one index in A with each position in B, and no positions are the same, I suggest adding this value into a third array, C. You can check which positions are the same by using a boolean flag. When your code is done, C will have any values that don't exist in both A and B.
I suggest following these tutorials to learn more about python syntax:
https://www.w3schools.com/python/
Good luck
I am using Matlab for some data collection, and I want to save the data after each trial (just in case something goes wrong). The data is organized as a cell array of cell arrays, basically in the format
data{target}{trial} = zeros(1000,19)
But the actual data gets up to >150 MB by the end of the collection, so saving everything after each trial becomes prohibitively slow.
So now I am looking at opting for the matfile approach (http://www.mathworks.de/de/help/matlab/ref/matfile.html), which would allow me to only save parts of the data. The problem: this doesn't support cells of cell arrays, which means I couldn't change/update the data for a single trial; I would have to re-save the entire target's data (100 trials).
So, my question:
Is there another different method I can use to save parts of the cell array to speed up saving?
(OR)
Is there a better way to format my data that would work with this saving process?
A not very elegant but possibly effective solution is to use trial as part of the variable name. That is, use not a cell array of cell arrays (data{target}{trial}), but just different cell arrays such as data_1{target}, data_2{target}, where 1, 2 are the values of the trial counter.
You could do that with eval: for example
trial = 1; % change this value in a for lopp
eval([ 'data_' num2str(trial) '{target} = zeros(1000,19);']); % fill data_1{target}
You can then save the data for each trial in a different file. For example, this
eval([ 'save temp_save_file_' num2str(trial) ' data_' num2str(trial)])
saves data_1 in file temp_save_file_1, etc.
Update:
Actually it does appear to be possible to index into cell arrays, just not iside cell arrays. Hence, if you store your data slightly differently it seems like you can use matfile to update only part of it. See this example:
x = cell(3,4);
save x;
matObj = matfile('x.mat','writable',true);
matObj.x(3,4) = {eye(10)};
Note that this gives me a version warning, but it seems to work.
Hope this does the trick. However, still look into the next part of my answer as it may help you even more.
For calculations it is usually not required to save to disk after every iteration. An easy way to get a speedup (at the cost of a little more risk) is to save only after every n trials.
Like this for example:
maxTrial = 99;
saveEvery = 10;
for trial = 1:maxTrial
myFun; %Do your calculations here
if trial == maxTrial || mod(trial, saveEvery) == 0
save %Put your save command here
end
end
If your data is always at (or within) a certain size, you can also choose to store your data in a matrix rather than a cell array, then you can use indexing to save only part of the file.
In response to #Luis I will post an other way to deal with the situation.
It is indeed an option to save data in named variables or files, but to save a named variable in a named file seems too much.
If you only change the name of the file, you can save everything without using eval:
assuming you are dealing with trial 't':
filename = ['temp_save_file_' + num2str(t)];
If you really want, you can use print commands to write it as 001 for example.
Now you can simply use this:
save(filename, myData)
To use this, construct the filename again and so something like this:
totalData = {}; %Initialize your total data
And then read them as you wrote them (inside a loop):
load(filename)
totalData{t} = myData
When I append arrays in google spreadsheets, all of the resulting elements are not rendered in cells. For example, if I enter the formula:
={{1,2,3}, {4,5,6}}
the values rendered in spreadsheet cells are 1,4,5,6. Any ideas about why this is happening, or alternatives? My broader problem is to accumulate rows from separate sheets into another sheet - I can do that via
={ImportRange(...), ImportRange(...)}
but the same problem is apparent (missing the second element and beyond from the first array).
Edit (2 Oct 2014)
I just happened upon this when someone upvoted. The information below is obsolete in the newest version of Sheets - you can now (have been able to for a few months) concatenate arrays inside embedded arrays. All the examples that I provided below will work, including the one I said "shouldn't work".
Embedded arrays in Google Sheets
An array of values may be populated by a single function using an embedded array. Each element in the embedded array (and this may be point of conjecture; it is more or less just my opinion) represents the value that will be populated in contiguous cells in the sheet. Semi-colons are row delimiters; commas (or backslashes in locales that use a comma for a decimal separator) are column delimiters. So this will successfully create a two-row, three-column array (all of the following examples assume a locale supporting comma column delimiters):
={1,2,3;4,5,6}
Embedded arrays within embedded arrays
As each element in an embedded array represents a cell in the spreadsheet, I think it is reasonable to assume that one should be able to populate a cell with another embedded array, as long as it does not overwrite other elements in the outer embedded array. So IMO something like this should (see point 3) be successful:
={{1;2;3},{4;5;6}}
However something like this shouldn't work (again IMO), as the second and third elements of the first embedded array would be "overwriting" the second embedded array:
={{1,2,3},{4,5,6}}
There is a bug associated with the first embedded array inside an embedded array
As +Jason pointed out, something like ={{1;2;3},{4;5;6},{7;8;9}} doesn't work in that the first embedded array only populates one element (but every other column is populated correctly). It is also interesting that that one element is auto-converted to a text string. This is (unfortunately) a long standing bug in Google Sheets. The same thing occurs when you attempt to invoke the SPLIT() function on an array (every element in the array is split successfully except for the first one).
I don't think embedded arrays within embedded arrays will help with your broader problem anyway
Embedded arrays can't really be used to append one array on to end of another anyway (due to the "overwriting" effect), and there is no native function that can do it directly. The VMERGE function which you can obtain via the Script gallery (credit to +ahab) will work out of the box:
=VMERGE(ImportRange(...);ImportRange(...);...)
or you can use native functions to do some string manipulation to achieve this. For example, for one-dimensional arrays:
=ArrayFormula(TRANSPOSE(SPLIT(CONCATENATE(ImportRange("key1";"A1:A10")&CHAR(9);ImportRange("key2";"A1:A10")&CHAR(9));CHAR(9))))
but as well as being clunky and not very readable, this type of formula can be very expensive performance-wise for large data sets (I would tend to recommend the VMERGE custom function option in preference).
It is possible to make a union in Google Spreadsheet very easily. For example:
={'Sheet1'!A2:A;'Sheet2'!A2:A;'Sheet3'!A2:A}
See more info in Google Docs Help: Using arrays in Google Sheets
Assuming you have 3 arrays A2:B7, D4:E12, and F2:G230 with the same number of columns but different lengths (often the case if you have the same table of data split into different tabs for each period), I think the easiest way is something like this:
=TRANSPOSE({TRANSPOSE(A2:B7), TRANSPOSE(D4:E12), TRANSPOSE(F2:G230)})