I have the following data called gg and yy.
> str(gg)
num [1:1992] 128 130 132 185 186 187 188 189 190 191 ...
> str(yy)
'data.frame': 2103 obs. of 2 variables:
$ grp : num 128 130 132 185 186 187 188 189 190 191 ...
$ predd: num -0.963 -1.518 1.712 -11.286 -8.195 ...
>
You'll notice that the first several values of gg match the first several from yy.
I would like to select rows from yy if the value yy$grp matches any value in gg. The issue is that gg and yy are of unequal length. Further, there are some values of gg that are not present in yy$grp and also some values of yy$grp not present in gg.
I can't seem to get this to work. It is basically an intersection of the two data sets based upon the index value I mentioned (gg, or yy$grp).
I've tried:
inters<-intersect(gg,yy$grp)
yyint<-yy[yy$grp==inters,]
but get the following
Warning message:
In yy$grp == inters :
longer object length is not a multiple of shorter object length
> str(yya)
'data.frame': 28 obs. of 2 variables:
$ grp : num 128 130 132 185 186 187 188 189 190 191 ...
$ predd: num -0.963 -1.518 1.712 -11.286 -8.195 ...
yya should be much longer, according to my plans at least.
Thanks.
As I mentioned, I think this is what you want:
yy[yy$grp %in% gg,]
Related
Some of the the values in columns Molecular.Weight and m.z are quite similar, often differing only by 1.0 or less. But there are some instances where its greater than 1.0. I would like to generate a new dataset that only includes the rows with a difference less than or equal to 1.0. However, it can be either column that has the higher number, so I am struggling to make an equation that works.
'data.frame': 544 obs. of 48 variables:
$ X : int 1 2 3 4 5 6 7 8 9 10 ...
$ No. : int 2 32 34 95 114 141 169 234 236 278 ...
$ RT..min. : num 0.89 3.921 0.878 2.396 0.845 ...
$ Molecular.Weight : num 70 72 72 78 80 ...
$ m.z : num 103 145 114 120 113 ...
$ HMDB.ID : chr "HMDB0006804" "HMDB0031647" "HMDB0006112" "HMDB0001505" ...
$ Name : chr "Propiolic acid" "Acrylic acid" "Malondialdehyde" "Benzene" ...
$ Formula : chr "C3H2O2" "C3H4O2" "C3H4O2" "C6H6" ...
$ Monoisotopic_Mass: num 70 72 72 78 80 ...
$ Delta.ppm. : num 1.295 0.833 1.953 1.023 0.102 ...
$ X1 : num 288.3 16.7 1130.9 3791.5 33.5 ...
$ X2 : num 276.8 13.4 1069.1 3228.4 44.1 ...
$ X3 : num 398.6 19.3 794.8 2153.2 15.8 ...
$ X4 : num 247.6 100.5 1187.5 1791.4 33.4 ...
$ X5 : num 98.4 162.1 1546.4 1646.8 45.3 ...
I had to do it in 2 parts because I couldn't figure out how to combine them but its still not giving me the right result.
The first section is supposed to filter out the values where Molecular.Weight might be greater than m.z by 1, and the second then filters out when m.z might be greater than Molecular.Weight. The first part seems to work and gives me a new dataset with around half the number of rows, but then when I do the second part on it, it gives me 1 row (and its not even correct because that one compound does fall within the 1.0 difference). Any help is super appreciated, thanks!
rawdata <- read.csv("Analysis negative + positive minus QC.csv")
filtered_data <-c()
for (i in 1:nrow(rawdata)) {
if (rawdata$m.z[i]-rawdata$Molecular.Weight[i]<1)
filtered_data <- rbind(filtered_data, rawdata[i,])
}
newdata <- c()
for (i in 1:row(filtered_data)) {
if ((filtered_data$Molecular.Weight[i] - filtered_data$m.z[i])>1)
newdata <- rbind(newdata, filtered_data[i,])
}
Consider the following code:
EmbedFeatures <- function(x,w) {
c_rev <- seq(from=w,to=1,by=-1)
em <- embed(x,w)
em <- em[,c_rev]
return(em)
}
m=matrix(1:1400,100,14)
X.tr<-c()
F<-dim(m)[2]
W=16
for(i in 1:F){ X.tr<-abind(list(X.tr,EmbedFeatures(m[,i],W)),along=3)}
this builds an array of features, each row has W=16 timesteps.
The dimensions are:
> dim(X.tr)
[1] 85 16 14
The following are the first samples:
> X.tr[1,,1]
[1] 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
> X.tr[1,,2]
[1] 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116
> X.tr[1,,3]
[1] 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216
I would like to use apply to build this array, but the following code does not work:
X.tr <- apply(m,2,EmbedFeatures, w=W)
since it gives me the following dimensions:
> dim(X.tr)
[1] 1360 14
How can I do it?
Firstly, thanks for providing a great reproducible example!
Now, as far as I know, you can't do this with apply. You can, however, do it with a combination of plyr::aaply, which allows you to return multidimensional arrays, and base::aperm, which allows you to transpose multidimensional arrays.
See here for aaply function details and here for aperm function details.
After running your code above, you can do:
library(plyr)
Y.tr <- plyr::aaply(m, 2, EmbedFeatures, w=W)
Z.tr <- aperm(Y.tr, c(2,3,1))
dim(Y.tr)
[1] 14 85 16
dim(Z.tr)
[1] 85 16 14
I turned those two lines of code into a function.
using_aaply <- function(m = m) {
Y.tr <- aaply(m, 2, EmbedFeatures, w=W)
Z.tr <- aperm(Y.tr, c(2,3,1))
return(Z.tr)
}
Then I did some microbenchmarking.
library(microbenchmark)
microbenchmark(for(i in 1:F){ X.tr<-abind(list(X.tr,EmbedFeatures(m[,i],W)),along=3)}, times=100)
Unit: milliseconds
expr
for (i in 1:F) { X.tr <- abind(list(X.tr, EmbedFeatures(m[, i], W)), along = 3) }
min lq mean median uq max neval
405.0095 574.9824 706.0845 684.8531 802.4413 1189.845 100
microbenchmark(using_aaply(m=m), times=100)
Unit: milliseconds
expr min lq mean median uq max
using_aaply(m = m) 4.873627 5.670474 7.797129 7.083925 9.674041 19.74449
neval
100
It seems like it's loads faster using aaply and aperm compared to abind in a for-loop.
Edit:
I've change my definition of allVals to the somewhat cleaner/simpler:
allVals=( $( printf '%s\n' \
$( printf '%s\n' \
{0..9}{0..9}{0,3,5,7} | \
sed -e 's#^0*##g' ) | \
awk '$1>='"$valMin"' && $1<='"$valMax" ) \
${exptVals[#]} )
I have a short BASH script used to produce space separated configuration files for a secondary executable. The scripted part is determining which values to print to column 1.
To accomplish this my script uses brace expansion to create an array of integers with the following rules:
Numbers are no more than 3 digits.
Numbers are integer (no decimal).
Products of 5 must be included in series.
I need at least one evenly spaced point between the [#][#]0 and [#][#]5 (i.e. a number ending in 3 or 7, as appropriate).
I use sed to clean up the case where the second most significant digit is blank (I'll probably replace the ' ' with '0' and write a simpler equivalent by remove leading '0's when I get around to it...).
Anyhow, these are values I input into a second program to produce computed predictions for certain properties. I also want to be sure to include numbers corresponding to certain experimental values I have... so I do that by creating an array of experimental values and then merging the two arrays together, sorting them and removing redundant values.
The script is given below (it was a oneliner -- I've edited it into script form for readability below):
#!/bin/bash
lineItem5=61
valMax=433
valMin=260
exptVals=( 257 261 265 269 273 277 281 285 289 293 297 \
301 305 309 313 317 321 325 329 333 337 341 \
345 349 353 357 361 365 369 373 377 381 385 \
389 393 397 401 405 409 413 417 421 425 429 \
433 )
allVals=( $( printf '%s\n' \
$( printf '%s\n' {' ',{1..9}}{' ',{1..9}}{0,3,5,7} | \
sed -e 's# \([1-9]\) 0 [1-9] 3 [1-9] 5 [1-9] 7 # \100 \103 \105 \107 #g' ) | \
awk '$1>='"$valMin"' && $1<='"$valMax" ) \
${exptVals[#]} )
sortVals=( $( printf '%s\n' ${allVals[#]} | sort -nr | uniq ) )
for ((t=0;t<${#sortVals[#]};t++)); do
printf '%s\n' "${sortVals[t]}"' -4000 -4000 200 '"${lineItem5}"' -1.0'
done
unset exptVals allVals sortVals
It works, but I would like to cut down on the number of lines (which equate to evaluated points and hence computation cost) and improve the spacing of values (which improves my statistical accuracy as each point of outputted properties depends on the previous calculations).
Specifically I'd like to remove the value ##7 if the sequence ##7 ##8 is encountered, and likewise ##3 if the sequence ##2 ##3... but only if the ##3 or ##7 value is not found in my list of experimental values. Also I want to change ##3 ##4 to ##2 ##4 and ##6 ##7 to ##6 ##8 to improve the spacing -- but only if the ##3 or ##7 are not in the experimental sequence.
So far the best way to do this I can think of is doing something like
valStart=0
for ((e=0; e<${#exptVals[#]}; e++)); do
for ((v=valStart; v<${#allT[#]}; v++)); do
if [[ ${allVals[v]} -ge ${exptVals[$((e+1))]} ]]; then
valStart=v
break
else
#Do edits to list here...
fi
done
done
The code isn't finished, but I think it would be moderately efficient as I don't have to loop through the second list entirely... just a small stretch of it (my experimental list is in order).
But I feel like there are easier ways to delete 'Y' from 'X Y' if 'Y' is not in array $vals or change 'Y' to 'Z' for 'X Y' if 'Y' is not in array $vals?
Is there a simple way to in a single expression using some sort of built in accomplish:
delete 'Y' from 'X Y' if 'Y' is not in array $vals
change 'Y' to 'Z' for 'X Y' if 'Y' is not in array $vals
...which does not involve looping through the values in bash-style loops (my brute-force method)?
The script you have makes calls to sed and awk to remove the spaces created by the brace expansion you used. A simpler brace expansion is:
$ echo {0..9}{0..9}{0,3,5,7}
The problem of the leading 0s is easy to solve with printf '%3.0f'.
A shorter list (as an example) will be created with this:
$ printf '%3.0f ' {0..1}{0..9}{0,3,5,7}
0 3 5 7 10 13 15 17 20 23 25 27 30 33 35 37 40 43
45 47 50 53 55 57 60 63 65 67 70 73 75 77 80 83 85 87
90 93 95 97 100 103 105 107 110 113 115 117 120 123 125 127 130 133
135 137 140 143 145 147 150 153 155 157 160 163 165 167 170 173 175 177
180 183 185 187 190 193 195 197
Once cleared this issue, we need to limit values between valMin and valMax.
Instead of calling an external awk to process a short list, a loop is better. With sorting (only called once) and printing, this script does about the same as yours with a lot less of external calls:
#!/bin/bash
lineItem5=61 valMin=260 valMax=433
exptVals=( 257 261 265 269 273 277 281 285 289 293 297 \
301 305 309 313 317 321 325 329 333 337 341 \
345 349 353 357 361 365 369 373 377 381 385 \
389 393 397 401 405 409 413 417 421 425 429 \
433 )
for v in $( printf '%3.0f\n' {0..9}{0..9}{0,3,5,7} )
do (( v>=valMin && v<=valMax )) && allVals+=( "$v" )
done
sortVals=( $(printf '%s\n' "${allVals[#]}" "${exptVals[#]}"|sort -nu) )
printf '%s ' "${sortVals[#]}"
Here we get to the core of your question. How to:
remove the value ##7 if the sequence ##7 ##8 is encountered
The usual wisdom to do this is to call sed. Something like:
printf '%s ' "${sortVals[#]}" | sed -e 's/\(..7 \)\(..8\)/\2/g'
That will convert ..7 ..8 to ..8 (the backreference \2).
Then, you may add more filters for more changes. Something similar to:
printf '%s ' "${sortVals[#]}" |
sed -e 's/\(..7 \)\(..8\)/\2/g' |
sed -e 's/\(..\)3\( ..4\)/\12\2/g'
echo
That will solve the ..7 ..8 to ..8 and the ..3 ..4 to ..2 ..4 items.
But your requirement of:
but only if the ##3 or ##7 value is not found in my list
Is more complex to meet. We need to scan all the values with grep, and execute different code for each option. One usual solution is to use grep:
if printf '%s ' "${sortVals[#]}" | grep -Eq '..3|..7'; then
cmd2=(cat)
else
cmd2=(sed -e 's/\(..2\)\( ..3\)/\1/g')
fi
But that means to scan all values with grep for each condition.
The command created: cmd2 is an array and may be used as this:
printf '%s ' "${sortVals[#]}" |
sed -e 's/\(..7 \)\(..8\)/\2/g' | "${cmd2[#]}" |
sed -e 's/\(..\)3\( ..4\)/\12\2/g' | "${cmd4[#]}"
echo
No grep
The values you are testing are only the last digit, which could be easily extracted with a modulo 10 math operation. And to make easier/faster
the testing of values, we can create an array of indexes like this:
unset indexVals; declare -A indexVals
for v in "${sortVals[#]}"; do indexVals[$((v%10))]=1; done
That's only one scan of values, no external tool called, and a big simplification of the testing of values (for example, for ..2 or ..3):
(( ${indexVals[2]-0} || ${indexVals[3]-0} ))
An script with all the changes is:
#!/bin/bash
lineItem5=61 valMin=260 valMax=433
exptVals=( 257 261 265 269 273 277 281 285 289 293 297 \
301 305 309 313 317 321 325 329 333 337 341 \
345 349 353 357 361 365 369 373 377 381 385 \
389 393 397 401 405 409 413 417 421 425 429 \
433 )
for v in $( printf '%3.0f\n' {0..9}{0..9}{0,3,5,7} )
do (( v>=valMin && v<=valMax )) && allVals+=( "$v" )
done
sortVals=( $(printf '%s\n' "${allVals[#]}" "${exptVals[#]}" | sort -nu) )
unset indexVals; declare -A indexVals
for v in "${sortVals[#]}"; do indexVals[$((v%10))]=1; done
cmd1=( sed -e 's/\(..7 \)\(..8\)/\2/g' )
(( ${indexVals[2]-0} || ${indexVals[3]-0} )) &&
cmd2=( cat ) ||
cmd2=( sed -e 's/\(..2\)\( ..3\)/\1/g' )
cmd3=( sed -e 's/\(..\)3\( ..4\)/\12\2/g' )
(( ${indexVals[3]-0} || ${indexVals[7]-0} )) &&
cmd4=( cat ) ||
cmd4=( sed -e 's/\(..6 ..\)7/\18/g' )
printf '%s ' "${sortVals[#]}" | "${cmd1[#]}" | "${cmd2[#]}" |
"${cmd3[#]}" | "${cmd4[#]}" ; echo
instead of generating the numbers by pattern, why not use awk to generate numbers as a numerical sequence?
for example,
$ awk -v from=100 -v to=200 -v ORS=' ' 'BEGIN{for(i=from;i<=to-10;i+=10)
print i,i+3,i+5,i+7; ORS="\n"; print""}'
100 103 105 107 110 113 115 117 120 123 125 127 130 133 135 137 140 143 145 147 150 153 155 157 160 163 165 167 170 173 175 177 180
183 185 187 190 193 195 197
I would like to create an array or vector of musical notes using a for loop. Every musical note, A, A#, B, C...etc is a 2^(1/12) ratio of the previous/next. E.G the note A is 440Hz, and A# is 440 * 2^(1/12) Hz = 446.16Hz.
Starting from 27.5Hz (A0), I want a loop that iterates 88 times to create an array of each notes frequency up to 4186Hz, so that will look like
f= [27.5 29.14 30.87 ... 4186.01]
So far, I've understood this much:
f = [];
for i=1:87,
%what goes here
% f = [27.5 * 2^(i/12)]; ?
end
return;
There is no need to do a loop for this in matlab, you can simply do:
f = 27.5 * 2.^((0:87)/12)
The answer:
f =
Columns 1 through 13
27.5 29.135 30.868 32.703 34.648 36.708 38.891 41.203 43.654 46.249 48.999 51.913 55
Columns 14 through 26
58.27 61.735 65.406 69.296 73.416 77.782 82.407 87.307 92.499 97.999 103.83 110 116.54
Columns 27 through 39
123.47 130.81 138.59 146.83 155.56 164.81 174.61 185 196 207.65 220 233.08 246.94
Columns 40 through 52
261.63 277.18 293.66 311.13 329.63 349.23 369.99 392 415.3 440 466.16 493.88 523.25
Columns 53 through 65
554.37 587.33 622.25 659.26 698.46 739.99 783.99 830.61 880 932.33 987.77 1046.5 1108.7
Columns 66 through 78
1174.7 1244.5 1318.5 1396.9 1480 1568 1661.2 1760 1864.7 1975.5 2093 2217.5 2349.3
Columns 79 through 88
2489 2637 2793.8 2960 3136 3322.4 3520 3729.3 3951.1 4186
maxind = 87;
f = zeros(1, maxind); % preallocate, better performance and avoids mlint warnings
for ii=1:maxind
f(ii) = 27.5 * 2^(ii/12);
end
The reason I named the loop variable ii is because i is the name of a builtin function. So it's considered bad practice to use that as a variable name.
Also, in your description you said you want to iterate 88 times, but the above loop only iterates 1 through 87 (both inclusive). If you want to iterate 88 times change maxind to 88.
So I implemented ciphersaber-1. It almost works, I can decrypt the cstest1.cs1. But i have trouble getting cstest2.cs1 to work.
The output is:
The Fourth Amendment to the Constitution of the Unite ▀Stat→s of America
"The right o☻ the people to be secure in their persons, houses, papers, and
effects, against unreasonab→e searches an╚A)┤Xx¹▼☻dcðþÈ_#0Uc.?n~J¿|,lómsó£k░7╠▄
íuVRÊ ╣├xð"↕(Gû┤.>!{³♫╚Tƒ}Àõ+»~C;ÔÙ²÷g.qÏø←1ß█yÎßsÈ÷g┐ÅJÔÞ┘Îö║AÝf╔ìêâß╗È;okn│CÚê
õ&æÄ[5&Þ½╔s╦Nå1En♂☻♫ôzÓ9»Á╝ÐÅ├ðzÝÎòeØ%W¶]¤▲´Oá╗e_Ú)╣ó0↑ï^☻P>ù♂¥¯▄‗♦£mUzMצվ~8å
ì½³░Ùã♠,H-tßJ!³*²RóÅ
So I must have a bug in initializing the state. The odd thing is that I can encrypt and decrypt long texts without problems, so the bug is symmetric.
I implemented the rc4 cipher as a reentrent single byte algorithm as you can see in rc4.c.
The state is stored in the rc4_state struct:
typedef unsigned char rc4_byte;
struct rc4_state_
{
rc4_byte i;
rc4_byte j;
rc4_byte state[256];
};
typedef struct rc4_state_ rc4_state;
The state is initialized with rc4_init:
void rc4_init(rc4_state* state, rc4_byte* key, size_t keylen)
{
rc4_byte i, j, n;
i = 0;
do
{
state->state[i] = i;
i++;
}
while (i != 255);
j = 0;
i = 0;
do
{
n = i % keylen;
j += state->state[i] + key[n];
swap(&state->state[i], &state->state[j]);
i++;
}
while (i != 255);
state->i = 0;
state->j = 0;
}
The actual encryption / decryption is done in rc4:
rc4_byte rc4(rc4_state* state, rc4_byte in)
{
rc4_byte n;
state->i++;
state->j += state->state[state->i];
swap(&state->state[state->i], &state->state[state->j]);
n = state->state[state->i] + state->state[state->j];
return in ^ state->state[n];
}
For completeness, swap:
void swap(rc4_byte* a, rc4_byte* b)
{
rc4_byte t = *a;
*a = *b;
*b = t;
}
I have been breaking my head on this for more than two days... The state, at least for the "asdfg" key is correct. Any help would be nice.
The whole thing can be found in my github reopsitory: https://github.com/rioki/ciphersaber/
I stumbled across your question while searching online, but since you haven't updated your code at GitHub yet, I figured you might still like to know what the problem was.
It's in this bit of code:
i = 0;
do
{
state->state[i] = i;
i++;
}
while (i != 255);
After this loop has iterated 255 times, i will have a value of 255 and the loop will terminate. As a result, the last byte of your state buffer is being left uninitialised.
This is easily fixed. Just change while (i != 255); to while (i);.
Sorry you haven't gotten feedback, I finally pulled this off in Python 3 today, but don't know enough about C to debug your code.
Some of the links on the main ciphersaber page are broken (pointing to ".com" instead of ".org"), so you might not have found the FAQ:
http://ciphersaber.gurus.org/faq.html
It includes the following debugging tips:
Make sure you are not reading or writing encrypted files as text files. You must use binary mode for file I/O.
If you are writing in the C language, be sure to store bytes as unsigned char.
Watch out for classic indexing problems. Do arrays in you chosen programming language start with 0 or 1?
Make sure you are writing out a random 10 byte IV when you encrypt and are reading the IV from the start of the file when you decrypt.
If your program still does not work, put in some statements to print out the S array after the key setup step. Then run your program to
decrypt the file cstest1.cs1 using asdfg as the key. Here is how the S
array should look:
file: cstest1.cs1
key: asdfg
176 32 49 160 15 112 58 8 186 19 50 161 60 17 82 153 37 141 131 127 59
2 165 103 98 53 9 57 41 150 174 64 36 62 191 154 44 136 149 158 226
113 230 227 247 155 221 34 125 20 163 95 128 219 1 181 201 146 88 204
213 80 143 164 145 234 134 248 100 77 188 235 76 217 194 35 75 99 126
92 243 177 52 180 83 140 198 42 151 18 91 33 16 192 101 48 97 220 114
110 124 72 139 218 142 118 81 84 31 29 195 68 209 172 200 214 93 240
61 22 206 123 152 7 203 10 119 171 79 250 109 137 199 167 11 104 211
129 208 216 178 207 242 162 30 120 65 115 87 170 47 69 244 212 45 85
73 222 225 185 63 0 179 210 108 245 202 46 96 148 51 173 24 182 89 116
3 67 205 94 231 23 21 13 169 215 190 241 228 132 252 4 233 56 105 26
12 135 223 166 238 229 246 138 239 54 5 130 159 236 66 175 189 147 193
237 43 40 117 157 86 249 74 27 156 14 133 251 196 187 197 102 106 39
232 255 121 122 253 111 90 38 55 70 184 78 224 25 6 107 168 254 144 28
183 71
I also found the "memorable test cases" helpful here:
http://www.cypherspace.org/adam/csvec/
Including:
key="Al"+ct="Al Dakota buys"(iv="Al Dakota "):
pt = "mead"
Even though the memorable test cases require cs2, upgrading to cs2 from cs1 is fairly trivial, you may be able to confidently convert your program to cs2 from cs1 even without fully debugging the rest of it.
Also note that the FAQ claims there used to be a file on the site that wouldn't decode, make sure your target file doesn't begin with "0e e3 f9 b2 40 11 fc 3e ..."
(Though I think that was a smaller test file, not the certificate.)
Oh, and also know that the site's not really up to date on the latest research into RC4 and derivatives. Just reserve this as a toy program unless all else fails.
Python
Here's one I wrote in Python for a question that later got deleted. It processes the file as a stream so memory usage is modest.
Usage
python encrypt.py <key> <rounds> < <infile> > <outfile>
python decrypt.py <key> <rounds> < <infile> > <outfile>
rc4.py
#!/usr/bin/env python
# coding: utf-8
import psyco
from sys import stdin,stdout,argv
def rc4(K):
R=range(256)
S=R[:]
T=bytearray(K*256)[:256]
j=0
for i in R*int(argv[2]):
j=j+S[i]+T[i]&255
S[i],S[j]=S[j],S[i]
i=j=0
while True:
B=stdin.read(4096)
if not B: break
for c in B:
i+=1&255
j=j+S[i]&255
S[i],S[j]=S[j],S[i]
stdout.write(chr(ord(c)^S[S[i]+S[j]&255]))
psyco.bind(rc4)
encrypt.py
from rc4 import *
import os
V=os.urandom(10)
stdout.write(V)
rc4(argv[1]+V)
decrypt.py
from rc4 import *
V=stdin.read(10)
rc4(argv[1]+V)