When using large arrays it would be nice to be able to adjust the array for a certain number of bytes per number. Mostly I want fast routines to read such adjusted multi byte numbers to singles on the stack and conversely to store singles in the array adjusted for a certain number of bytes. In a 64 bit system there is a need for other single number arrays than one byte (c# c!) and eight bytes (# !).
So how to implement
cs# ( ad b -- n )
cs! ( n ad b -- )
where b is the number of bytes. The word cs! seems to work as
: cs! ( n ad b -- ) >r sp# cell+ swap r> cmove drop ;
but how about cs# and how to do it in pure ANS Forth without sp# or similar words?
The Forth200*x* committee has put quite some time into developing a Memory Access wordset that would suite. We have not included it into the standard thus far due to its size.
The compatible way is to use C# and bitwise operations. To use the same byte order in memory as Forth system there is need to detect endianness and compile the suitable versions of the certain definitions.
\ These definitions use little-endian format in memory.
\ Assumption: char size and address unit size equal to 1 octet.
: MB! ( x addr u -- )
ROT >R OVER + SWAP
BEGIN 2DUP U> WHILE R> DUP 8 RSHIFT >R OVER C! 1+ REPEAT
2DROP RDROP
;
: MB# ( addr u -- x )
0 >R OVER +
BEGIN 2DUP U< WHILE 1- DUP C# R> 8 LSHIFT OR >R REPEAT
2DROP R>
;
For higher performance it could be better to use implementation specific features (including W#, T#, Q#, SP#, etc) or even inline Forth-assembler.
Note that a straightforward definition via DO loop usually has worse performance (depends on optimizer; 10% in SP-Forth/4.21). The code for reference:
: MB! ( x addr u -- )
OVER + SWAP ?DO DUP I C! 8 RSHIFT LOOP DROP
;
: MB# ( addr u -- x )
DUP 0= IF NIP EXIT THEN
0 -ROT
1- OVER + DO 8 LSHIFT I C# OR -1 +LOOP
;
We can't use ?DO in the second case because of decreasing the loop index and +LOOP semantics: it leaves circle when the index crosses "the boundary between the loop limit minus one and the loop limit".
\ little-endian (eg. pc, android)
: mb! ( n ad i -- ) 2>r here ! here 2r> cmove ;
: mb# ( ad i -- n ) here 0 over ! swap cmove here # ;
\ big-endian (eg. mac)
: mb! ( n ad i -- ) 2>r here ! here cell + r# - 2r> cmove ;
: mb# ( ad i -- n ) here 0 over ! cell + over - swap cmove here # ;
\ little-endian test
1 here ! here c# negate .
Of course HERE could be any one cell buffer.
Thanks ruvim for parsing the process forward!
Related
In a disassembled dll (by IDA), I reached an array, which is commented as an array of int (but it may be of byte):
.rdata:000000018003CC00 ; int boxA[264]
.rdata:000000018003CC00 boxA dd 0 ; DATA XREF: BlockPrepXOR+5FC↑r
.rdata:000000018003CC04 db 0Eh
.rdata:000000018003CC05 db 0Bh
.rdata:000000018003CC06 db 0Dh
.rdata:000000018003CC07 db 9
.rdata:000000018003CC08 db 1Ch
.rdata:000000018003CC09 db 16h
.rdata:000000018003CC0A db 1Ah
.rdata:000000018003CC0B db 12h
.rdata:000000018003CC0C db 12h
.rdata:000000018003CC0D db 1Dh
.rdata:000000018003CC0E db 17h
.rdata:000000018003CC0F db 1Bh
Can I interpret the data as
{000000h, E0B0D09h, 1C161A12h, ..} or
{0, 90D0B0Eh, 121A161Ch, ...} or
{00h,00h,00h,00h, 0Eh, 0Bh, ..} ?
From the comment (from IDA), can you confirm that the array ends at CC00h + 253*4 = D01Fh ? I have another array starting at D020h:
.rdata:000000018003D01D db 0F9h ; ù
.rdata:000000018003D01E db 0A2h ; ¢
.rdata:000000018003D01F db 3Fh ; ?
.rdata:000000018003D020 array4_1248 db 1 ; DATA XREF: BlockPrepXOR+39A↑o
.rdata:000000018003D021 db 2
.rdata:000000018003D022 db 4
.rdata:000000018003D023 db 8
That's just the AES decryption's T8 matrix as described in this paper.
You can easily identify it by looking for the DWORDs values on Google (e.g. this is one of the results).
So that's just data for an AES decryption function.
Note also that the interpretation of a sequence of bytes as a sequence of multi-byte data (WORDs, DWORDs, QWORDs, and so on) depends on the architecture.
For x86, only the little-endian interpretation is correct (this is your case 2) but data may undergo arbitrary manipulations (e.g. it can be bswapped) so, when looking on Google, always use both the little and the big-endian versions of the data.
It's also worth noting that IDA can interpret the bytes as DWORDs (type d twice or use the context menù), showing the correct value based on the architecture of disassembled binary.
What I'm really stuck on is how I'm supposed to make this loop properly I tried doing every combination for each branch but it either never stops or it stops at the wrong number of loops (supposed to stop after amount of values in RPN_START). I even tried what I did in a previous lab which also failed to work so I'm completely lost as to how I'm supposed to get this program to loop. I understand that the RPN_OUT is the end pointer of the array so I tried putting RPN_START with CPY to compare the values but doing that made it stop after one loop. So I don't know at all which branch I'm supposed to use. Any help/tips/advice would be greatly appreciated. Thank you so much!
ORG DATA
;RPN_IN FCB $06,$03,$2F,$04,$2A,$02,$2B ; 63/4*2+=10
;RPN_IN FCB $05,$01,$02,$2B,$04,$2A,$2B,$03,$2D ; 512+4*+3-
;RPN_IN FCB $02,$03,$2A,$05,$2A,$02,$2F,$01, $2B ; ( ( (2 * 3) * 5) / 2) + 1
RPN_IN FCB $11,$10,$2F,$15,$2A ; ( (11 / 10) ) * 15
RPN_OUT RMB 1
RPN_START FDB RPN_IN ; Pointer to start of RPN array
RPN_END FDB RPN_OUT ; Pointer to end of RPN array
ANSWER RMB 1
TEMP FCB $00
ORG PROG
Entry: ; KEEP THIS LABEL!!
LDS #PROG
LDX #RPN_IN
LOOP:
TFR X,A
LDAA X
...
PSHA
RET:
INX
CPX #RPN_OUT
BNE LOOP
PULA
STAA ANSWER
I've been working on trying to port a small meanshift tracking program, which is sort of different from the meanshift that involves performing back-projection, from MATLAB to C, but I'm having some problems with it.
I don't know how to interpret some of the lines from MATLAB to C(OpenCV).
https://drive.google.com/drive/folders/0B2r9FmkcbNwAbHdtVEVkSW1SQW8
I put the 2 .m files, 1 .cpp file and a directory where pictures are placed for both programs to use on my googledrive.
"demo_MultiMeanShift_1st_ver_0413.m" is what I'd like to port to C,
"boxfilter.m" is the function that's found in the following website:
http://blog.csdn.net/wds555/article/details/23176313
(It's a Chinese website however)
and "Meanshift_demo.cpp" is what I've done so far.
There're mainly two parts that I don't know how to interpret from MATLAB to C:
1st part:
bas = zeros(hei,wid,N) ;
for i = 1 : 1 : N
bas(:,:,i) = boxfilter( ones(hei,wid), floor(r/i) ) ;
end
Ic_mean = zeros(hei,wid,dep,N) ;
for i = 1 : 1 : N
for d = 1 : 1 : dep
%Average pixel value(s) of the object being tracked
Ic_mean(pc(2)-(r+sw) : pc(2)+(r+sw), pc(1)-(r+sw) : pc(1)+(r+sw), d, i) = boxfilter(Ip(pc(2)-(r+sw) : pc(2)+(r+sw), pc(1)-(r+sw) : pc(1)+(r+sw),d), floor(r/i)) ./ bas(pc(2)-(r+sw) : pc(2)+(r+sw), pc(1)-(r+sw) : pc(1)+(r+sw),i);
%Ic_mean(:,:,d,i) = boxfilter(Ip(:,:,d), floor(r/i)) ./ bas(:,:,i);
end
end
dis = zeros(1,N) ;
2nd part:
for i = -sw + pc(2) : 2 : sw + pc(2)
for j = -sw + pc(1) : 2 : sw + pc(1)
for d = 1 : 1 : N
dis(d) = sqrt( ... % (p1(R1, G1, B1) - p2(R2, G2, B2))^2
( Ic_mean(i,j,1,d) - Ip_mean(pc(2),pc(1),1,d) )^2 ...
+ ( Ic_mean(i,j,2,d) - Ip_mean(pc(2),pc(1),2,d) )^2 ...
+ ( Ic_mean(i,j,3,d) - Ip_mean(pc(2),pc(1),3,d) )^2 );
end
if disMin > mean(dis)
disMin = mean(dis) ;
i_hold = i ;
j_hold = j ;
end
end
end
In MATLAB I can read, access and change pixel values directly, such as:
Img(x,y) = 0 to set some pixel value to be 0
or
Img(:,:,1) = 1 to set the pixels of certain channel to all be 0.
Can I also do those things as rapidly as what's shown above is in OpenCV?
In MATLAB I can read, access and change pixel values directly, such
as:
Img(x,y) = 0 to set some pixel value to be 0 or Img(:,:,1) = 1 to set
the pixels of certain channel to all be 0.
Can I also do those things as rapidly as what's shown above is in
OpenCV?
Of course this is possible. To set value of single pixel use:
img.at<img_single_element_type>(y,x) = 0;
where img_single_element_type depends on mat type and might be double, unsigned char, int, etc... See documentation for more information.
To set values in whole image (or part of image) use setTo method.
I don't know much about Matlab, so i can't help you with porting this code, but look at this and this project. It's similar project (object tracker - Open TLD(Tracking Learning Detection) aka Predator) written in Matlab(first link) and ported to C++(second link). Hope it helps.
I came across the following programming interview problem:
Challenge 1: N-grams
An N-gram is a sequence of N consecutive characters from a given word. For the word "pilot" there are three 3-grams: "pil", "ilo" and "lot".
For a given set of words and an n-gram length
Your task is to
• write a function that finds the n-gram that is the most frequent one among all the words
• print the result to the standard output (stdout)
• if there are multiple n-grams having the same maximum frequency please print the one that is the smallest lexicographically (the first one according to the dictionary sorting order)
Note that your function will receive the following arguments:
• text
○ which is a string containing words separated by whitespaces
• ngramLength
○ which is an integer value giving the length of the n-gram
Data constraints
• the length of the text string will not exceed 250,000 characters
• all words are alphanumeric (they contain only English letters a-z, A-Z and numbers 0-9)
Efficiency constraints
• your function is expected to print the result in less than 2 seconds
Example
Input
text: “aaaab a0a baaab c”
Output aaa
ngramLength: 3
Explanation
For the input presented above the 3-grams sorted by frequency are:
• "aaa" with a frequency of 3
• "aab" with a frequency of 2
• "a0a" with a frequency of 1
• "baa" with a frequency of 1
If I have only one hour to solve the problem and I chose to use the C language to solve it: is it a good idea to implement a Hash Table to count the frequency of the N-grams with that amount of time? because in the C library there is no implementation of a Hash Table...
If yes, I was thinking to implement a Hash Table using separate chaining with ordered linked lists. Those implementations reduce the time that you have to solve the problem....
Is that the fastest option possible?
Thank you!!!
If implementation efficiency is what matters and you are using C, I would initialize an array of pointers to the starts of n-grams in the string, use qsort to sort the pointers according to the n-gram that they are part of, and then loop over that sorted array and figure out counts.
This should execute fast enough, and there is no need to code any fancy data structures.
Sorry for posting python but this is what I would do:
You might get some ideas for the algorithm. Notice this program solves an order of magnitude more words.
from itertools import groupby
someText = "thibbbs is a test and aaa it may haaave some abbba reptetitions "
someText *= 40000
print len(someText)
n = 3
ngrams = []
for word in filter(lambda x: len(x) >= n, someText.split(" ")):
for i in range(len(word)-n+1):
ngrams.append(word[i:i+n])
# you could inline all logic here
# add to an ordered list for which the frequiency is the key for ordering and the paylod the actual word
ngrams_freq = list([[len(list(group)), key] for key, group in groupby(sorted(ngrams, key=str.lower))])
ngrams_freq_sorted = sorted(ngrams_freq, reverse=True)
popular_ngrams = []
for freq in ngrams_freq_sorted:
if freq[0] == ngrams_freq_sorted[0][0]:
popular_ngrams.append(freq[1])
else:
break
print "Most popular ngram: " + sorted(popular_ngrams, key=str.lower)[0]
# > 2560000
# > Most popular ngram: aaa
# > [Finished in 1.3s]**
So the basic recipe for this problem would be:
Find all n-grams in string
Map all duplicate entries into a new structure that has the n-gram and the number of times it occurs
You can find my c++ solution here: http://ideone.com/MNFSis
Given:
const unsigned int MAX_STR_LEN = 250000;
const unsigned short NGRAM = 3;
const unsigned int NGRAMS = MAX_STR_LEN-NGRAM;
//we will need a maximum of "the length of our string" - "the length of our n-gram"
//places to store our n-grams, and each ngram is specified by NGRAM+1 for '\0'
char ngrams[NGRAMS][NGRAM+1] = { 0 };
Then, for the first step - this is the code:
const char *ptr = str;
int idx = 0;
//notTerminated checks ptr[0] to ptr[NGRAM-1] are not '\0'
while (notTerminated(ptr)) {
//noSpace checks ptr[0] to ptr[NGRAM-1] are isalpha()
if (noSpace(ptr)) {
//safely copy our current n-gram over to the ngrams array
//we're iterating over ptr and because we're here we know ptr and the next NGRAM spaces
//are valid letters
for (int i=0; i<NGRAM; i++) {
ngrams[idx][i] = ptr[i];
}
ngrams[idx][NGRAM] = '\0'; //important to zero-terminate
idx++;
}
ptr++;
}
At this point, we have a list of all n-grams. Lets find the most popular one:
FreqNode head = { "HEAD", 0, 0, 0 }; //the start of our list
for (int i=0; i<NGRAMS; i++) {
if (ngrams[i][0] == '\0') break;
//insertFreqNode takes a start node, this where we will start to search for duplicates
//the simplest description is like this:
// 1 we search from head down each child, if we find a node that has text equal to
// ngrams[i] then we update it's frequency count
// 2 if the freq is >= to the current winner we place this as head.next
// 3 after program is complete, our most popular nodes will be the first nodes
// I have not implemented sorting of these - it's an exercise for the reader ;)
insertFreqNode(&head, ngrams[i]);
}
//as the list is ordered, head.next will always be the most popular n-gram
cout << "Winner is: " << head.next->str << " " << " with " << head.next->freq << " occurrences" << endl
Good luck to you!
Just for fun, I wrote a SQL version (SQL Server 2012):
if object_id('dbo.MaxNgram','IF') is not null
drop function dbo.MaxNgram;
go
create function dbo.MaxNgram(
#text varchar(max)
,#length int
) returns table with schemabinding as
return
with
Delimiter(c) as ( select ' '),
E1(N) as (
select 1 from (values
(1),(1),(1),(1),(1),(1),(1),(1),(1),(1)
)T(N)
),
E2(N) as (
select 1 from E1 a cross join E1 b
),
E6(N) as (
select 1 from E2 a cross join E2 b cross join E2 c
),
tally(N) as (
select top(isnull(datalength(#text),0))
ROW_NUMBER() over (order by (select NULL))
from E6
),
cteStart(N1) as (
select 1 union all
select t.N+1 from tally t cross join delimiter
where substring(#text,t.N,1) = delimiter.c
),
cteLen(N1,L1) as (
select s.N1,
isnull(nullif(charindex(delimiter.c,#text,s.N1),0) - s.N1,8000)
from cteStart s
cross join delimiter
),
cteWords as (
select ItemNumber = row_number() over (order by l.N1),
Item = substring(#text, l.N1, l.L1)
from cteLen l
),
mask(N) as (
select top(#length) row_Number() over (order by (select NULL))
from E6
),
topItem as (
select top 1
substring(Item,m.N,#length) as Ngram
,count(*) as Length
from cteWords w
cross join mask m
where m.N <= datalength(w.Item) + 1 - #length
and #length <= datalength(w.Item)
group by
substring(Item,m.N,#length)
order by 2 desc, 1
)
select d.s
from (
select top 1 NGram,Length
from topItem
) t
cross apply (values (cast(NGram as varchar)),(cast(Length as varchar))) d(s)
;
go
which when invoked with the sample input provided by OP
set nocount on;
select s as [ ] from MaxNgram(
'aaaab a0a baaab c aab'
,3
);
go
yields as desired
------------------------------
aaa
3
If you're not bound to C, I've written this Python script in about 10 minutes which processes 1.5Mb file, containing more than 265 000 words looking for 3-grams in 0.4s (apart from printing the values on the screen)
The text used for the test is Ulysses of James Joyce, you can find it free here https://www.gutenberg.org/ebooks/4300
Words separators here are both space and carriage return \n
import sys
text = open(sys.argv[1], 'r').read()
ngram_len = int(sys.argv[2])
text = text.replace('\n', ' ')
words = [word.lower() for word in text.split(' ')]
ngrams = {}
for word in words:
word_len = len(word)
if word_len < ngram_len:
continue
for i in range(0, (word_len - ngram_len) + 1):
ngram = word[i:i+ngram_len]
if ngram in ngrams:
ngrams[ngram] += 1
else:
ngrams[ngram] = 1
ngrams_by_freq = {}
for key, val in ngrams.items():
if val not in ngrams_by_freq:
ngrams_by_freq[val] = [key]
else:
ngrams_by_freq[val].append(key)
ngrams_by_freq = sorted(ngrams_by_freq.items())
for key in ngrams_by_freq:
print('{} with frequency of {}'.format(key[1:], key[0]))
You can convert trigram into RADIX50 code.
See http://en.wikipedia.org/wiki/DEC_Radix-50
In radix50, output value for trigram fits into 16-bit unsigned int value.
Thereafter, you can use radix-encoded trigram as index in the array.
So, your code would be like:
uint16_t counters[1 << 16]; // 64K counters
bzero(counters, sizeof(counters));
for(const char *p = txt; p[2] != 0; p++)
counters[radix50(p)]++;
Thereafter, just search for max value in the array, and decode index into trigram back.
I used this trick, when implemented Wilbur-Khovayko algorithm for fuzzy search ~10 years ago.
You can download source here: http://itman.narod.ru/source/jwilbur1.tar.gz.
You can solve this problem in O(nk) time where n is the number of words and k is the average number of n-grams per word.
You're correct in thinking that a hash table is a good solution to the problem.
However, since you have limited time to code a solution, I'd suggest using open addressing instead of a linked list. The implementation may be simpler: if you reach a collision you just walk farther along the list.
Also, be sure to allocate enough memory to your hash table: something about twice the size of the expected number of n-grams should be fine. Since the expected number of n-grams is <=250,000 a hash table of 500,000 should be more than sufficient.
In terms of coding speed, the small input length (250,000) makes sorting and counting a feasible option. The quickest way is probably to generate an array of pointers to each n-gram, sort the array using an appropriate comparator, and then walk along it keeping track of the which n-gram appeared the most.
One simple python solution for this question
your_str = "aaaab a0a baaab c"
str_list = your_str.split(" ")
str_hash = {}
ngram_len = 3
for str in str_list:
start = 0
end = ngram_len
len_word = len(str)
for i in range(0,len_word):
if end <= len_word :
if str_hash.get(str[start:end]):
str_hash[str[start:end]] = str_hash.get(str[start:end]) + 1
else:
str_hash[str[start:end]] = 1
start = start +1
end = end +1
else:
break
keys_sorted =sorted(str_hash.items())
for ngram in sorted(keys_sorted,key= lambda x : x[1],reverse = True):
print "\"%s\" with a frequency of %s" % (ngram[0],ngram[1])
I have a huge (4,5 GB) csv file.. I need to perform basic cut and paste, replace operations for some columns.. the data is pretty well organized.. the only problem is I cannot play with it with Excel because of the size (2000 rows, 550000 columns).
here is some part of the data:
ID,Affection,Sex,DRB1_1,DRB1_2,SENum,SEStatus,AntiCCP,RFUW,rs3094315,rs12562034,rs3934834,rs9442372,rs3737728
D0024949,0,F,0101,0401,SS,yes,?,?,A_A,A_A,G_G,G_G
D0024302,0,F,0101,7,SN,yes,?,?,A_A,G_G,A_G,?_?
D0023151,0,F,0101,11,SN,yes,?,?,A_A,G_G,G_G,G_G
I need to remove 4th, 5th, 6th, 7th, 8th and 9th columns;
I need to find every _ character from column 10 onwards and replace it with a space ( ) character;
I need to replace every ? with zero (0);
I need to replace every comma with a tab;
I need to remove first row (that has column names;
I need to replace every 0 with 1, every 1 with 2 and every ? with 0 in 2nd column;
I need to replace F with 2, M with 1 and ? with 0 in 3rd column;
so that in the resulting file the output reads:
D0024949 1 2 A A A A G G G G
D0024302 1 2 A A G G A G 0 0
D0023151 1 2 A A G G G G G G
(both input and output should read one line per row, ne extra blank row)
Is there a memory efficient way of doing that with java(and I need a code to do that) or a usable tool for playing with this large data so that I can easily apply Excel functionality..
You need two things:
- Knowledge of Regular Expressions (aka Regex, Regexes)
- PowerGrep