How do I set word delimiters? - arrays

Question

User's guide chapter 6.1.5 The Word Chunk A word is a string of characters delimited by space, tab, or return characters or enclosed by double quotes. Is it possible to have additional word delimiters?
I have the following code snippet taken from the User's Guide chapter 6.5.1 'When to use arrays', p. 184
on mouseUp
--cycle through each word adding each instance to an array
repeat for each word tWord in field "sample text"
add 1 to tWordCount[tWord]
end repeat
-- combine the array into text
combine tWordCount using return and comma
answer tWordCount
end mouseUp
It counts the number of occurences of each word form in the field "Sample text".
I realize that full stops after words are counted as part of the word with the default setting.
How do I change the settings that a full stop (and, or a comma) is considered a word boundary?

Answer 1

Alternatively you could simply remove the offending characters before processing.
This can be done using either the REPLACE function or the "REPLACETEXT function.
The REPLACETEXT function can use a regular expression matchstring but is slower than the REPLACE function. So here I am using the REPLACE function.
on mouseUp
put field "sample" into twords
--remove all trailing puncuation and quotes
replace "." with "" in twords
replace "," with "" in twords
replace "?" with "" in twords
replace ";" with "" in twords
replace ":" with "" in twords
replace quote with "" in twords
--hyphenated words need to be seperated?
replace "-" with " " in twords
repeat for each word tword in twords
add 1 to twordcount[tword]
end repeat
combine twordcount using return and comma
answer twordcount
end mouseUp

Answer 2

I think you are asking a question about delimiters. Some delimiters are built-in:
spaces for words,
commas for items,
return (CR) for lines.
The ability to create your own custom delimiter property (the itemDelimiter) is a powerful feature of the language, and pertains to "items". You can set this to any single character:
set the itemDelimiter to "C"
answer the number of items in "XXCXXCXX" --call this string "theText"
The result will be "3"
As others have pointed out, the method of replacing one string for another allows formidable control over custom parsing of text:
replace "C" with space in theText
yields "XX XX XX"
Craig Newman

Answer 3

As the User's guide says in chapter 6.1.5 The Word Chunk A word is a string of characters delimited by space, tab, or return characters or enclosed by double quotes.
There is itemDelimiter but not wordDelimiter.
So punctuation as to be removed first before adding the word to the word count array.
This may be done with a function effectiveWord.
function effectiveWord aWord
put last char of aWord into it
if it is "." then delete last char of aWord
if it is "," then delete last char of aWord
if it is ":" then delete last char of aWord
if it is ";" then delete last char of aWord
return aWord
end effectiveWord
on mouseUp
--cycle through each word adding each instance to an array
repeat for each word tWord in field "Sample text"
add 1 to tWordCount[effectiveWord(tWord)]
end repeat
-- combine the array into text
combine tWordCount using return and comma
answer tWordCount
end mouseUp