I am currently programming in C to find the complexity of functions in a program based on the number of lines in the functions. I will have to fopen an existing C file and proceed with the calculation. I know that there maybe some builtin tools for finding it. But still I want it to be programmed manually. Is there any specific method to find the start and end of the various functions in a C file?
Run this through C preprocessor. This way you strip comments, unroll macros, include #includes etc. Unless you want complexity of the user-readable code, this will produce results much more true.
Remove fixed strings. Anything between "" goes, note escaped quote \" doesn't close the string.
Scan the file. First { increases count of functions and begins scanning the body of a function. Observe depth. { increases depth, } decreases, as depth reaches 0 another } is the end of the function. Next { will be a new function, but as you scan the outside, if before reaching next { or EOF you encounter a ; - cancel any data collected on the last piece. That wasn't a function, it was a struct, an union or something like that.
I would recommend a 2-pass approach.
Pass 1: Remove any open or close braces inside comments (and optionally those in preprocessor directives).
Pass 2: Count open and close braces and whenever they match up (#open == #close) a function ends. The next open brace denotes the start of a new function.
This approach is not fail-safe. It may fail if the code contains preprocessor statements that violate good programming practice. If you encounter such code you may want to run your tool on the code after it has passed through the preprocessor stage.
I finally found a nice way to do this!
doxygen already does a lot of things to process functions and other things nicely.
generate doxygen conf like doxygen -g doxygen_conf
open the conf file with your favorite editor and set GENERATE_XML = YES. You might also wanna set RECURSIVE = YES and others needed for your project, and run doxygen. set also INPUT = [PATH_TO_PROJECT_BASE].
In your doxygen build directory, you will find html/ and xml/.
cd80#cd80 ~/lab/VulnVizOnLinux/linux-5.4.109 » cd build_doc
cd80#cd80 ~/lab/VulnVizOnLinux/linux-5.4.109/build_doc » ls
ExtractFunctions.ipynb html xml
cd80#cd80 ~/lab/VulnVizOnLinux/linux-5.4.109/build_doc »
(ignore ExtractFunctions.ipynb, that's mine)
cd to xml and open any of xml files and analyze it for a while.
Here's how I did it.
import os
import xml.etree.ElementTree as ET
base_path = '/home/cd80/lab/VulnVizOnLinux/linux-5.4.109/'
open_files = {}
doc = ET.parse('/home/cd80/lab/VulnVizOnLinux/linux-5.4.109/build_doc/xml/4_2kernel_2module-plts_8c.xml')
root = doc.getroot()
for func in root.findall(".//memberdef/[#kind='function']"):
name = func.find('./name').text
location = func.find('./location')
if 'bodyend' not in location.keys():
continue # this memberdef is not a definition of function
bodystart = int(location.attrib.get('bodystart'))
bodyend = int(location.attrib.get('bodyend'))
file_path = location.attrib.get('file')
file_path = os.path.join(base_path, file_path)
if file_path not in open_files.keys():
with open(file_path, 'rb') as f:
code = f.read().decode('utf-8')
open_files[file_path] = code
else:
code = open_files[file_path]
func_def = '\n'.join(code.split("\n")[bodystart-1:bodyend])
print(func_def)
print('='*30)
Result:
static struct plt_entry __get_adrp_add_pair(u64 dst, u64 pc,
enum aarch64_insn_register reg)
{
u32 adrp, add;
adrp = aarch64_insn_gen_adr(pc, dst, reg, AARCH64_INSN_ADR_TYPE_ADRP);
add = aarch64_insn_gen_add_sub_imm(reg, reg, dst % SZ_4K,
AARCH64_INSN_VARIANT_64BIT,
AARCH64_INSN_ADSB_ADD);
return (struct plt_entry){ cpu_to_le32(adrp), cpu_to_le32(add) };
}
==============================
struct plt_entry get_plt_entry(u64 dst, void *pc)
{
struct plt_entry plt;
static u32 br;
if (!br)
br = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_16,
AARCH64_INSN_BRANCH_NOLINK);
plt = __get_adrp_add_pair(dst, (u64)pc, AARCH64_INSN_REG_16);
plt.br = cpu_to_le32(br);
return plt;
}
==============================
bool plt_entries_equal(const struct plt_entry *a, const struct plt_entry *b)
{
u64 p, q;
/*
* Check whether both entries refer to the same target:
* do the cheapest checks first.
* If the 'add' or 'br' opcodes are different, then the target
* cannot be the same.
*/
if (a->add != b->add || a->br != b->br)
return false;
p = ALIGN_DOWN((u64)a, SZ_4K);
q = ALIGN_DOWN((u64)b, SZ_4K);
/*
* If the 'adrp' opcodes are the same then we just need to check
* that they refer to the same 4k region.
*/
if (a->adrp == b->adrp && p == q)
return true;
return (p + aarch64_insn_adrp_get_offset(le32_to_cpu(a->adrp))) ==
(q + aarch64_insn_adrp_get_offset(le32_to_cpu(b->adrp)));
}
==============================
static bool in_init(const struct module *mod, void *loc)
{
return (u64)loc - (u64)mod->init_layout.base < mod->init_layout.size;
}
==============================
u64 module_emit_plt_entry(struct module *mod, Elf64_Shdr *sechdrs,
void *loc, const Elf64_Rela *rela,
Elf64_Sym *sym)
{
struct mod_plt_sec *pltsec = !in_init(mod, loc) ? &mod->arch.core :
&mod->arch.init;
struct plt_entry *plt = (struct plt_entry *)sechdrs[pltsec->plt_shndx].sh_addr;
int i = pltsec->plt_num_entries;
int j = i - 1;
u64 val = sym->st_value + rela->r_addend;
if (is_forbidden_offset_for_adrp(&plt[i].adrp))
i++;
plt[i] = get_plt_entry(val, &plt[i]);
/*
* Check if the entry we just created is a duplicate. Given that the
* relocations are sorted, this will be the last entry we allocated.
* (if one exists).
*/
if (j >= 0 && plt_entries_equal(plt + i, plt + j))
return (u64)&plt[j];
pltsec->plt_num_entries += i - j;
if (WARN_ON(pltsec->plt_num_entries > pltsec->plt_max_entries))
return 0;
return (u64)&plt[i];
}
==============================
static int cmp_rela(const void *a, const void *b)
{
const Elf64_Rela *x = a, *y = b;
int i;
/* sort by type, symbol index and addend */
i = cmp_3way(ELF64_R_TYPE(x->r_info), ELF64_R_TYPE(y->r_info));
if (i == 0)
i = cmp_3way(ELF64_R_SYM(x->r_info), ELF64_R_SYM(y->r_info));
if (i == 0)
i = cmp_3way(x->r_addend, y->r_addend);
return i;
}
==============================
static bool duplicate_rel(const Elf64_Rela *rela, int num)
{
/*
* Entries are sorted by type, symbol index and addend. That means
* that, if a duplicate entry exists, it must be in the preceding
* slot.
*/
return num > 0 && cmp_rela(rela + num, rela + num - 1) == 0;
}
==============================
static unsigned int count_plts(Elf64_Sym *syms, Elf64_Rela *rela, int num,
Elf64_Word dstidx, Elf_Shdr *dstsec)
{
unsigned int ret = 0;
Elf64_Sym *s;
int i;
for (i = 0; i < num; i++) {
u64 min_align;
switch (ELF64_R_TYPE(rela[i].r_info)) {
case R_AARCH64_JUMP26:
case R_AARCH64_CALL26:
if (!IS_ENABLED(CONFIG_RANDOMIZE_BASE))
break;
/*
* We only have to consider branch targets that resolve
* to symbols that are defined in a different section.
* This is not simply a heuristic, it is a fundamental
* limitation, since there is no guaranteed way to emit
* PLT entries sufficiently close to the branch if the
* section size exceeds the range of a branch
* instruction. So ignore relocations against defined
* symbols if they live in the same section as the
* relocation target.
*/
s = syms + ELF64_R_SYM(rela[i].r_info);
if (s->st_shndx == dstidx)
break;
/*
* Jump relocations with non-zero addends against
* undefined symbols are supported by the ELF spec, but
* do not occur in practice (e.g., 'jump n bytes past
* the entry point of undefined function symbol f').
* So we need to support them, but there is no need to
* take them into consideration when trying to optimize
* this code. So let's only check for duplicates when
* the addend is zero: this allows us to record the PLT
* entry address in the symbol table itself, rather than
* having to search the list for duplicates each time we
* emit one.
*/
if (rela[i].r_addend != 0 || !duplicate_rel(rela, i))
ret++;
break;
case R_AARCH64_ADR_PREL_PG_HI21_NC:
case R_AARCH64_ADR_PREL_PG_HI21:
if (!IS_ENABLED(CONFIG_ARM64_ERRATUM_843419) ||
!cpus_have_const_cap(ARM64_WORKAROUND_843419))
break;
/*
* Determine the minimal safe alignment for this ADRP
* instruction: the section alignment at which it is
* guaranteed not to appear at a vulnerable offset.
*
* This comes down to finding the least significant zero
* bit in bits [11:3] of the section offset, and
* increasing the section's alignment so that the
* resulting address of this instruction is guaranteed
* to equal the offset in that particular bit (as well
* as all less signficant bits). This ensures that the
* address modulo 4 KB != 0xfff8 or 0xfffc (which would
* have all ones in bits [11:3])
*/
min_align = 2ULL << ffz(rela[i].r_offset | 0x7);
/*
* Allocate veneer space for each ADRP that may appear
* at a vulnerable offset nonetheless. At relocation
* time, some of these will remain unused since some
* ADRP instructions can be patched to ADR instructions
* instead.
*/
if (min_align > SZ_4K)
ret++;
else
dstsec->sh_addralign = max(dstsec->sh_addralign,
min_align);
break;
}
}
if (IS_ENABLED(CONFIG_ARM64_ERRATUM_843419) &&
cpus_have_const_cap(ARM64_WORKAROUND_843419))
/*
* Add some slack so we can skip PLT slots that may trigger
* the erratum due to the placement of the ADRP instruction.
*/
ret += DIV_ROUND_UP(ret, (SZ_4K / sizeof(struct plt_entry)));
return ret;
}
==============================
int module_frob_arch_sections(Elf_Ehdr *ehdr, Elf_Shdr *sechdrs,
char *secstrings, struct module *mod)
{
unsigned long core_plts = 0;
unsigned long init_plts = 0;
Elf64_Sym *syms = NULL;
Elf_Shdr *pltsec, *tramp = NULL;
int i;
/*
* Find the empty .plt section so we can expand it to store the PLT
* entries. Record the symtab address as well.
*/
for (i = 0; i < ehdr->e_shnum; i++) {
if (!strcmp(secstrings + sechdrs[i].sh_name, ".plt"))
mod->arch.core.plt_shndx = i;
else if (!strcmp(secstrings + sechdrs[i].sh_name, ".init.plt"))
mod->arch.init.plt_shndx = i;
else if (!strcmp(secstrings + sechdrs[i].sh_name,
".text.ftrace_trampoline"))
tramp = sechdrs + i;
else if (sechdrs[i].sh_type == SHT_SYMTAB)
syms = (Elf64_Sym *)sechdrs[i].sh_addr;
}
if (!mod->arch.core.plt_shndx || !mod->arch.init.plt_shndx) {
pr_err("%s: module PLT section(s) missing\n", mod->name);
return -ENOEXEC;
}
if (!syms) {
pr_err("%s: module symtab section missing\n", mod->name);
return -ENOEXEC;
}
for (i = 0; i < ehdr->e_shnum; i++) {
Elf64_Rela *rels = (void *)ehdr + sechdrs[i].sh_offset;
int numrels = sechdrs[i].sh_size / sizeof(Elf64_Rela);
Elf64_Shdr *dstsec = sechdrs + sechdrs[i].sh_info;
if (sechdrs[i].sh_type != SHT_RELA)
continue;
/* ignore relocations that operate on non-exec sections */
if (!(dstsec->sh_flags & SHF_EXECINSTR))
continue;
/* sort by type, symbol index and addend */
sort(rels, numrels, sizeof(Elf64_Rela), cmp_rela, NULL);
if (!str_has_prefix(secstrings + dstsec->sh_name, ".init"))
core_plts += count_plts(syms, rels, numrels,
sechdrs[i].sh_info, dstsec);
else
init_plts += count_plts(syms, rels, numrels,
sechdrs[i].sh_info, dstsec);
}
pltsec = sechdrs + mod->arch.core.plt_shndx;
pltsec->sh_type = SHT_NOBITS;
pltsec->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
pltsec->sh_addralign = L1_CACHE_BYTES;
pltsec->sh_size = (core_plts + 1) * sizeof(struct plt_entry);
mod->arch.core.plt_num_entries = 0;
mod->arch.core.plt_max_entries = core_plts;
pltsec = sechdrs + mod->arch.init.plt_shndx;
pltsec->sh_type = SHT_NOBITS;
pltsec->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
pltsec->sh_addralign = L1_CACHE_BYTES;
pltsec->sh_size = (init_plts + 1) * sizeof(struct plt_entry);
mod->arch.init.plt_num_entries = 0;
mod->arch.init.plt_max_entries = init_plts;
if (tramp) {
tramp->sh_type = SHT_NOBITS;
tramp->sh_flags = SHF_EXECINSTR | SHF_ALLOC;
tramp->sh_addralign = __alignof__(struct plt_entry);
tramp->sh_size = sizeof(struct plt_entry);
}
return 0;
}
==============================
Dirty but works just as how I wanted
Related
I'm trying to short the cpu id of my microcontroller (STM32F1).
The cpu id is composed by 3 word ( 3 x 4 bytes). This is the id string built from the 3 word: 980416578761680031125348904
I found a very useful library that do this.
The library is Hashids and there is a C code.
I try to build a test code on PC with "Code Blocks IDE" and the code works.
But when I move the code into the embedded side (Keil v5 IDE), I get an error on strdup() function: "strdup implicit declaration of function".
The problem is related to the strdup function isn't a standard library function and ins't included into string.h.
I will avoid to replace the strdup function with a custom function (that mimic the behaviour of strdup) to avoid memory leak because strdup copy strings using malloc.
Is there a different approach to compress long numbers?
Thanks for the help!
<---Appendix--->
This is the function that uses the strdup.
/* common init */
struct hashids_t *
hashids_init3(const char *salt, size_t min_hash_length, const char *alphabet)
{
struct hashids_t *result;
unsigned int i, j;
size_t len;
char ch, *p;
hashids_errno = HASHIDS_ERROR_OK;
/* allocate the structure */
result = _hashids_alloc(sizeof(struct hashids_t));
if (HASHIDS_UNLIKELY(!result)) {
hashids_errno = HASHIDS_ERROR_ALLOC;
return NULL;
}
/* allocate enough space for the alphabet and its copies */
len = strlen(alphabet) + 1;
result->alphabet = _hashids_alloc(len);
result->alphabet_copy_1 = _hashids_alloc(len);
result->alphabet_copy_2 = _hashids_alloc(len);
if (HASHIDS_UNLIKELY(!result->alphabet || !result->alphabet_copy_1
|| !result->alphabet_copy_2)) {
hashids_free(result);
hashids_errno = HASHIDS_ERROR_ALLOC;
return NULL;
}
/* extract only the unique characters */
result->alphabet[0] = '\0';
for (i = 0, j = 0; i < len; ++i) {
ch = alphabet[i];
if (!strchr(result->alphabet, ch)) {
result->alphabet[j++] = ch;
}
}
result->alphabet[j] = '\0';
/* store alphabet length */
result->alphabet_length = j;
/* check length and whitespace */
if (result->alphabet_length < HASHIDS_MIN_ALPHABET_LENGTH) {
hashids_free(result);
hashids_errno = HASHIDS_ERROR_ALPHABET_LENGTH;
return NULL;
}
if (strchr(result->alphabet, ' ')) {
hashids_free(result);
hashids_errno = HASHIDS_ERROR_ALPHABET_SPACE;
return NULL;
}
/* copy salt */
result->salt = strdup(salt ? salt : HASHIDS_DEFAULT_SALT);
result->salt_length = (unsigned int) strlen(result->salt);
/* allocate enough space for separators */
result->separators = _hashids_alloc((size_t)
(ceil((float)result->alphabet_length / HASHIDS_SEPARATOR_DIVISOR) + 1));
if (HASHIDS_UNLIKELY(!result->separators)) {
hashids_free(result);
hashids_errno = HASHIDS_ERROR_ALLOC;
return NULL;
}
/* non-alphabet characters cannot be separators */
for (i = 0, j = 0; i < strlen(HASHIDS_DEFAULT_SEPARATORS); ++i) {
ch = HASHIDS_DEFAULT_SEPARATORS[i];
if ((p = strchr(result->alphabet, ch))) {
result->separators[j++] = ch;
/* also remove separators from alphabet */
memmove(p, p + 1,
strlen(result->alphabet) - (p - result->alphabet));
}
}
/* store separators length */
result->separators_count = j;
/* subtract separators count from alphabet length */
result->alphabet_length -= result->separators_count;
/* shuffle the separators */
hashids_shuffle(result->separators, result->separators_count,
result->salt, result->salt_length);
/* check if we have any/enough separators */
if (!result->separators_count
|| (((float)result->alphabet_length / (float)result->separators_count)
> HASHIDS_SEPARATOR_DIVISOR)) {
unsigned int separators_count = (unsigned int)ceil(
(float)result->alphabet_length / HASHIDS_SEPARATOR_DIVISOR);
if (separators_count == 1) {
separators_count = 2;
}
if (separators_count > result->separators_count) {
/* we need more separators - get some from alphabet */
int diff = separators_count - result->separators_count;
strncat(result->separators, result->alphabet, diff);
memmove(result->alphabet, result->alphabet + diff,
result->alphabet_length - diff + 1);
result->separators_count += diff;
result->alphabet_length -= diff;
} else {
/* we have more than enough - truncate */
result->separators[separators_count] = '\0';
result->separators_count = separators_count;
}
}
/* shuffle alphabet */
hashids_shuffle(result->alphabet, result->alphabet_length,
result->salt, result->salt_length);
/* allocate guards */
result->guards_count = (unsigned int) ceil((float)result->alphabet_length
/ HASHIDS_GUARD_DIVISOR);
result->guards = _hashids_alloc(result->guards_count + 1);
if (HASHIDS_UNLIKELY(!result->guards)) {
hashids_free(result);
hashids_errno = HASHIDS_ERROR_ALLOC;
return NULL;
}
if (HASHIDS_UNLIKELY(result->alphabet_length < 3)) {
/* take some from separators */
strncpy(result->guards, result->separators, result->guards_count);
memmove(result->separators, result->separators + result->guards_count,
result->separators_count - result->guards_count + 1);
result->separators_count -= result->guards_count;
} else {
/* take them from alphabet */
strncpy(result->guards, result->alphabet, result->guards_count);
memmove(result->alphabet, result->alphabet + result->guards_count,
result->alphabet_length - result->guards_count + 1);
result->alphabet_length -= result->guards_count;
}
/* set min hash length */
result->min_hash_length = min_hash_length;
/* return result happily */
return result;
}
The true question seems to be
Is there a different approach to compress long numbers?
There are many. They differ in several respects, including which bits of the input contribute to the output, how many inputs map to the same output, and what manner of transformations of the input leave the output unchanged.
As a trivial examples, you can compress the input to a single bit by any of these approaches:
Choose the lowest-order bit of the input
Choose the highest-order bit of the input
The output is always 1
etc
Or you can compress to 7 bits by using using the number of 1 bits in the input as the output.
None of those particular options is likely to be of interest to you, of course.
Perhaps you would be more interested in producing 32-bit outputs for your 96-bit inputs. Do note that in that case on average there will be at least 264 possible inputs that map to each possible output. That depends only on the sizes of input and output, not on any details of the conversion.
For example, suppose that you have
uint32_t *cpuid = ...;
pointing to the hardware CPU ID. You can produce a 32-bit value from it that depends on all the bits of the input simply by doing this:
uint32_t cpuid32 = cpuid[0] ^ cpuid[1] ^ cpuid[2];
Whether that would suit your purpose depends on how you intend to use it.
You can easily implement strdup yourself like this:
char* strdup (const char* str)
{
size_t size = strlen(str);
char* result = malloc(size);
if(result != NULL)
{
memcpy(result, str, size+1);
}
return result;
}
That being said, using malloc or strdup on an embedded system is most likely just nonsense practice, see this. Nor would you use float numbers. Overall, that library seems to have been written by a desktop-minded person.
If you are implementing something like for example a chained hash table on an embedded system, you would use a statically allocated memory pool and not malloc. I'd probably go with a non-chained one for that reason (upon duplicates, pick next free spot in the buffer).
Unique device ID register (96 bits) is located under address 0x1FFFF7E8. It is factory programmed and is read-only. You can read it directly without using any other external library. For example:
unsigned int b = *(0x1FFFF7E8);
should give you the first 32 bits (31:0) of the unique device ID. If you want to retrieve a string as in case of the library mentioned, the following should work:
sprintf(id, "%08X%08X%08X", *(0x1FFFF7E8), *(0x1FFFF7E8 + 4), *(0x1FFFF7E8 + 8);
Some additional casting may be required, but generally that's what the library did. Please refer to STM32F1xx Reference Manual (RM0008), section 30.2 for more details. The exact memory location to read from is different in case of Cortex-M4 family of the MCUs.
I was wondering if anyone can think of a reason how a read from a pointer could cause a segmentation when the pointer is:
1. non NULL pointer.
The thing I'm trying to do is walk the shared library and access their symbol table. The segmentation happens when I try to access the ELF hash table to get the amount of symbols within the symbol table.
This is not noticed in x86 platform.
It happens only when executing on MIPS64 platform.
The code are based on from this link:
How to interpret the dynamic symbol table in an ELF executable?
static void
btrace_dl_symtab_walk(struct dl_phdr_info *info,
btrace_dl_lib_t *ctx) {
ElfW(Dyn*) dyn;
ElfW(Sym*) sym = NULL;
ElfW(Word*) hash;
ElfW(Word) sym_cnt = 0;
char* strtab = NULL;
char* sym_name = NULL;
unsigned int i;
int j;
/*
* Make indicator to show all of them acomplished before going forward
*/
for (j = 0; j < info->dlpi_phnum; j++) {
if (info->dlpi_phdr[j].p_type == PT_DYNAMIC) {
dyn = (ElfW(Dyn)*)(info->dlpi_addr + info->dlpi_phdr[j].p_vaddr);
while(dyn->d_tag != DT_NULL) {
if (dyn->d_tag == DT_HASH) {
hash = (ElfW(Word*))dyn->d_un.d_ptr;
if (!hash) {
return;
}
/*
* SEGFAULT happens here
*/
printf("Before Seg Fault\n");
sym_cnt = *(hash + 1); //<=============== This line causes seg fault
printf("Never reached here\n");
} else if(dyn->d_tag == DT_GNU_HASH) {
/*
* Since there is no simply way to find entry count
* in GNU hash table, we have no choice but to
* count by hand
*/
uint32_t *buckets;
uint32_t *hashval;
hash = (ElfW(Word*))dyn->d_un.d_ptr;
buckets = hash + 4 + (hash[2]*sizeof(size_t)/4);
for (i = sym_cnt = 0; i < hash[0]; i++) {
if (buckets[i] > sym_cnt) {
sym_cnt = buckets[i];
}
}
if (sym_cnt) {
sym_cnt -= hash[1];
hashval = buckets + hash[0] + sym_cnt;
do {
sym_cnt++;
} while (!(*hashval++ & 1));
}
sym_cnt += hash[1];
}else if (dyn->d_tag == DT_STRTAB) {
strtab = (char*)dyn->d_un.d_ptr;
} else if (dyn->d_tag == DT_SYMTAB) {
sym = (ElfW(Sym*))dyn->d_un.d_ptr;
break;
}
dyn++;
}
break;
}
}
// Other acitivities
}
Any guidance are welcome. Thank you
I was wondering if anyone can think of a reason how a read from a pointer could cause a segmentation when the pointer is: 1. non NULL pointer.
The fact that pointer is not NULL does not imply that you can read from it. It could be invalid for any number of reasons, e.g.
char *p = mmap(...);
munmap(p, ...);
char c = p[0]; // p points into unmapped memory, SIGSEGV likely.
This is not noticed in x86 platform. It happens only when executing on MIPS64 platform.
Are you using the same version of GLIBC on both platforms?
If I recall correctly, older versions of GLIBC did not relocate DT_HASH, but newer versions do. This may also be architecture-specific.
You'll want to print the value of hash, and compare it to the value of dyn. If hash is small, you'll need to relocate it by info->dlpi_addr.
I'm studying Tom Forsyth's Linear-Speed Vertex Cache Optimization and i don't understand how he calculates the ACMR. From what i have read i already know that ACMR = number of cache misses / number of triangles, but what i don't understand is what kind of cache is being used (i.e. FIFO or LRU?).
I have written a test program that calculates and prints the ACMR of a given 3d model using a FIFO cache, can you please tell me if this code is ok? or should i use an LRU cache instead?
/* the number of entries in a FIFO cache */
#define FIFO_CACHE_SIZE 32
struct fifo_cache {
long entries[FIFO_CACHE_SIZE];
};
/**
* init_cache - initializes a FIFO cache
* #cache: A pointer to the FIFO cache structure to be initialized.
*
* Before a FIFO cache can be used, it must be initialized by calling this
* function.
*/
static void init_cache(struct fifo_cache *cache)
{
int i = 0;
/* initialize cache entries to an invalid value */
for (i = 0;i < FIFO_CACHE_SIZE;i++)
cache->entries[i] = -1;
}
/**
* check_entry - checks if the same entry is already added to the cache
* #cache: A pointer to the FIFO cache structure to be searched.
* #entry: An entry to be searched for.
*
* Return: If the same entry was found, the return value is nonzero. Otherwise,
* the return value is zero.
*/
static int check_entry(const struct fifo_cache *cache, u16 entry)
{
int i = 0;
for (i = 0;i < FIFO_CACHE_SIZE;i++) {
if (cache->entries[i] == (long)entry)
return 1;
}
return 0;
}
/**
* add_entry - adds a new entry to the FIFO cache
* #cache: A pointer to the FIFO cache structure the entry will be added to.
* #entry: An entry to add.
*/
static void add_entry(struct fifo_cache *cache, u16 entry)
{
long aux = 0;
long aux2 = 0;
int i = 0;
aux = cache->entries[0];
cache->entries[0] = (long)entry;
for (i = 1;i < FIFO_CACHE_SIZE;i++) {
aux2 = cache->entries[i];
cache->entries[i] = aux;
aux = aux2;
}
}
/**
* calculate_acmr - calculates the average cache miss ratio (aka. ACMR)
* #indices: The list of vertex indices.
* #count: The number of vertex indices in the #indices list.
*/
float calculate_acmr(const u16 *indices, size_t count)
{
struct fifo_cache cache = {0};
long total = 0; /* the total number of cache misses */
long i = 0;
/* initialize the cache */
init_cache(&cache);
for (i = 0;i < count;i++) {
if (!check_entry(&cache, indices[i])) {
/* an entry doesn't exist in the cache, so add it */
add_entry(&cache, indices[i]);
total++;
}
}
return ((float)total / (count / 3));
}
I found the answer. Modern GPUs uses FIFO caches for simplicity and speed, so it makes sense to calculate the ACMR using FIFO cache. The code given above is correct, so i'll keep using that.
You are correct that is the way hardware does it. Additionally you may want to read this: http://www.realtimerendering.com/blog/acmr-and-atvr/
For whatever reason, MinGW's gcc can't seem to find the '#define' block for this variable. The full error is
.../mapped.c:396:28: error: '_zzip_strcasecmp' undeclared (first use in this function)
Contents of mapped.c :
/*
* NOTE: this is part of libzzipmmapped (i.e. it is not libzzip).
* ==================
*
* These routines are fully independent from the traditional zzip
* implementation. They assume a readonly mmapped sharedmem block
* representing a complete zip file. The functions show how to
* parse the structure, find files and return a decoded bytestream.
*
* These routines are a bit simple and really here for documenting
* the way to access a zip file. The complexity of zip access comes
* from staggered reading of bytes and reposition of a filepointer in
* a big archive with lots of files and long compressed datastreams.
* Plus varaints of drop-in stdio replacements, obfuscation routines,
* auto fileextensions, drop-in dirent replacements, and so on...
*
* Author:
* Guido Draheim <guidod#gmx.de>
*
* Copyright (c) 2003,2004,2006 Guido Draheim
* All rights reserved,
* use under the restrictions of the
* Lesser GNU General Public License
* or alternatively the restrictions
* of the Mozilla Public License 1.1
*/
#define _ZZIP_DISK_FILE_STRUCT 1
#ifdef __linux__
#define _GNU_SOURCE _glibc_developers_are_idiots_to_call_strndup_gnu_specific_
#endif
#include <zzip/mmapped.h>
#include <zzip/format.h>
#include <zzip/fetch.h>
#include <zzip/__mmap.h>
#include <zzip/__fnmatch.h>
#include <stdlib.h>
#include <sys/stat.h>
#if defined ZZIP_HAVE_UNISTD_H
#include <unistd.h>
#elif defined ZZIP_HAVE_IO_H
#include <io.h>
#endif
#if defined ZZIP_HAVE_STRING_H
#include <string.h>
#elif defined ZZIP_HAVE_STRINGS_H
#include <strings.h>
#endif
#if __STDC_VERSION__+0 > 199900L
#define ___
#define ____
#else
#define ___ {
#define ____ }
#endif
/** => zzip_disk_mmap
* This function does primary initialization of a disk-buffer struct.
*/
int
zzip_disk_init(ZZIP_DISK* disk, void* buffer, zzip_size_t buflen)
{
disk->buffer = (zzip_byte_t*) buffer;
disk->endbuf = (zzip_byte_t*) buffer + buflen;
disk->reserved = 0;
disk->flags = 0;
disk->mapped = 0;
/* do not touch disk->user */
/* do not touch disk->code */
return 0;
}
/** => zzip_disk_mmap
* This function allocates a new disk-buffer with => malloc(3)
*/
zzip__new__ ZZIP_DISK*
zzip_disk_new(void)
{
ZZIP_DISK* disk = malloc(sizeof(disk));
if (! disk) return disk;
zzip_disk_init (disk, 0, 0);
return disk;
}
/** turn a filehandle into a mmapped zip disk archive handle
*
* This function uses the given file-descriptor to detect the length of the
* file and calls the system => mmap(2) to put it in main memory. If it is
* successful then a newly allocated ZZIP_DISK* is returned with
* disk->buffer pointing to the mapview of the zipdisk content.
*/
zzip__new__ ZZIP_DISK*
zzip_disk_mmap(int fd)
{
struct stat st;
if (fstat (fd, &st) || ! st.st_size) return 0;
___ ZZIP_DISK* disk = zzip_disk_new (); if (! disk) return 0;
disk->buffer = _zzip_mmap (& disk->mapped, fd, 0, st.st_size);
if (disk->buffer == MAP_FAILED) { free (disk); return 0; }
disk->endbuf = disk->buffer + st.st_size;
return disk; ____;
}
/** => zzip_disk_mmap
* This function is the inverse of => zzip_disk_mmap and using the system
* munmap(2) on the buffer area and => free(3) on the ZZIP_DISK structure.
*/
int
zzip_disk_munmap(ZZIP_DISK* disk)
{
if (! disk) return 0;
_zzip_munmap (disk->mapped, disk->buffer, disk->endbuf-disk->buffer);
free (disk);
return 0;
}
/** => zzip_disk_mmap
*
* This function opens the given archive by name and turn the filehandle
* to => zzip_disk_mmap for bringing it to main memory. If it can not
* be => mmap(2)'ed then we slurp the whole file into a newly => malloc(2)'ed
* memory block. Only if that fails too then we return null. Since handling
* of disk->buffer is ambigous it should not be snatched away please.
*/
ZZIP_DISK* zzip__new__
zzip_disk_open(char* filename)
{
# ifndef O_BINARY
# define O_BINARY 0
# endif
struct stat st;
if (stat (filename, &st) || ! st.st_size) return 0;
___ int fd = open (filename, O_RDONLY|O_BINARY);
if (fd <= 0) return 0;
___ ZZIP_DISK* disk = zzip_disk_mmap (fd);
if (disk) return disk;
___ zzip_byte_t* buffer = malloc (st.st_size);
if (! buffer) return 0;
if ((st.st_size == read (fd, buffer, st.st_size)) &&
(disk = zzip_disk_new ()))
{
disk->buffer = buffer;
disk->endbuf = buffer+st.st_size;
disk->mapped = -1;
}else free (buffer);
return disk; ____;____;____;
}
/** => zzip_disk_mmap
*
* This function will release all data needed to access a (mmapped)
* zip archive, including any malloc()ed blocks, sharedmem mappings
* and it dumps the handle struct as well.
*/
int
zzip_disk_close(ZZIP_DISK* disk)
{
if (! disk) return 0;
if (disk->mapped != -1) return zzip_disk_munmap (disk);
free (disk->buffer);
free (disk);
return 0;
}
/* ====================================================================== */
/* helper functions */
#ifdef ZZIP_HAVE_STRNDUP
#define _zzip_strndup strndup
#else
/* if your system does not have strndup: */
zzip__new__ static char* _zzip_strndup(char* p, size_t maxlen)
{
if (! p) return 0;
___ zzip_byte_t* r = malloc (maxlen+1);
if (! r) return r;
strncpy (r, p, maxlen);
r[maxlen] = '\0';
return r; ____;
}
#endif
#if defined ZZIP_HAVE_STRCASECMP || defined strcasecmp
#define _zzip_strcasecmp strcasecmp
#else
/* if your system does not have strcasecmp: */
static int _zzip_strcasecmp(char* __zzip_restrict a, char* _zzip_restrict b)
{
if (! a) return (b) ? 1 : 0;
if (! b) return -1;
while (1)
{
int v = tolower(*a) - tolower(*b);
if (v) return v;
if (! *a) return 1;
if (! *b) return -1;
a++; b++;
}
}
#endif
/** helper functions for (mmapped) zip access api
*
* This function augments the other zzip_disk_entry_* helpers: here we move
* a disk_entry pointer (as returned by _find* functions) into a pointer to
* the data block right after the file_header. Only disk->buffer would be
* needed to perform the seek but we check the mmapped range end as well.
*/
zzip_byte_t*
zzip_disk_entry_to_data(ZZIP_DISK* disk, struct zzip_disk_entry* entry)
{
struct zzip_file_header* file =
zzip_disk_entry_to_file_header(disk, entry);
if (file) return zzip_file_header_to_data (file);
return 0;
}
/** => zzip_disk_entry_to_data
* This function does half the job of => zzip_disk_entry_to_data where it
* can augment with => zzip_file_header_to_data helper from format/fetch.h
*/
struct zzip_file_header*
zzip_disk_entry_to_file_header(ZZIP_DISK* disk, struct zzip_disk_entry* entry)
{
zzip_byte_t* file_header = /* (struct zzip_file_header*) */
(disk->buffer + zzip_disk_entry_fileoffset (entry));
if (disk->buffer > file_header || file_header >= disk->endbuf)
return 0;
return (struct zzip_file_header*) file_header;
}
/** => zzip_disk_entry_to_data
* This function is a big helper despite its little name: in a zip file the
* encoded filenames are usually NOT zero-terminated but for common usage
* with libc we need it that way. Secondly, the filename SHOULD be present
* in the zip central directory but if not then we fallback to the filename
* given in the file_header of each compressed data portion.
*/
zzip__new__ char*
zzip_disk_entry_strdup_name(ZZIP_DISK* disk, struct zzip_disk_entry* entry)
{
if (! disk || ! entry) return 0;
___ char* name; zzip_size_t len;
struct zzip_file_header* file;
if ((len = zzip_disk_entry_namlen (entry)))
name = zzip_disk_entry_to_filename (entry);
else if ((file = zzip_disk_entry_to_file_header (disk, entry)) &&
(len = zzip_file_header_namlen (file)))
name = zzip_file_header_to_filename (file);
else
return 0;
if ((zzip_byte_t*) name < disk->buffer ||
(zzip_byte_t*) name+len > disk->endbuf)
return 0;
return _zzip_strndup (name, len); ____;
}
/** => zzip_disk_entry_to_data
* This function is similar creating a reference to a zero terminated
* string but it can only exist in the zip central directory entry.
*/
zzip__new__ char*
zzip_disk_entry_strdup_comment(ZZIP_DISK* disk, struct zzip_disk_entry* entry)
{
if (! disk || ! entry) return 0;
___ char* text; zzip_size_t len;
if ((len = zzip_disk_entry_comment (entry)))
text = zzip_disk_entry_to_comment (entry);
else
return 0;
if ((zzip_byte_t*) text < disk->buffer ||
(zzip_byte_t*) text+len > disk->endbuf)
return 0;
return _zzip_strndup (text, len); ____;
}
/* ====================================================================== */
/** => zzip_disk_findfile
*
* This function is the first call of all the zip access functions here.
* It contains the code to find the first entry of the zip central directory.
* Here we require the mmapped block to represent a real zip file where the
* disk_trailer is _last_ in the file area, so that its position would be at
* a fixed offset from the end of the file area if not for the comment field
* allowed to be of variable length (which needs us to do a little search
* for the disk_tailer). However, in this simple implementation we disregard
* any disk_trailer info telling about multidisk archives, so we just return
* a pointer to the zip central directory.
*
* For an actual means, we are going to search backwards from the end
* of the mmaped block looking for the PK-magic signature of a
* disk_trailer. If we see one then we check the rootseek value to
* find the first disk_entry of the root central directory. If we find
* the correct PK-magic signature of a disk_entry over there then we
* assume we are done and we are going to return a pointer to that label.
*
* The return value is a pointer to the first zzip_disk_entry being checked
* to be within the bounds of the file area specified by the arguments. If
* no disk_trailer was found then null is returned, and likewise we only
* accept a disk_trailer with a seekvalue that points to a disk_entry and
* both parts have valid PK-magic parts. Beyond some sanity check we try to
* catch a common brokeness with zip archives that still allows us to find
* the start of the zip central directory.
*/
struct zzip_disk_entry*
zzip_disk_findfirst(ZZIP_DISK* disk)
{
if (disk->buffer > disk->endbuf-sizeof(struct zzip_disk_trailer))
return 0;
___ zzip_byte_t* p = disk->endbuf-sizeof(struct zzip_disk_trailer);
for (; p >= disk->buffer ; p--)
{
zzip_byte_t* root; /* (struct zzip_disk_entry*) */
if (zzip_disk_trailer_check_magic(p)) {
root = disk->buffer + zzip_disk_trailer_get_rootseek (
(struct zzip_disk_trailer*)p);
if (root > p)
{ /* the first disk_entry is after the disk_trailer? can't be! */
zzip_size_t rootsize = zzip_disk_trailer_get_rootsize (
(struct zzip_disk_trailer*)p);
if (disk->buffer+rootsize > p) continue;
/* a common brokeness that can be fixed: we just assume the
* central directory was written directly before the trailer:*/
root = p - rootsize;
}
} else if (zzip_disk64_trailer_check_magic(p)) {
if (sizeof(void*) < 8) return 0; /* EOVERFLOW */
root = disk->buffer + zzip_disk64_trailer_get_rootseek (
(struct zzip_disk64_trailer*)p);
if (root > p) continue;
} else continue;
if (root < disk->buffer) continue;
if (zzip_disk_entry_check_magic(root))
return (struct zzip_disk_entry*) root;
}____;
return 0;
}
/** => zzip_disk_findfile
*
* This function takes an existing disk_entry in the central root directory
* (e.g. from zzip_disk_findfirst) and returns the next entry within in
* the given bounds of the mmapped file area.
*/
struct zzip_disk_entry*
zzip_disk_findnext(ZZIP_DISK* disk, struct zzip_disk_entry* entry)
{
if ((zzip_byte_t*)entry < disk->buffer ||
(zzip_byte_t*)entry > disk->endbuf-sizeof(entry) ||
! zzip_disk_entry_check_magic (entry) ||
zzip_disk_entry_sizeto_end (entry) > 64*1024)
return 0;
entry = zzip_disk_entry_to_next_entry (entry);
if ((zzip_byte_t*)entry > disk->endbuf-sizeof(entry) ||
! zzip_disk_entry_check_magic (entry) ||
zzip_disk_entry_sizeto_end (entry) > 64*1024 ||
zzip_disk_entry_skipto_end (entry) + sizeof(entry) > disk->endbuf)
return 0;
else
return entry;
}
/** search for files in the (mmapped) zip central directory
*
* This function is given a filename as an additional argument, to find the
* disk_entry matching a given filename. The compare-function is usually
* strcmp or strcasecmp or perhaps strcoll, if null then strcmp is used.
* - use null as argument for "after"-entry when searching the first
* matching entry, otherwise the last returned value if you look for other
* entries with a special "compare" function (if null then a doubled search
* is rather useless with this variant of _findfile).
*/
struct zzip_disk_entry*
zzip_disk_findfile(ZZIP_DISK* disk, char* filename,
struct zzip_disk_entry* after, zzip_strcmp_fn_t compare)
{
struct zzip_disk_entry* entry = (! after ? zzip_disk_findfirst (disk)
: zzip_disk_findnext (disk, after));
if (! compare)
compare = (zzip_strcmp_fn_t)( (disk->flags&1) ?
(_zzip_strcasecmp) : (strcmp));
for (; entry ; entry = zzip_disk_findnext (disk, entry))
{
/* filenames within zip files are often not null-terminated! */
char* realname = zzip_disk_entry_strdup_name (disk, entry);
if (realname && ! compare(filename, realname))
{
free (realname);
return entry;
}
free (realname);
}
return 0;
}
/** => zzip_disk_findfile
*
* This function uses a compare-function with an additional argument
* and it is called just like fnmatch(3) from POSIX.2 AD:1993), i.e.
* the argument filespec first and the ziplocal filename second with
* the integer-flags put in as third to the indirect call. If the
* platform has fnmatch available then null-compare will use that one
* and otherwise we fall back to mere strcmp, so if you need fnmatch
* searching then please provide an implementation somewhere else.
* - use null as argument for "after"-entry when searching the first
* matching entry, or the last disk_entry return-value to find the
* next entry matching the given filespec.
*/
struct zzip_disk_entry*
zzip_disk_findmatch(ZZIP_DISK* disk, char* filespec,
struct zzip_disk_entry* after,
zzip_fnmatch_fn_t compare, int flags)
{
struct zzip_disk_entry* entry = (! after ? zzip_disk_findfirst (disk)
: zzip_disk_findnext (disk, after));
if (! compare) {
compare = (zzip_fnmatch_fn_t) _zzip_fnmatch;
if (disk->flags&1) disk->flags |= _zzip_fnmatch_CASEFOLD;
}
for (; entry ; entry = zzip_disk_findnext (disk, entry))
{
/* filenames within zip files are often not null-terminated! */
char* realname = zzip_disk_entry_strdup_name(disk, entry);
if (realname && ! compare(filespec, realname, flags))
{
free (realname);
return entry;
}
free (realname);
}
return 0;
}
/* ====================================================================== */
/** => zzip_disk_fopen
*
* the ZZIP_DISK_FILE* is rather simple in just encapsulating the
* arguments given to this function plus a zlib deflate buffer.
* Note that the ZZIP_DISK pointer does already contain the full
* mmapped file area of a zip disk, so open()ing a file part within
* that area happens to be a lookup of its bounds and encoding. That
* information is memorized on the ZZIP_DISK_FILE so that subsequent
* _read() operations will be able to get the next data portion or
* return an eof condition for that file part wrapped in the zip archive.
*/
zzip__new__ ZZIP_DISK_FILE*
zzip_disk_entry_fopen (ZZIP_DISK* disk, ZZIP_DISK_ENTRY* entry)
{
/* keep this in sync with zzip_mem_entry_fopen */
struct zzip_file_header* header =
zzip_disk_entry_to_file_header (disk, entry);
if (! header) return 0;
___ ZZIP_DISK_FILE* file = malloc(sizeof(ZZIP_DISK_FILE));
if (! file) return file;
file->buffer = disk->buffer;
file->endbuf = disk->endbuf;
file->avail = zzip_file_header_usize (header);
if (! file->avail || zzip_file_header_data_stored (header))
{ file->stored = zzip_file_header_to_data (header); return file; }
file->stored = 0;
file->zlib.opaque = 0;
file->zlib.zalloc = Z_NULL;
file->zlib.zfree = Z_NULL;
file->zlib.avail_in = zzip_file_header_csize (header);
file->zlib.next_in = zzip_file_header_to_data (header);
if (! zzip_file_header_data_deflated(header) ||
inflateInit2(&file->zlib, -MAX_WBITS) != Z_OK)
{ free (file); return 0; }
return file;
____;
}
/** openening a file part wrapped within a (mmapped) zip archive
*
* This function opens a file found by name, so it does a search into
* the zip central directory with => zzip_disk_findfile and whatever
* is found first is given to => zzip_disk_entry_fopen
*/
zzip__new__ ZZIP_DISK_FILE*
zzip_disk_fopen (ZZIP_DISK* disk, char* filename)
{
ZZIP_DISK_ENTRY* entry = zzip_disk_findfile (disk, filename, 0, 0);
if (! entry) return 0; else return zzip_disk_entry_fopen (disk, entry);
}
/** => zzip_disk_fopen
*
* This function reads more bytes into the output buffer specified as
* arguments. The return value is null on eof or error, the stdio-like
* interface can not distinguish between these so you need to check
* with => zzip_disk_feof for the difference.
*/
zzip_size_t
zzip_disk_fread (void* ptr, zzip_size_t sized, zzip_size_t nmemb,
ZZIP_DISK_FILE* file)
{
zzip_size_t size = sized*nmemb;
if (size > file->avail) size = file->avail;
if (file->stored)
{
memcpy (ptr, file->stored, size);
file->stored += size;
file->avail -= size;
return size;
}
file->zlib.avail_out = sized*nmemb;
file->zlib.next_out = ptr;
___ zzip_size_t total_old = file->zlib.total_out;
___ int err = inflate (& file->zlib, Z_NO_FLUSH);
if (err == Z_STREAM_END)
file->avail = 0;
else if (err == Z_OK)
file->avail -= file->zlib.total_out - total_old;
else
return 0;
return file->zlib.total_out - total_old;
____;____;
}
/** => zzip_disk_fopen
* This function releases any zlib decoder info needed for decompression
* and dumps the ZZIP_DISK_FILE* then.
*/
int
zzip_disk_fclose (ZZIP_DISK_FILE* file)
{
if (! file->stored)
inflateEnd (& file->zlib);
free (file);
return 0;
}
/** => zzip_disk_fopen
*
* This function allows to distinguish an error from an eof condition.
* Actually, if we found an error but we did already reach eof then we
* just keep on saying that it was an eof, so the app can just continue.
*/
int
zzip_disk_feof (ZZIP_DISK_FILE* file)
{
return ! file || ! file->avail;
}
Does anyone have any idea what's going on? Sorry if it may look obvious to some of you, I'm a newbie with this stuff.
Thanks!
Edit: Since I couldn't figure what's wrong, I decided to use the visual studio solution that came with the source code and I finally managed to build it successfully. Thanks to everyone who tried to help me with this.
The reason for your error is that in C language names enclosed in () are not treated as macros. That's actually a feature of the language that exists specifically for that purpose: to give you an opportunity to "ignore" macro definitions when you want to.
When you attempt to compile the code with GCC, it appears that strcasecmp is available. The true branch of the #if is taken by the compiler, which means that _zzip_strcasecmp gets defined as a macro
#define _zzip_strcasecmp strcasecmp
This will work fine as long as you call it as
_zzip_strcasecmp( /* whatever */ )
But this will not work when you refer to it as (_zzip_strcasecmp) here
compare = (zzip_strcmp_fn_t)( (disk->flags&1) ?
(_zzip_strcasecmp) : (strcmp));
The compiler sees (_zzip_strcasecmp) as a request to ignore macros when looking for _zzip_strcasecmp. The compiler will look for non-macro definition of _zzip_strcasecmp and fail with the error message, since such definition does not exist.
When you compile in Visual Siudio, strcasecmp is not available. The false branch of the #if is taken and _zzip_strcasecmp gets defined internally as an ordinary function. So, everything works fine.
One way to fix it for macro version is to remove these excessive () around function names in the problematic erroneous statement
compare = (zzip_strcmp_fn_t)( (disk->flags&1) ?
_zzip_strcasecmp : strcmp);
This should compile in GCC assuming strcasecmp is really available there.
I don't know why they put these superfluous () in the original version. Might be an instance of "rampant parenthesizing" and misguided "better safe than sorry" sentiment. Might actually be done intentionally to prevent use of macros in this context (in which case the bad guy is whoever wrote that #if).
I am trying to write a function to clean up the hash table that is generated by this code
/*
* Markov chain random text generator.
*/
#include <string.h>
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include "eprintf.h"
enum {
NPREF = 2, /* number of prefix words */
NHASH = 4093, /* size of state hash table array */
MAXGEN = 10000 /* maximum words generated */
};
typedef struct State State;
typedef struct Suffix Suffix;
struct State { /* prefix + suffix list */
char* pref[NPREF]; /* prefix words */
Suffix* suf; /* list of suffixes */
State* next; /* next in hash table */
};
struct Suffix { /* list of suffixes */
char* word; /* suffix */
Suffix* next; /* next in list of suffixes */
};
State* lookup(char *prefix[], int create);
void build(char *prefix[], FILE*);
void generate(int nwords);
void add(char *prefix[], char *word);
State* statetab[NHASH]; /* hash table of states */
char NONWORD[] = "\n"; /* cannot appear as real word */
/* markov main: markov-chain random text generation */
int main(void)
{
int i, nwords = MAXGEN;
char *prefix[NPREF]; /* current input prefix */
int c;
long seed;
setProgName("markov");
seed = time(NULL);
srand(seed);
for (i = 0; i < NPREF; i++) /* set up initial prefix */
prefix[i] = NONWORD;
build(prefix, stdin);
add(prefix, NONWORD);
generate(nwords);
return 0;
}
const int MULTIPLIER = 31; /* for hash() */
/* hash: compute hash value for array of NPREF strings */
unsigned int hash(char* s[NPREF])
{
unsigned int h;
unsigned char *p;
int i;
h = 0;
for (i = 0; i < NPREF; i++)
for (p = (unsigned char *) s[i]; *p != '\0'; p++)
h = MULTIPLIER * h + *p;
return h % NHASH;
}
/* lookup: search for prefix; create if requested. */
/* returns pointer if present or created; NULL if not. */
/* creation doesn't strdup so strings mustn't change later. */
State* lookup(char *prefix[NPREF], int create)
{
int i, h;
State *sp;
h = hash(prefix);
for (sp = statetab[h]; sp != NULL; sp = sp->next) {
for (i = 0; i < NPREF; i++)
if (strcmp(prefix[i], sp->pref[i]) != 0)
break;
if (i == NPREF) /* found it */
return sp;
}
if (create) {
sp = (State *) emalloc(sizeof(State));
for (i = 0; i < NPREF; i++)
sp->pref[i] = prefix[i];
sp->suf = NULL;
sp->next = statetab[h];
statetab[h] = sp;
}
return sp;
}
/* addsuffix: add to state. suffix must not change later */
void addsuffix(State *sp, char *suffix)
{
Suffix *suf;
suf = (Suffix *) emalloc(sizeof(Suffix));
suf->word = suffix;
suf->next = sp->suf;
sp->suf = suf;
}
/* add: add word to suffix list, update prefix */
void add(char *prefix[NPREF], char *suffix)
{
State *sp;
sp = lookup(prefix, 1); /* create if not found */
addsuffix(sp, suffix);
/* move the words down the prefix */
memmove(prefix, prefix+1, (NPREF-1)*sizeof(prefix[0]));
prefix[NPREF-1] = suffix;
}
/* build: read input, build prefix table */
void build(char *prefix[NPREF], FILE *f)
{
char buf[100], fmt[10];
/* create a format string; %s could overflow buf */
sprintf(fmt, "%%%ds", sizeof(buf)-1);
while (fscanf(f, fmt, buf) != EOF)
add(prefix, estrdup(buf));
}
/* generate: produce output, one word per line */
void generate(int nwords)
{
State *sp;
Suffix *suf;
char *prefix[NPREF], *w;
int i, nmatch;
for (i = 0; i < NPREF; i++) /* reset initial prefix */
prefix[i] = NONWORD;
for (i = 0; i < nwords; i++) {
sp = lookup(prefix, 0);
if (sp == NULL)
eprintf("internal error: lookup failed");
nmatch = 0;
for (suf = sp->suf; suf != NULL; suf = suf->next)
if (rand() % ++nmatch == 0) /* prob = 1/nmatch */
w = suf->word;
if (nmatch == 0)
eprintf("internal error: no suffix %d %s", i, prefix[0]);
if (strcmp(w, NONWORD) == 0)
break;
printf("%s\n", w);
memmove(prefix, prefix+1, (NPREF-1)*sizeof(prefix[0]));
prefix[NPREF-1] = w;
}
}
Here is what I have so far for my clean function
/*Clean Function*/
void clean_up(State *sp)
{
State *temp;
Suffix *temp2, temp3;
for(int h = 0; h < NHASH; h++)
{
for (sp = statetab[h]; sp != NULL; sp = sp->next)
{
while(sp->suf != NULL)
{
temp2= sp->suf;
temp3= *temp2->next;
free(temp2);
sp->suf= &temp3;
}
}
}
}
I think im on the right track, I'm going through each index in the hash table, then going from state to state and freeing the suffixes. I'm not sure what to do about the prefixes, because I have to free them before I can free each state. Any help would be greatly appreciated.
In your code, you are copying into a temp3 node, which lives in automatic memory ("on the stack") pointing sp->suf to this memory will (on the next iteration of the loop) cause free to be called with the address of this object (which has not been obtained by malloc, and thus cannot be freed by free() )
void clean_up(State *sp)
{
State *temp;
Suffix *temp2, **pp;
for(int h = 0; h < NHASH; h++)
{
for (sp = statetab[h]; sp != NULL; sp = sp->next)
{
for (pp = &sp->suf; *pp; *pp = temp2)
{
temp2 = (*pp)->next;
free(*pp);
}
}
}
}
The example code is derived from the Markov program in The Practice of Programming by Kernighan and Pike, a most excellent book.
Given that you are trying to clean up the statetab, the main clean-up function doesn't need any argument. You do have to be careful not to free the states directly in statetab, but you do need to release auxilliary states chained off statetab[i].next.
typedef struct State State;
typedef struct Suffix Suffix;
struct State { /* prefix + suffix list */
char* pref[NPREF]; /* prefix words */
Suffix* suf; /* list of suffixes */
State* next; /* next in hash table */
};
struct Suffix { /* list of suffixes */
char* word; /* suffix */
Suffix* next; /* next in list of suffixes */
};
State* statetab[NHASH]; /* hash table of states */
static void free_state(State *state);
static void free_suffix(Suffix *suffix);
static void cleanup(void)
{
for (int i = 0; i < NHASH; i++)
free_state(statetab[i]);
}
static void free_state(State *state)
{
if (state != 0)
{
for (int i = 0; i < NPREF; i++)
free(state->pref[i]);
free_suffix(state->suf);
if (state->next != 0)
{
free_state(state->next);
free(state->next);
}
}
}
static void free_suffix(Suffix *suffix)
{
if (suffix != 0)
{
free(suffix->word);
free_suffix(suffix->next);
free(suffix);
}
}
Do you see how I've designed the free_xxxx() code based on the design of the xxxx structure?
Caveat Lector: uncompiled code, much less tested code.
I dug up the code from the TPOP site, and tried to apply it. I made some fixes to the freeing code above (syntax error fixed, the null checks in free_state() and free_suffix()), but the code as a whole was not designed to allow the data to be freed.
There are a couple of problems. First, a few of the prefixes are not allocated (NONWORD). It might be possible to avoid releasing those by testing whether a prefix is NONWORD, but that's nasty. It might be possible to allocate those prefixes too (replace NONWORD by estrdup(NONWORD)). I think there's another place, somewhere, that a non-allocated pointer is being stashed in a prefix in the state table; I'm getting crashes in malloc() complaining of 'freeing non-allocated memory' (which is distinct from 'double freeing allocated memory', I believe), but I've not managed to resolve that.
However, that then changes to another problem; the prefixes are reused. That is, almost every prefix in the system is used as the the second word of one prefix, then as the first word of the next prefix. Thus, you can't readily free the prefixes.
If you were to design this so that the memory could be released, then you'd probably design it so that there was a system of 'atoms' (immutable strings) such that each word was allocated once and reused as often as necessary (see C Interfaces and Implementations: Techniques for Creating Reusable Code by D Hanson for the source of the term). The code freeing the state table would then concentrate only on the non-word data. There'd be code to release the complete set of atoms as well.
I ran the Markov program under valgrind without the cleanup; there are no memory access problems and no leaked data; it is all still accessible at program exit. I was using a data file of about 15,000 words (and about 2900 distinct words), and the statistics were:
==9610== HEAP SUMMARY:
==9610== in use at exit: 695,269 bytes in 39,567 blocks
==9610== total heap usage: 39,567 allocs, 0 frees, 695,269 bytes allocated
==9610==
==9610== LEAK SUMMARY:
==9610== definitely lost: 0 bytes in 0 blocks
==9610== indirectly lost: 0 bytes in 0 blocks
==9610== possibly lost: 0 bytes in 0 blocks
==9610== still reachable: 695,269 bytes in 39,567 blocks
So, you set yourself an interesting exercise. However, I think it is not achievable without reworking some of the memory allocation mechanism so that the data can be freed cleanly.
(On BSD, and hence on Mac OS X too, there are a pair of functions in <stdlib.h> called setprogname() and getprogname(). On BSD, setprogname() is called automatically before the main() gets going (with argv[0], I believe). The declaration in eprintf.h conflicts with the declaration in <stdlib.h>, which may be why the code in the question uses setProgName() instead of the original setprogname(). I chose to fix setprogname() in eprintf.h so that it took a const char * argument and therefore matched the declaration in <stdlib.h>.)
TPOP was previously at
http://plan9.bell-labs.com/cm/cs/tpop and
http://cm.bell-labs.com/cm/cs/tpop but both are now (2015-08-10) broken.
See also Wikipedia on TPOP.