Creating a segregated list for a malloc() clone - c

Question

so I want to create my own malloc. I was given a library that gives me my own sbrk() function to increase my heap which basically uses malloc() to create "simulated" virtual memory.
Anyway, I stumbled across problems when implementing my free() method. Resulting in pointer confusion and questioning the whole existence of the universe. I thus create an example code.
#include <stdio.h>
#include <stdlib.h>
int main(void) {
size_t *seg_lists[4] = {NULL};
// Allocate two memory blocks of size 2
// First 8 bytes hold the size
// the last 2 8 bytes partition hold the data
void *mem1 = malloc(3*sizeof(size_t));
void *mem2 = malloc(3*sizeof(size_t));
// Set size fields
*(size_t *)mem1 = 2;
*(size_t *)mem2 = 2;
// Move pointers to start of body
mem1 = (size_t *)mem1 + 1;
mem2 = (size_t *)mem2 + 1;
// Set data to body of mem1
size_t* mem1_first = (size_t *)mem1;
size_t* mem1_second = (size_t *)mem1 + 1;
*mem1_first = 100;
*mem1_second = 200;
// Set data to body of mem2
size_t* mem2_first = (size_t *)mem2;
size_t* mem2_second = (size_t *)mem2 + 1;
*mem2_first = 300;
*mem2_second = 400;
// Output bodies of mem1 and mem2
printf("mem1=[%zu|%zu|%zu]\n", *((size_t *)mem1-1), *mem1_first, *mem1_second);
printf("mem2=[%zu|%zu|%zu]\n", *((size_t *)mem2 - 1), *mem2_first, *mem2_second);
// We first free mem2 and add it to the seg_list with index 1
size_t *head = seg_lists[1];
seg_lists[1] = mem2;
*(size_t *)mem2 = (size_t)head;
// We now want to free mem1 and add it to the list.
// I.e. we set the head to mem1 and the first 8 bytes of the body of
// mem2 to the address of mem1
head = seg_lists[1];
seg_lists[1] = mem1;
*(size_t *)mem1 = (size_t)head;
// Follow first entry in seg_lists[1] and print the 2nd 8 bytes of it's body
size_t *first = *seg_lists[1];
printf("*first=%zu\n", *(first+1));
}
So first I create two memory blocks of size 2. I.e. they can hold up to 16 bytes of data. The first 8 bytes are used to store the size. So it looks like this:
mem1=[2|100|200]
mem2=[2|300|400]
Next I want to free both memory blocks. The address of the next free block should be saved in the first 8 bytes of the body of previous bloc whereas the last should always point to NULL.
After freeing mem1, we want seg_list[1] point to the address of mem1 and the first 8 bytes of the body of mem1 to hold NULL.
After freeing mem2, we want seg_list[1] to point to the address of mem2 and the first 8 bytes of the body of mem2 should hold the address of mem1.
Now, I did the "save address of X in the first 8 bytes of the body of Y" by actually saving the hex value of the address in those bytes.
Now someone kept trying to tell me that's wrong. I shouldn't do that. I can't safe the address as a value but I can't see why I couldn't. It obviously works and for me, memory just holds data. An address is a hex value which I can store as plain data and then just treat it like however I want.
So is it really that bad what I'm doing here? Shouldn't I safe an address as "plain data"?