computing physical addresses from virtual addresses in C - c

I'm struggling to write a function that translates virtual memory address to physical ones. I want the function to return the physical memory address of a virtual 16-bit address.
I'm just using 16-bit long virtual addresses that ignore permission bits (read,write, etc.). There are 256 pages in the page table so the base table register, which is BTR, just points to the table.
I want the function to either return:
Success: the physical address (a void*)
Page Fault: the virtual page number (an integer)
Protection Fault: a copy of the PTE
Here is the page table entry:
31 30..28 27..............................................4 3 2 1 0
+-----+------+-------------------------------------------------+-+-+-+-+
|Valid|unused| 24-bit Physical Page Number |P|R|W|X|
+-----+------+-------------------------------------------------+-+-+-+-+
I'm trying to learn how virtual memory works. I'm confused on how I would take a 16-bit virtual address and map it to a 32-bit page table entry, and then how to get the physical address from there. I've defined result_t and a union of the page table entry below, but I'm unsure how to use them. I've gotten a lot of help from looking online but everything is getting muddle together, I just want to learn how everything works straightforwardly.
Here are some necessary definitions:
extern void* BTR;
typedef struct result_st {
enum {SUCCESS, PAGEFAULT,
PROTFAULT, NOTIMPLEMENTED} status;
union {
void* pa;
unsigned vpn;
unsigned pte;
} value;
} result_t;
static result_t success(void* pa) {
result_t res;
res.status=SUCCESS;
res.value.pa = pa;
return res;
}
typedef union {
unsigned All;
struct {
unsigned Valid :1;
unsigned Unused :3;
unsigned PhysicalPageNumber :24;
unsigned SupervisoryMode :1;
unsigned Read :1;
unsigned Execute :1;
unsigned Write :1;
};
} PageTableEntry;
static int is_valid (unsigned pte) {
return 1 & (pte >> 31);
}
Here is the function I am writing:
result_t legacy(unsigned short va)
{
result_t result;
unsigned pte = va << 8;
result.value.pte = pte;
// This is my attempt so far.
// I want to use is_Valid() somewhere
void* pa = pte >> 8 | (va & 255);
return success(pa);
}
Thanks for any advice you can provide!

You are missing the definition of the actual page table. I assume it's like this (assuming I have understood your question correctly):
#define PAGE_TABLE_SIZE 256
PageTable page_table[PAGE_TABLE_SIZE];
Then your code would look something like this:
#define VIRT_PAGE_SIZE_BITS 8
/* Get virtual page number by dividing by the virt page size */
unsigned int virt_page_num = va >> VIRT_PAGE_SIZE_BITS;
assert(virt_page_num < PAGE_TABLE_SIZE); // Or do proper error handling
/* Use virtual page number to index into page table */
PageTableEntry pte = page_table[virt_page_num];
if (is_valid(pte)) {
if (is_access_ok(pte)) {
unsigned int phys_page_num = pte.PhysicalPageNumber;
return success(phys_page_num);
} else {
/* Protection fault code goes here */
}
} else {
/* Page fault code goes here */
}

Related

C Virtual to physical address mapping

I am beginner in C, I am trying to convert virtual addresses to physical.
My code so far, but don't know how to the translation.
I am using 4Kb pages.
The conversion I know is: for givin address: 0x12C000, the last three numbers won't be changed 000 and the remaining 12C will be converted, and then combine the fixed numbers with the converted. correct me if wrong, and how do I do taht in code ?
#include <stdio.h>
int main() {
uint page_table[512] = { 0 };
page_table[200] = 0x1234;
page_table[300] = 0x2345 ;
page_table[511] = 0x8000 ;
uint page_table_size = (sizeof(array)/sizeof(array[0]));
uint page_size_bits 12 // // 2^12 = 4KB;
uint mask_offset ((1<<page_size_bits)-1);
// example of correct outputs:
uint log_addr = 0x12C000;
/* should be 0x2345000 */
uint correctoutput;
log_addr = 0x12CFFF;
/* should be 0x2345FFF */
correctoutput;
log_addr = 0x1FF84A;
/* should be 0x800084A */
correctoutput;
}
Unless you are running in a privileged mode of the CPU, which usually means you are in the kernel you can't. The OS + CPU team up to prevent you from accessing the page tables, and the only way you could do this is by accessing the page tables.
If you did have access to them, and you were a 32bit 386 program running without PAE, it would look something like this:
extern void *mapphys(unsigned pa);
extern unsigned getcr3(void);
unsigned vtop(unsigned va) {
unsigned *pt, t;
pt = mapphys(getcr3());
t = pdir[va>>22];
unmapphys(pt);
if (t & 1) {
pt = mapphys(t & ~ 0xfff);
t = pt[(va >> 12) & 0x3ff];
unmapphys(pt);
if (t & 1) {
return (t &~ 0xfff) | (va & 0xfff);
}
error("page table entry undefined for %#x\n", va);
return -1;
} else {
error("page directory entry undefined for %#x\n", va);
return -2;
}
}
where maphys, unmapphys provide and remove a usable pointer to the given physical address, and getcr3() returns the page table base register from the 386.
That said, there are ways of constructing these sorts of page tables so that you can indirectly address them. For example, if you made the last (index 1023) entry in the page directory table point at the page directory table, then you can using the high 4M-4K of address space as a sort of page table map; and the final 4K of address space is a map of the page directory itself. With this setup, I can instead:
unsigned vtop(unsigned va) {
unsigned *pmap = (unsigned *)0xff800000;
unsigned *pdmap = (unsigned *)0xfffff000;
if (pdmap[va>>22] & 1) {
if (pmap[va>>12] & 1) {
return (pmap[va>>12]&~0xfff) | (va & 0xfff);
}
error("page table entry undefined for %#x\n, va);
return -1;
}
error("page dir entry undefined for %#x\n", va);
return -2;
}
It is worth noting that in this recursive page table, which index you choose is not important, and the mechanism is generally applicable to any page table definition which is the interior and leaf nodes have compatible layouts.
You need to calculate two values: The index into the physical memory (page_table) and the offset to add to the page_table entries address.
page_table_index = log_address >> 12;
page_offset = log_address & 0xfff;
physical = (page_table[page_table_index] << 12) | page_offset;
The lower 12 bits are the offset. You keep that part of the address. You shift these 12 bits off and you are left with the page number in the page_table array. You can then look up the page_table to obtain the base address, make space for the 12 bits in that address and insert the offset.

Translate virtual address to physical from multi-level pagetabels

I'm trying to convert an virtual memory address to a physical one, but can't get it to work. I'm currently doing an operating system as an assignment, and now I have to implement a printf function for the usermode, so when you invoke the write syscall the system should print the content of the array in usermode to the serial port(for now), and to do that i have to convert the address from virtual to physical.
Here is my code from the syscall handler:
Pcb* pcb = getCR3(); // contains the page directory for usermode
setCR3(kernelPageDir); // set the CR3 register to the kernel page directory
uint32_t tableNum = (vAddr >> 22) & 0x3ffUL; // get the upper 10 bits
uint32_t pageIndex = (vAddr >> 12) & 0x3ffUL // get the middle 10 bits
uint32_t offset = vAddr & 0xfffUL; // get the 12 lower bits
uint32_t* topTable = pcb->pageDirectory[tableNum]; // Access the top level table
uint32_t lowTable = topTable[pageIndex]; // Entry to the 2nd table
uint32_t* addr = lowTable + offset; // Should be the physical address
serialPrintf("Structure: tableNum=%08x pageIndex=%08x offset=%08x\n", tableNum, pageIndex, offset);
serialPrintf("Address: topTable=%08x lowTable=%08x addr=%08x\n",topTable, lowTable, addr);
serialPrintf("Char:%c", (char*)addr[0]);
When I run the code, it gives me a page fault when trying to access the value of it:
Structure: tableNum=00000020 pageIndex=00000048 offset=00000378
Address: topTable=00000000 lowTable=0015d000 addr=0015d378
Page fault! errcode=00000000 addr=0015d378
Here is the part from the book that explains the structure of the pages:
If you fill out a pte for the last (index 1023) entry in the second level table which is pointed to by the last (index 1023) of the first level table, then:
0xfffff000 .. 0xfffffffc will by an alias of the first level table,
and
0xffc00000 .. 0xffffffff will be an alias of the entire (sparse) page table.
For example:
int vtop(unsigned vaddr, unsigned *pa) {
unsigned *pdtb = (unsigned *)0xfffff000;
unsigned *pte = (unsigned *)0xffc00000;
if (ptdb[vaddr>>20] & 1) {
if (pte[vaddr>>12] & 1) {
*pa = pte[vaddr>>12] &~0xfff;
return 0;
}
return 2;
}
return 1;
}
You can do this for any index and adjust the pointer values, but if gets more confusing, and 0xffc/20 is out of the way.

How to get "valid data length" of a file?

There is a function to set the "valid data length" value: SetFileValidData, but I didn't find a way to get the "valid data length" value.
I want to know about given file if the EOF is different from the VDL, because writing after the VDL in case of VDL<EOF will cause a performance penalty as described here.
I found this page, claims that:
there is no mechanism to query the value of the VDL
So the answer is "you can't".
If you care about performance you can set the VDL to the EOF, but then note that you may allow access old garbage on your disk - the part between those two pointers, that supposed to be zeros if you would access that file without setting the VDL to point the EOF.
Looked into this. No way to get this information via any API, even the e.g. NtQueryInformationFile API (FileEndOfFileInformation only worked with NtSetInformationFile). So finally I read this by manually reading NTFS records. If anyone has a better way, please tell! This also obviously only works with full system access (and NTFS) and might be out of sync with the in-memory information Windows uses.
#pragma pack(push)
#pragma pack(1)
struct NTFSFileRecord
{
char magic[4];
unsigned short sequence_offset;
unsigned short sequence_size;
uint64 lsn;
unsigned short squence_number;
unsigned short hardlink_count;
unsigned short attribute_offset;
unsigned short flags;
unsigned int real_size;
unsigned int allocated_size;
uint64 base_record;
unsigned short next_id;
//char padding[470];
};
struct MFTAttribute
{
unsigned int type;
unsigned int length;
unsigned char nonresident;
unsigned char name_lenght;
unsigned short name_offset;
unsigned short flags;
unsigned short attribute_id;
unsigned int attribute_length;
unsigned short attribute_offset;
unsigned char indexed_flag;
unsigned char padding1;
//char padding2[488];
};
struct MFTAttributeNonResident
{
unsigned int type;
unsigned int lenght;
unsigned char nonresident;
unsigned char name_length;
unsigned short name_offset;
unsigned short flags;
unsigned short attribute_id;
uint64 starting_vnc;
uint64 last_vnc;
unsigned short run_offset;
unsigned short compression_size;
unsigned int padding;
uint64 allocated_size;
uint64 real_size;
uint64 initial_size;
};
#pragma pack(pop)
HANDLE GetVolumeData(const std::wstring& volfn, NTFS_VOLUME_DATA_BUFFER& vol_data)
{
HANDLE vol = CreateFileW(volfn.c_str(), GENERIC_WRITE | GENERIC_READ,
FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_ATTRIBUTE_NORMAL, NULL);
if (vol == INVALID_HANDLE_VALUE)
return vol;
DWORD ret_bytes;
BOOL b = DeviceIoControl(vol, FSCTL_GET_NTFS_VOLUME_DATA,
NULL, 0, &vol_data, sizeof(vol_data), &ret_bytes, NULL);
if (!b)
{
CloseHandle(vol);
return INVALID_HANDLE_VALUE;
}
return vol;
}
int64 GetFileValidData(HANDLE file, HANDLE vol, const NTFS_VOLUME_DATA_BUFFER& vol_data)
{
BY_HANDLE_FILE_INFORMATION hfi;
BOOL b = GetFileInformationByHandle(file, &hfi);
if (!b)
return -1;
NTFS_FILE_RECORD_INPUT_BUFFER record_in;
record_in.FileReferenceNumber.HighPart = hfi.nFileIndexHigh;
record_in.FileReferenceNumber.LowPart = hfi.nFileIndexLow;
std::vector<BYTE> buf;
buf.resize(sizeof(NTFS_FILE_RECORD_OUTPUT_BUFFER) + vol_data.BytesPerFileRecordSegment - 1);
NTFS_FILE_RECORD_OUTPUT_BUFFER* record_out = reinterpret_cast<NTFS_FILE_RECORD_OUTPUT_BUFFER*>(buf.data());
DWORD bout;
b = DeviceIoControl(vol, FSCTL_GET_NTFS_FILE_RECORD, &record_in,
sizeof(record_in), record_out, 4096, &bout, NULL);
if (!b)
return -1;
NTFSFileRecord* record = reinterpret_cast<NTFSFileRecord*>(record_out->FileRecordBuffer);
unsigned int currpos = record->attribute_offset;
MFTAttribute* attr = nullptr;
while ( (attr==nullptr ||
attr->type != 0xFFFFFFFF )
&& record_out->FileRecordBuffer + currpos +sizeof(MFTAttribute)<buf.data() + bout)
{
attr = reinterpret_cast<MFTAttribute*>(record_out->FileRecordBuffer + currpos);
if (attr->type == 0x80
&& record_out->FileRecordBuffer + currpos + attr->attribute_offset+sizeof(MFTAttributeNonResident)
< buf.data()+ bout)
{
if (attr->nonresident == 0)
return -1;
MFTAttributeNonResident* dataattr = reinterpret_cast<MFTAttributeNonResident*>(record_out->FileRecordBuffer
+ currpos + attr->attribute_offset);
return dataattr->initial_size;
}
currpos += attr->length;
}
return -1;
}
[...]
NTFS_VOLUME_DATA_BUFFER vol_data;
HANDLE vol = GetVolumeData(L"\\??\\D:", vol_data);
if (vol != INVALID_HANDLE_VALUE)
{
int64 vdl = GetFileValidData(alloc_test->getOsHandle(), vol, vol_data);
if(vdl>=0) { [...] }
[...]
}
[...]
The SetValidData (according to MSDN) can be used to create for example a large file without having to write to the file. For a database this will allocate a (contiguous) storage area.
As a result, it seems the file size on disk will have changed without any data having been written to the file.
By implication, any GetValidData (which does not exist) just returns the size of the file, so you can use GetFileSize which returns the "valid" file size.
I think you are confused as to what "valid data length" actually means. Check this answer.
Basically, while SetEndOfFile lets you increase the length of a file quickly, and allocates the disk space, if you skip to the (new) end-of-file to write there, all the additionally allocated disk space would need to be overwritten with zeroes, which is kind of slow.
SetFileValidData lets you skip that zeroing-out. You're telling the system, "I am OK with whatever is in those disk blocks, get on with it". (This is why you need the SE_MANAGE_VOLUME_NAME priviledge, as it could reveal priviledged data to unpriviledged users if you don't overwrite the data. Users with this priviledge can access the raw drive data anyway.)
In either case, you have set the new effective size of the file. (Which you can read back.) What, exactly, should a seperate "read file valid data" report back? SetFileValidData told the system that whatever is in those disk blocks is "valid"...
Different approach of explanation:
The documentation mentions that the "valid data length" is being tracked; the purpose for this is for the system to know which range (from end-of-valid-data to end-of-file) it still needs to zero out, in the context of SetEndOfFile, when necessary (e.g. you closing the file). You don't need to read back this value, because the only way it could be different from the actual file size is because you, yourself, did change it via the aforementioned functions...

Initializing, constructing and converting struct to byte array causes misalignment

I am trying to design a data structure (I have made it much shorter to save space here but I think you get the idea) to be used for byte level communication:
/* PACKET.H */
#define CM_HEADER_SIZE 3
#define CM_DATA_SIZE 16
#define CM_FOOTER_SIZE 3
#define CM_PACKET_SIZE (CM_HEADER_SIZE + CM_DATA_SIZE + CM_FOOTER_SIZE)
// + some other definitions
typedef struct cm_header{
uint8_t PacketStart; //Start Indicator 0x5B [
uint8_t DeviceId; //ID Of the device which is sending
uint8_t PacketType;
} CM_Header;
typedef struct cm_footer {
uint16_t DataCrc; //CRC of the 'Data' part of CM_Packet
uint8_t PacketEnd; //should be 0X5D or ]
} CM_Footer;
//Here I am trying to conver a few u8[4] tp u32 (4*u32 = 16 byte, hence data size)
typedef struct cm_data {
union {
struct{
uint8_t Value_0_0:2;
uint8_t Value_0_1:2;
uint8_t Value_0_2:2;
uint8_t Value_0_3:2;
};
uint32_t Value_0;
};
//same thing for Value_1, 2 and 3
} CM_Data;
typedef struct cm_packet {
CM_Header Header;
CM_Data Data;
CM_Footer Footer;
} CM_Packet;
typedef struct cm_inittypedef{
uint8_t DeviceId;
CM_Packet Packet;
} CM_InitTypeDef;
typedef struct cm_appendresult{
uint8_t Result;
uint8_t Reason;
} CM_AppendResult;
extern CM_InitTypeDef cmHandler;
The goal here is to make reliable structure for transmitting data over USB interface. At the end the CM_Packet should be converted to an uint8_t array and be given to data transmit register of an mcu (stm32).
In the main.c file I try to init the structure as well as some other stuff related to this packet:
/* MAIN.C */
uint8_t packet[CM_PACKET_SIZE];
int main(void) {
//use the extern defined in packet.h to init the struct
cmHandler.DeviceId = 0x01; //assign device id
CM_Init(&cmHandler); //construct the handler
//rest of stuff
while(1) {
CM_GetPacket(&cmHandler, (uint8_t*)packet);
CDC_Transmit_FS(&packet, CM_PACKET_SIZE);
}
}
And here is the implementation of packet.h which screws up everything so bad. I added the packet[CM_PACKET_SIZE] to watch but it is like it is just being generated randomly. Sometimes by pure luck I can see in this array some of the values that I am interested in! but it is like 1% of the time!
/* PACKET.C */
CM_InitTypeDef cmHandler;
void CM_Init(CM_InitTypeDef *cm_initer) {
cmHandler.DeviceId = cm_initer->DeviceId;
static CM_Packet cmPacket;
cmPacket.Header.DeviceId = cm_initer->DeviceId;
cmPacket.Header.PacketStart = CM_START;
cmPacket.Footer.PacketEnd = CM_END;
cm_initer->Packet = cmPacket;
}
CM_AppendResult CM_AppendData(CM_InitTypeDef *handler, uint8_t identifier,
uint8_t *data){
CM_AppendResult result;
switch(identifier){
case CM_VALUE_0:
handler->Packet.Data.Value_0_0 = data[0];
handler->Packet.Data.Value_0_1 = data[1];
handler->Packet.Data.Value_0_2 = data[2];
handler->Packet.Data.Value_0_3 = data[3];
break;
//Also cases for CM_VALUE_0, 1 , 2
//to build up the CM_Data sturct of CM_Packet
default:
result.Result = CM_APPEND_FAILURE;
result.Reason = CM_APPEND_CASE_ERROR;
return result;
break;
}
result.Result = CM_APPEND_SUCCESS;
result.Reason = 0x00;
return result;
}
void CM_GetPacket(CM_InitTypeDef *handler, uint8_t *packet){
//copy the whole struct in the given buffer and later send it to USB host
memcpy(packet, &handler->Packet, sizeof(CM_PACKET_SIZE));
}
So, the problem is this code gives me 99% of the time random stuff. It never has the CM_START which is the start indicator of packet to the value I want to. But most of the time it has the CM_END byte correctly! I got really confused and cant find out the reason. Being working on an embedded platform which is hard to debugg I am kind of lost here...
If you transfer data to another (different) architecture, do not just pass a structure as a blob. That is the way to hell: endianess, alignment, padding bytes, etc. all can (and likely will) cause trouble.
Better serialize the struct in a conforming way, possily using some interpreted control stream so you do not have to write every field out manually. (But still use standard functions to generate that stream).
Some areas of potential or likely trouble:
CM_Footer: The second field might very well start at a 32 or 64 bit boundary, so the preceeding field will be followed by padding. Also, the end of that struct is very likely to be padded by at least 1 bytes on a 32 bit architecture to allow for proper alignment if used in an array (the compiler does not care you if you actually need this). It might even be 8 byte aligned.
CM_Header: Here you likely (not guaranteed) get one uint8_t with 4*2 bits with the ordering not standardized. The field my be followed by 3 unused bytes which are required for the uint32_t interprettion of the union.
How do you guarantee the same endianess (for >uint8_t: high byte first or low byte first?) for host and target?
In general, the structs/unions need not have the same layout for host and target. Even if the same compiler is used, their ABIs may differ, etc. Even if it is the same CPU, there might be other system constraints. Also, for some CPUs, different ABIs (application binary interface) exist.

Activate paging in kernel programming

I'm reading this tutorial "http://www.jamesmolloy.co.uk/tutorial_html/6.-Paging.html" about kenrel programming and the author is using the below struct to build the page directory
typedef struct page_directory
{
/**
Array of pointers to pagetables.
**/
page_table_t *tables[1024];
/**
Array of pointers to the pagetables above, but gives their *physical*
location, for loading into the CR3 register.
**/
u32int tablesPhysical[1024];
/**
The physical address of tablesPhysical. This comes into play
when we get our kernel heap allocated and the directory
may be in a different location in virtual memory.
**/
u32int physicalAddr;
} page_directory_t;
my question is why he is loading the address of the page directory like this in the function void switch_page_directory(page_directory_t *new);
asm volatile("mov %0, %%cr3":: "r"(&dir->tablesPhysical));
and not like this
asm volatile("mov %0, %%cr3":: "r"(current_directory ));
I've been testing with as shown in the code below
#include<stdio.h>
#include<stdlib.h>
typedef struct page
{
unsigned int present : 1; // Page present in memory
unsigned int rw : 1; // Read-only if clear, readwrite if set
unsigned int user : 1; // Supervisor level only if clear
unsigned int accessed : 1; // Has the page been accessed since last refresh?
unsigned int dirty : 1; // Has the page been written to since last refresh?
unsigned int unused : 7; // Amalgamation of unused and reserved bits
unsigned int frame : 20; // Frame address (shifted right 12 bits)
} page_t;
typedef struct page_table
{
page_t pages[1024];
} page_table_t;
typedef struct page_directory
{
page_table_t *tables[1024];
unsigned int tablesPhysical[1024];
unsigned int physicalAddr;
} page_directory_t;
int main()
{
page_directory_t *n;
n = malloc(sizeof(page_directory_t));
printf("n=%p i=%p y=%p\n", n,&n->tablesPhysical, &n->tables);
}
the result is below
n=0x833b008 i=0x833c008 y=0x833b008
I'm not sure why the address is always the same for the printf?
your function is not written correctly, and you don't include the whole relevant code in the question.
That said, I'd say that the text is unclear, but I'd read the Multitasking segment, which covers what current_directory is eventually used for.

Resources