Develop your own operating system
Re: Develop your own operating system
Yeah, i can understand that. Do what you're interested in. I can't wait to get to the point where i make a window manager and a desktop environment. I'm curious how well my graphics driver will be able to handle it. VirtualBox provides a graphics adapter that's easy to control, but there's no 2d or 3d acceleration. Also, unless i make a window manager in ring0, i'll have to first write a disk driver and a file system driver so i can go to ring3. Unless i run the entire OS in ram, but that's no fun
¯\_(ツ)_/¯ It works on my machine...
Re: Develop your own operating system
Enabling Paging - Virtual Memory Management
Virtual memory is a way to isolate the address spaces of each process from eachother. Most importantly, it allows you to isolate ring3 (user mode) processes from kernel memory. If a user mode process tries to write to kernel memory, it will raise a page fault, which will allow you to kill the process and keep running without crashing. Otherwise, it would be very easy for a user mode program to overwrite kernel routines, causing crashes or strange behavior that is difficult to debug.
Enabling virtual memory paging requires a bit of work. We have to set up a page directory for each process, and fill in page tables to put in the page directory. We want to make a "higher half" kernel, so we'll be mapping our kernel to 0xC0100000. That means we won't be able to access memory using its physical location.
In other words, we'll have to make changes to various areas of our code. For example, our console code writes to video memory at 0x000B8000. We'll have to change this to 0xC00B8000, or we'll generate a page fault.
You'll want to make sure to set up your page fault handler to notify you of a page fault and halt the processor. When enabling paging, it's almost guaranteed you're going to run into a page fault unless you do everything perfectly - or you follow this guide to the letter. It's also a good idea to set up your double fault handler, as this will be called if an exception goes unhandled. If more than 2 exceptions go unhandled, the processor will triple-fault. A triple-fault causes the processor to reset, which appears as an instant reboot to the user.
For now, add the following code to your page fault handler in 'idt.c':
If you like, set up something similar for your general protection fault handler and your double fault handler.
Now, inside your 'memory' folder, create 2 new files: 'vm_manager.h' and 'vm_manager.c'.
Paste the following in 'vm_manager.h':
Looking at the above header, you'll see vmm_init, which is the function that sets up our page directory and page tables.
What exactly are these page directories and page tables? They're structures used by the processor to translate a virtual address into a physical one. Whenever there is any kind of memory access, no matter if it's a read, write, or execute, the processor checks these structures for a virtual memory to physical memory translation. Let's take a look...
A Page directory entry is made up of 1024 32-bit values, with the following format:
-Bits 31-12 are for a 4-kb page aligned address, which points to a page table structure
-Bits 11-9 are free to use for whatever you want
-Bit 8 is ignored
-Bit 7 denotes page size - 4mb when set, 4kb when cleared
-Bit 6 is 0
-Bit 5 is set by the CPU when the page is accessed
-Bit 4 disables caching when set
-Bit 3 when set, uses write-through caching, otherwise uses write-back caching
-Bit 2 when set, means this memory can be accessed by all. Otherwise, only ring0 can access it
-Bit 1 when set, the page is read/write, otherwise it's read-only
-Bit 0 when set, the page is present, otherwise not present
A Page table entry is very similar in structure to a page directory entry. A Page table is also an array of 1024 32-bit values:
-Bits 31-12 are for a 4-kb page aligned address, which points to a page aligned physical memory address
-Bits 11-9 are free to use for whatever you want
-Bit 8 when set, prevents the page from being cached in the TLB (must be enabled in CR4)
-Bit 7 is 0
-Bit 6 when set, the page has been written to
-Bit 5 is set by the CPU when the page is accessed
-Bit 4 disables caching when set
-Bit 3 when set, uses write-through caching, otherwise uses write-back caching
-Bit 2 when set, means this memory can be accessed by all. Otherwise, only ring0 can access it
-Bit 1 when set, the page is read/write, otherwise it's read-only
-Bit 0 when set, the page is present, otherwise not present
It can be a "bit" confusing, but this is how it's set up:
You create a page directory array, an array of 1024 DWORD values. Each element of the page directory array represents a single page table. Each page table covers 4MB of memory. A page table is an array of 1024 DWORD values. Each entry in the page table array represents a physical memory address that the virtual address of that table entry points to.
The page directory and page tables will be accessed depending on which virtual memory address you're accessing.
Each entry in the page directory represents a 4MB block of virtual memory. So:
Accesses to memory addresses between 0 and 4mb will go to page directory 0.
Accesses to memory addresses between 4mb and 8mb will go to page directory 1.
Accesses to memory addresses between 8mb and 12mb will go to page directory 2.
And so on.
Once you find the proper page directory, you then search the page table at that page directory.
Each entry in the page table represents a 4kb page of virtual memory. For each page of memory you try to access, it checks the page directory entry that corresponds to the 4mb block of memory the page is in, then it checks the page table entry corresponding to that specific page.
For example, if you want to read a value at address 0x00401000, which is "4mb and 4kb":
Hopefully that's clear enough. Basically, the page directory is a page table index of 4mb chunks of virtual memory, and each page table is a 4kb index of each 4mb chunk of virtual memory.
Whenever a page of memory is accessed, first the page directory entry that the address falls under is checked. If the page directory entry is not present, is not writeable (on memory write accesses), or the cpu is in ring3 trying to access ring0 memory, then a page fault will occur. If the page directory entry checked out, it then checks the corresponding entry in the page table. It also checks to make sure the page is present, checks for write access if needed, and if the cpu has permission to access this page and throws a page fault if not.
Contrary to what you may think, page faults are not always a bad thing. Like i mentioned above, they can be used to catch an error in a user mode process (null pointer exceptions, accessing unallocated memory "segmentation fault", etc). Also, if you want to swap a page from disk, then typically you set the 'Present' bit for that page to 0 (not present) so that an access of that page will cause a page fault, where you can load the page from disk.
Update your 'pm_manager.c' with the following function:
Make sure to add the definition to the header as well.
The above function allows us to mark sections of physical memory as allocated. More on this later.
Copy the following code into 'vm_manager.c':
Now let's look at what's needed to get our kernel running its virtual address space.
-We'll need to tell the linker to adjust the memory address of all symbols, except the entry point
-We have to remap the GDT to it's virtual address
-We have to move the kernel stack address to it's virtual counterpart
-We need to identity page the kernel (see below)
-We need to adjust any hard-coded memory addresses to their virtual counterparts
-We need to unmap the identity pages of the kernel after we call k_main
That's a lot of stuff! Most if it is pretty simple though.
Identity paging is mapping a virtual address to the same physical address. If we don't identity map the kernel, then as soon as we turn on paging, we'll immediately page fault. The page fault won't go to our page fault handler because it isn't installed yet, so it'll cause the CPU to triple fault! Why? Well, before we enable paging we're still able to access physical memory just fine. That means that while executing the code to enable paging, the EIP CPU register is executing at a physical address. If we enable paging without identity paging the kernel, then EIP will be executing at a virtual memory address that isn't mapped to any physical address. The processor won't know what to execute, so it throws an exception, which will lead to a triple fault.
Aside from that, identity paging is also needed because the GDT is stored at a physical address, so we have to remap that once virtual memory is enabled. Also, the kernel stack is at a physical address when paging is turned on, so we'll have to adjust that to its virtual address too or the stack will also point to a virtual address that isn't mapped.
The first thing we should update is our linker script:
Take a look at this section:
See the line that says ". = 1m;"? That's telling the linker to load the kernel into memory starting at the 1mb mark. Then we define the ._text section. After that, you can see we add 0xC0000000 to it. This tells the linker that the following sections are to be loaded into memory at 1M + the size of the ._text section + 0xC0000000. Lets take a look at the beginning of each of the following sections:
See the "AT(ADDR(.xxxx) - 0xC0000000" part of each section definition? This is telling the linker to load that section into memory at "." - 0xC0000000. This has the effect of the linker loading all sections into memory at the 1mb mark (0x001xxxxx), while linking the symbols at 0xC01xxxxx - the virtual address of our kernel.
Alright, so we have a function to initialize our virtual memory manager and set up a page directory and a few tables. Our linker script now puts all symbols at their proper virtual address. So, we can just call vmm_init() and go about our business, right? Not quite.
Let's take a look at kboot.asm:
You'll want to update your kboot.asm file with this code.
If you look at the code at the top of the above snippet, you'll see that we move a value into ESP. Currently, we've been using a small stack that was reserved inside the binary image of the kernel. Now that we're enabling paging, it is a good time to set up a proper stack for the kernel. One thing to keep in mind is that the stack grows DOWNWARD. This means that if your stack starts at address 0x1000, and you push some stuff onto it, it's not going to go 0x1000 ... 0x1004 ... 0x1008, it's going to go 0x1000 ... 0x0ffc ... 0x0ff8. So, we move the end of the memory range for the stack into ESP so that it can make use of all the space.
After we set up the new stack, we set up the GDT, then we add 0xC0000000 to EBX. Remember that grub puts the memory map pointer in EBX? Well that address is a physical one, so we have to adjust it to it's virtual address instead. Next, we move the vmm_init function pointer into eax, subtract 0xC0000000 from it, then call it. Are you noticing a pattern here? Yes, vmm_init is mapped to a virtual address that doesn't exist yet, so we have to adjust it to it's physical address to call it. Keep in mind that when you do this, the vmm_init function will not have access to any global variables or any other functions unless you create a pointer to them and adjust the value of the pointer. It can access variables on the stack without any problem, though.
Next, we come across this:
You can see we adjust ESP to its proper virtual address. So we can't use the stack until paging is actually enabled. No problem. Next we move eax into a register called cr3. Where did eax get its value? When we called the vmm_init function, the return value ended up in eax. The return value was the physical address that the new page directory was created at. So we're telling the processor where our page directory is. Then, we move the cr0 register into eax, OR it with 0x80000000, then move it back into cr0. This sets the flag needed to enable paging. After the 'mov cr0, eax' instruction, virtual memory paging is officially enabled.
At this point, if you did not identity page your kernel, you will get a triple fault and your virtual machine (or real machine) will reset. In that case, make sure you map virtual addresses 0x00100000 - 0x00400000 to the same physical addresses.
After we enable paging, we remap the GDT to its virtual address and then call our k_main function.
Inside of our k_main function, we call vmm_finish_init():
This simply unmaps the identity-paged kernel pages that were used while we were still executing at a physical address. We don't need these pages anymore, so we can unmap them. It also marks the physical memory addresses we used in our vmm_init() function as allocated, so the physical memory manager won't hand them out to the virtual memory manager later on. If doing this breaks your kernel, make sure you're not still accessing some memory location by it's physical address rather than it's virtual one.
For example, while writing this guide i spent 2 days tracking down a triple fault. My interrupt handlers were installed, but were not being called. If i did *NOT* call vmm_finish_init(), my kernel ran perfectly. So what could cause a triple fault when my interrupt handlers were all installed? In my case, it was that i forgot to adjust ESP to it's virtual address, so simply trying to pop or push something onto the stack was causing a page fault. Since the stack was inaccessible, when it tried to push the error code onto the stack for the page fault handler, it generated another exception, which then generated a third exception... and bam! - triple fault. So don't let it happen to you.
Now comes allocation. We're currently running in a virtual address space, so we can't use pmm_allocate for memory allocation anymore (see, i told you we wouldn't use it much), so we need a way to allocate free memory.
This following function is a real doozy, but i've commented it pretty heavily so just read it slowly and follow along. Keep in mind this function is incomplete. Upon failure, it does not completely clean up all allocations it made while trying to allocate the requested memory. It also only allocates memory in userspace memory, which is any address beneath our kernel (0xC0000000). Also, i finished writing and testing this function about an hour ago. It may contain a few bugs, which i will fix as i find them and update this code. Other than that, the function works fully as intended.
Since the function is about 400 lines of code, you can get it here:
http://code.suck-o.com/42562" onclick="window.open(this.href);return false;
As we continue developing our kernel, we'll keep revisiting our virtual memory manager code to make changes and additions to it. It's okay for our kernel to directly allocate memory using vmm_allocate, but on a per-process level, it's much better to allocate a heap for each process and provide memory from that heap. In this way, you can grow the heap in chunks - which means keeping the number of vmm_allocate() calls to a minimum.
I will leave it up to you to write your own vmm_free function. Keep in mind that we're going to allocate a heap for each process. Each process will notify the kernel when it can free some memory. I would make each process call vmm_free with the pointer originally supplied by vmm_allocate and the size of the allocation. In which case, you will only be able to free contiguous blocks of virtual memory.
Good luck!
The Keyboard Driver - Getting Keyboard Input
The Programmable Interval Timer - PIT
The Real Time Clock - RTC - Not sure if i'll cover this or not
Multi-Threading and Context Switching
Writing a Graphics Driver
Virtual memory is a way to isolate the address spaces of each process from eachother. Most importantly, it allows you to isolate ring3 (user mode) processes from kernel memory. If a user mode process tries to write to kernel memory, it will raise a page fault, which will allow you to kill the process and keep running without crashing. Otherwise, it would be very easy for a user mode program to overwrite kernel routines, causing crashes or strange behavior that is difficult to debug.
Enabling virtual memory paging requires a bit of work. We have to set up a page directory for each process, and fill in page tables to put in the page directory. We want to make a "higher half" kernel, so we'll be mapping our kernel to 0xC0100000. That means we won't be able to access memory using its physical location.
In other words, we'll have to make changes to various areas of our code. For example, our console code writes to video memory at 0x000B8000. We'll have to change this to 0xC00B8000, or we'll generate a page fault.
You'll want to make sure to set up your page fault handler to notify you of a page fault and halt the processor. When enabling paging, it's almost guaranteed you're going to run into a page fault unless you do everything perfectly - or you follow this guide to the letter. It's also a good idea to set up your double fault handler, as this will be called if an exception goes unhandled. If more than 2 exceptions go unhandled, the processor will triple-fault. A triple-fault causes the processor to reset, which appears as an instant reboot to the user.
For now, add the following code to your page fault handler in 'idt.c':
Code: Select all
setBgColor(CLR_BLUE);
setFgColor(CLR_LIGHTGRAY);
clearScreen();
printf("A Page Fault has occurred.");
while(1)
__asm("hlt"); // halt the processor
Now, inside your 'memory' folder, create 2 new files: 'vm_manager.h' and 'vm_manager.c'.
Paste the following in 'vm_manager.h':
Code: Select all
#ifndef VM_MANAGER_H
#define VM_MANAGER_H
#if !defined(__cplusplus)
#include <stdbool.h> /* C doesn't have booleans by default. */
#endif
#include <stddef.h>
#include <stdint.h>
#include "pm_manager.h"
#define SZ_4_MB 0x00400000 // 4mb value
#define SZ_1_MB 0x00100000 // 1mb value
#define SZ_4_KB 0x00001000 // 4kb value
#define PAGE_SIZE SZ_4_KB // alias
#define SIZE_TO_PAGE_COUNT(s) ((s % 0x1000 != 0) ? (s / 0x1000 + 1) : (s / 0x1000)) // converts a size in bytes to a number of pages
#define PAGE_DIRECTORY ((page_directory_t*) 0xFFFFF000) // this address is a self-reference to the page directory
#define EDIT_PAGE_TABLE(addr) (((page_table_t*) 0xFFFFD000)[1022] = ((addr & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1)) // map a page table for editing
#define PAGE_TABLE ((page_table_t*) 0xFFFFE000) // page tables are mapped to this virtual address for editing
typedef uint32_t page_directory_t;
typedef uint32_t page_table_t;
page_directory_t* vmm_init(void);
void vmm_finish_init(void);
void vmm_invalidate_page(unsigned long addr);
void* vmm_allocate(unsigned long pages);
#endif
What exactly are these page directories and page tables? They're structures used by the processor to translate a virtual address into a physical one. Whenever there is any kind of memory access, no matter if it's a read, write, or execute, the processor checks these structures for a virtual memory to physical memory translation. Let's take a look...
A Page directory entry is made up of 1024 32-bit values, with the following format:
-Bits 31-12 are for a 4-kb page aligned address, which points to a page table structure
-Bits 11-9 are free to use for whatever you want
-Bit 8 is ignored
-Bit 7 denotes page size - 4mb when set, 4kb when cleared
-Bit 6 is 0
-Bit 5 is set by the CPU when the page is accessed
-Bit 4 disables caching when set
-Bit 3 when set, uses write-through caching, otherwise uses write-back caching
-Bit 2 when set, means this memory can be accessed by all. Otherwise, only ring0 can access it
-Bit 1 when set, the page is read/write, otherwise it's read-only
-Bit 0 when set, the page is present, otherwise not present
A Page table entry is very similar in structure to a page directory entry. A Page table is also an array of 1024 32-bit values:
-Bits 31-12 are for a 4-kb page aligned address, which points to a page aligned physical memory address
-Bits 11-9 are free to use for whatever you want
-Bit 8 when set, prevents the page from being cached in the TLB (must be enabled in CR4)
-Bit 7 is 0
-Bit 6 when set, the page has been written to
-Bit 5 is set by the CPU when the page is accessed
-Bit 4 disables caching when set
-Bit 3 when set, uses write-through caching, otherwise uses write-back caching
-Bit 2 when set, means this memory can be accessed by all. Otherwise, only ring0 can access it
-Bit 1 when set, the page is read/write, otherwise it's read-only
-Bit 0 when set, the page is present, otherwise not present
It can be a "bit" confusing, but this is how it's set up:
You create a page directory array, an array of 1024 DWORD values. Each element of the page directory array represents a single page table. Each page table covers 4MB of memory. A page table is an array of 1024 DWORD values. Each entry in the page table array represents a physical memory address that the virtual address of that table entry points to.
The page directory and page tables will be accessed depending on which virtual memory address you're accessing.
Each entry in the page directory represents a 4MB block of virtual memory. So:
Accesses to memory addresses between 0 and 4mb will go to page directory 0.
Accesses to memory addresses between 4mb and 8mb will go to page directory 1.
Accesses to memory addresses between 8mb and 12mb will go to page directory 2.
And so on.
Once you find the proper page directory, you then search the page table at that page directory.
Each entry in the page table represents a 4kb page of virtual memory. For each page of memory you try to access, it checks the page directory entry that corresponds to the 4mb block of memory the page is in, then it checks the page table entry corresponding to that specific page.
For example, if you want to read a value at address 0x00401000, which is "4mb and 4kb":
Code: Select all
#define SIZE_OF_4MB 0x00400000
unsigned long read_address = 0x00401000;
unsigned long *PAGE_TABLE = PAGE_DIRECTORY[read_address / SIZE_OF_4MB]; // page directory entry 1
unsigned long PAGE = (read_address % SIZE_OF_4MB) / 0x1000; // PAGE = 1
unsigned long page_table_entry = PAGE_TABLE[PAGE]; // from here we can read or set data in the page table
Whenever a page of memory is accessed, first the page directory entry that the address falls under is checked. If the page directory entry is not present, is not writeable (on memory write accesses), or the cpu is in ring3 trying to access ring0 memory, then a page fault will occur. If the page directory entry checked out, it then checks the corresponding entry in the page table. It also checks to make sure the page is present, checks for write access if needed, and if the cpu has permission to access this page and throws a page fault if not.
Contrary to what you may think, page faults are not always a bad thing. Like i mentioned above, they can be used to catch an error in a user mode process (null pointer exceptions, accessing unallocated memory "segmentation fault", etc). Also, if you want to swap a page from disk, then typically you set the 'Present' bit for that page to 0 (not present) so that an access of that page will cause a page fault, where you can load the page from disk.
Update your 'pm_manager.c' with the following function:
Code: Select all
#define SIZE_TO_PAGE_COUNT(s) ((s % 0x1000 != 0) ? (s / 0x1000 + 1) : (s / 0x1000)) // converts a size in bytes to a number of pages
void pmm_set_allocated(unsigned long memaddr, unsigned long size)
{
unsigned long s_page = ADDR_TO_PAGE(memaddr);
unsigned long pagecount = SIZE_TO_PAGE_COUNT(size);
unsigned long s_block, s_bit;
s_block = s_page / 32;
s_bit = s_page - (s_block * 32);
for(unsigned long i = 0; i < pagecount; i++)
{
page_bitmap[s_block] |= (1 << (31 - s_bit));
s_bit++;
if(s_bit >= 32)
{
s_block++;
s_bit = 0;
}
}
}
The above function allows us to mark sections of physical memory as allocated. More on this later.
Copy the following code into 'vm_manager.c':
Code: Select all
#include "vm_manager.h"
/* *
*
* Page Directory Layout:
*
* Bits 31-12 (20 bits): 4kb Page-aligned page table address
* Bits 11-9 (3 bits): Use for whatever
* 1 Bit flags:
*
* G - Ignored
* S - Size; 1 for 4mb pages, 0 for 4kb pages
* 0 - Must be 0
* A - Accessed; is set to 1 when accessed
* D - Cache Disable; When set, page will not be cached
* W - Write-through caching; When set, uses write-through caching, otherwise uses write-back caching
* U - User/Supervisor; When set, can be accessed by all, otherwise only by kernel
* R - Read/Write; When set, page is read-write, otherwise read-only
* P - Present; When set, the page is present
*
* Page Table Layout:
* Like above, except:
*
* Address is a 4kb Page-aligned physical memory address
*
* 1 Bit flags:
*
* G - Global; Will not be flushed when CR3 is updated; Must set 'Global Enable' bit in CR4 to enable this feature
* 0 - 0
* D - Dirty; When set, the page has been written to
* A - Same as above
* C - Same as 'D' above
* W - Same as above
* U - Same as above
* R - Same as above
* P - Same as above
*
* */
extern void flush_tlb(void);
page_directory_t* vmm_init(void)
{
// Page Directory address
page_directory_t *page_directory = (page_directory_t*) 0x00400000;
// Page table for the kernel
page_table_t *kernel_page_table = (page_table_t*) 0x00401000;
// Page table fo the kernel stack - 64kb
page_table_t *stack_page_table = (page_table_t*) 0x00403000;
// Page table used to make the Page Directory self-referencing at address 0xFFFFF000
page_table_t *self_ref = (page_table_t*) 0x00402000;
// Set up the kernel page table entry
page_directory_t kpte = ((unsigned long) kernel_page_table & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1; // kernel page table
page_directory_t spte = ((unsigned long) stack_page_table & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1; // stack page table
page_directory_t srte = ((unsigned long) self_ref & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1; // self referencing page table
page_directory[768] = kpte;
page_directory[0] = kpte; // identity page the kernel
page_directory[1] = spte;
page_directory[769] = spte; // identity page the stack
page_directory[1023] = srte; // self referencing page directory entry
unsigned long kpage_phys_addr = 0x00001000;
unsigned long spage_phys_addr = 0x00500000;
// Loop through and clear out the kernel page table and self referencing page table
for(int i = 0; i < 1024; i++)
{
kernel_page_table[i] = 0;
stack_page_table[i] = 0;
self_ref[i] = 0;
}
// map the kernel page table and stack page table to their physical locations
for(int i = 1; i < 512; i++)
{
kernel_page_table[i] = (kpage_phys_addr & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1;
kpage_phys_addr += PAGE_SIZE;
}
for(int i = 256; i < 256 + 16; i++)
{
stack_page_table[i] = (spage_phys_addr & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1;
spage_phys_addr += PAGE_SIZE;
}
// make the kernel and stack page tables self-referencing
kernel_page_table[1023] = ((page_table_t) kernel_page_table & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1;
stack_page_table[1023] = ((page_table_t) stack_page_table & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1;
self_ref[1023] = ((unsigned long) page_directory & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1; // tell the self referencing page the address of the page directory
self_ref[1021] = ((unsigned long) self_ref & 0xFFFFF000) | (1 << 3) | (1 << 1) | 1; // map this page table to itself at 0xFFFFD000
return page_directory;
}
void vmm_finish_init(void)
{
// unmap the identity paged kernel pages
PAGE_DIRECTORY[0] = PAGE_DIRECTORY[0] & ((unsigned long) ~1);
pmm_set_allocated(0x1000, 511 * 0x1000);
pmm_set_allocated(0x400000, 4);
pmm_set_allocated(0x500000, 16);
// clear the changed pages from the cache
for(unsigned long i = 0; i < 512; i++)
vmm_invalidate_page(0x00000000 + (i * 0x1000));
}
// This function takes a single page and invalidates its entry in the TLB
void vmm_invalidate_page(unsigned long addr)
{
asm volatile("invlpg (%0)" ::"r" (addr) : "memory");
}
-We'll need to tell the linker to adjust the memory address of all symbols, except the entry point
-We have to remap the GDT to it's virtual address
-We have to move the kernel stack address to it's virtual counterpart
-We need to identity page the kernel (see below)
-We need to adjust any hard-coded memory addresses to their virtual counterparts
-We need to unmap the identity pages of the kernel after we call k_main
That's a lot of stuff! Most if it is pretty simple though.
Identity paging is mapping a virtual address to the same physical address. If we don't identity map the kernel, then as soon as we turn on paging, we'll immediately page fault. The page fault won't go to our page fault handler because it isn't installed yet, so it'll cause the CPU to triple fault! Why? Well, before we enable paging we're still able to access physical memory just fine. That means that while executing the code to enable paging, the EIP CPU register is executing at a physical address. If we enable paging without identity paging the kernel, then EIP will be executing at a virtual memory address that isn't mapped to any physical address. The processor won't know what to execute, so it throws an exception, which will lead to a triple fault.
Aside from that, identity paging is also needed because the GDT is stored at a physical address, so we have to remap that once virtual memory is enabled. Also, the kernel stack is at a physical address when paging is turned on, so we'll have to adjust that to its virtual address too or the stack will also point to a virtual address that isn't mapped.
The first thing we should update is our linker script:
Code: Select all
/* The bootloader will look at this image and start execution at the symbol
designated as the entry point. */
ENTRY(_start)
/* Tell where the various sections of the object files will be put in the final
kernel image. */
SECTIONS
{
/* Begin putting sections at 1 MiB, a conventional place for kernels to be
loaded at by the bootloader. */
. = 1M;
._text BLOCK(4K) : ALIGN(4K)
{
*(.multiboot)
*(._text)
}
. += 0xC0000000;
/* First put the multiboot header, as it is required to be put very early
early in the image or the bootloader won't recognize the file format.
Next we'll put the .text section. */
.text ALIGN(4K) : AT(ADDR(.text) - 0xC0000000)
{
*(.text)
}
/* Read-only data. */
.rodata ALIGN(4K) : AT(ADDR(.rodata) - 0xC0000000)
{
*(.rodata)
}
/* Read-write data (initialized) */
.data ALIGN(4K) : AT(ADDR(.data) - 0xC0000000)
{
*(.data)
}
/* Read-write data (uninitialized) and stack */
.bss ALIGN(4K) : AT(ADDR(.bss) - 0xC0000000)
{
*(COMMON)
*(.bss)
*(.bootstrap_stack)
}
_KERNEL_END_ADDR = .;
/* The compiler may produce other sections, by default it will put them in
a segment with the same name. Simply add stuff here as needed. */
}
Code: Select all
. = 1M;
._text BLOCK(4K) : ALIGN(4K)
{
*(.multiboot)
*(._text)
}
. += 0xC0000000;
Code: Select all
.text ALIGN(4K) : AT(ADDR(.text) - 0xC0000000)
{
*(.text)
}
.rodata ALIGN(4K) : AT(ADDR(.rodata) - 0xC0000000)
{
*(.rodata)
}
.data ALIGN(4K) : AT(ADDR(.data) - 0xC0000000)
{
*(.data)
}
.bss ALIGN(4K) : AT(ADDR(.bss) - 0xC0000000)
{
*(COMMON)
*(.bss)
*(.bootstrap_stack)
}
Alright, so we have a function to initialize our virtual memory manager and set up a page directory and a few tables. Our linker script now puts all symbols at their proper virtual address. So, we can just call vmm_init() and go about our business, right? Not quite.
Let's take a look at kboot.asm:
Code: Select all
cli
mov esp, 0x00500000 + (0x1000 * 16)
gdt_install:
lgdt [m_gdt]
mov ax, 0x10
mov ds, ax
mov es, ax
mov fs, ax
mov gs, ax
mov ss, ax
jmp 0x08:gdt_return
gdt_start:
; null descriptor
dd 0
dd 0
;code descriptor
dw 0xFFFF ; limit low
dw 0x0000 ; base low
db 0x00 ; base middle
db 0x9A ; access
db 0xCF ; granularity
db 0x00 ; base high
;data descriptor
dw 0xFFFF ; limit low
dw 0x0000 ; base low
db 0x00 ; base middle
db 0x92 ; access
db 0xCF ; granularity
db 0x00 ; base high
gdt_end:
m_gdt:
dw gdt_end - gdt_start - 1;
dd gdt_start
gdt_return:
add ebx, 0xC0000000;
push ebx ; grub memory map
extern vmm_init
mov eax, vmm_init
sub eax, 0xC0000000
call eax
add esp, 0xC0000000
mov cr3, eax
mov eax, cr0
or eax, 0x80000000
mov cr0, eax
jmp reload_gdt
m_gdt2:
dw gdt_end - gdt_start - 1
dd gdt_start + 0xC0000000
reload_gdt:
lgdt [m_gdt2]
extern k_main
call k_main
If you look at the code at the top of the above snippet, you'll see that we move a value into ESP. Currently, we've been using a small stack that was reserved inside the binary image of the kernel. Now that we're enabling paging, it is a good time to set up a proper stack for the kernel. One thing to keep in mind is that the stack grows DOWNWARD. This means that if your stack starts at address 0x1000, and you push some stuff onto it, it's not going to go 0x1000 ... 0x1004 ... 0x1008, it's going to go 0x1000 ... 0x0ffc ... 0x0ff8. So, we move the end of the memory range for the stack into ESP so that it can make use of all the space.
After we set up the new stack, we set up the GDT, then we add 0xC0000000 to EBX. Remember that grub puts the memory map pointer in EBX? Well that address is a physical one, so we have to adjust it to it's virtual address instead. Next, we move the vmm_init function pointer into eax, subtract 0xC0000000 from it, then call it. Are you noticing a pattern here? Yes, vmm_init is mapped to a virtual address that doesn't exist yet, so we have to adjust it to it's physical address to call it. Keep in mind that when you do this, the vmm_init function will not have access to any global variables or any other functions unless you create a pointer to them and adjust the value of the pointer. It can access variables on the stack without any problem, though.
Next, we come across this:
Code: Select all
add esp, 0xC0000000
mov cr3, eax
mov eax, cr0
or eax, 0x80000000
mov cr0, eax
At this point, if you did not identity page your kernel, you will get a triple fault and your virtual machine (or real machine) will reset. In that case, make sure you map virtual addresses 0x00100000 - 0x00400000 to the same physical addresses.
After we enable paging, we remap the GDT to its virtual address and then call our k_main function.
Inside of our k_main function, we call vmm_finish_init():
Code: Select all
void vmm_finish_init(void)
void vmm_finish_init(void)
{
// unmap the identity paged kernel pages
PAGE_DIRECTORY[0] = PAGE_DIRECTORY[0] & ((unsigned long) ~1);
pmm_set_allocated(0x1000, 511 * 0x1000);
pmm_set_allocated(0x400000, 4);
pmm_set_allocated(0x500000, 16);
// clear the changed pages from the cache
for(unsigned long i = 0; i < 512; i++)
vmm_invalidate_page(0x00000000 + (i * 0x1000));
}
For example, while writing this guide i spent 2 days tracking down a triple fault. My interrupt handlers were installed, but were not being called. If i did *NOT* call vmm_finish_init(), my kernel ran perfectly. So what could cause a triple fault when my interrupt handlers were all installed? In my case, it was that i forgot to adjust ESP to it's virtual address, so simply trying to pop or push something onto the stack was causing a page fault. Since the stack was inaccessible, when it tried to push the error code onto the stack for the page fault handler, it generated another exception, which then generated a third exception... and bam! - triple fault. So don't let it happen to you.
Now comes allocation. We're currently running in a virtual address space, so we can't use pmm_allocate for memory allocation anymore (see, i told you we wouldn't use it much), so we need a way to allocate free memory.
This following function is a real doozy, but i've commented it pretty heavily so just read it slowly and follow along. Keep in mind this function is incomplete. Upon failure, it does not completely clean up all allocations it made while trying to allocate the requested memory. It also only allocates memory in userspace memory, which is any address beneath our kernel (0xC0000000). Also, i finished writing and testing this function about an hour ago. It may contain a few bugs, which i will fix as i find them and update this code. Other than that, the function works fully as intended.
Since the function is about 400 lines of code, you can get it here:
http://code.suck-o.com/42562" onclick="window.open(this.href);return false;
As we continue developing our kernel, we'll keep revisiting our virtual memory manager code to make changes and additions to it. It's okay for our kernel to directly allocate memory using vmm_allocate, but on a per-process level, it's much better to allocate a heap for each process and provide memory from that heap. In this way, you can grow the heap in chunks - which means keeping the number of vmm_allocate() calls to a minimum.
I will leave it up to you to write your own vmm_free function. Keep in mind that we're going to allocate a heap for each process. Each process will notify the kernel when it can free some memory. I would make each process call vmm_free with the pointer originally supplied by vmm_allocate and the size of the allocation. In which case, you will only be able to free contiguous blocks of virtual memory.
Good luck!
The Keyboard Driver - Getting Keyboard Input
The Programmable Interval Timer - PIT
The Real Time Clock - RTC - Not sure if i'll cover this or not
Multi-Threading and Context Switching
Writing a Graphics Driver
¯\_(ツ)_/¯ It works on my machine...
Re: Develop your own operating system
Turns out there is limit to the number of characters in a post. It's 60,000. Lol.
Anyway, i guess i'll just continue the guide in my above post. Sorry that the section on virtual memory took so long to get done. I ran into a few problems while writing the code, not to mention how much code i had to write... lol. Anyway, i hope everything in the virtual memory section is clear. If anybody has any questions or comments, feel free to post them here.
Until next time, fellas.
Anyway, i guess i'll just continue the guide in my above post. Sorry that the section on virtual memory took so long to get done. I ran into a few problems while writing the code, not to mention how much code i had to write... lol. Anyway, i hope everything in the virtual memory section is clear. If anybody has any questions or comments, feel free to post them here.
Until next time, fellas.
¯\_(ツ)_/¯ It works on my machine...
Re: Develop your own operating system
woot thx man , will be waiting for next
Re: Develop your own operating system
Made this a sticky cause holy fuck man!
Re: Develop your own operating system
Indeed very impressive!
"The best place to hide a tree, is in a forest"
Re: Develop your own operating system
I made same changes to the section on virtual memory. Just grammar and spelling fixes and stuff.
So has anybody read the full section on virtual memory? I actually had difficulty explaining some parts of it, so hopefully it all came across clearly.
Anyway, the section on keyboard input shouldn't take nearly as long, so expect to see it soon ^_^
So has anybody read the full section on virtual memory? I actually had difficulty explaining some parts of it, so hopefully it all came across clearly.
Anyway, the section on keyboard input shouldn't take nearly as long, so expect to see it soon ^_^
¯\_(ツ)_/¯ It works on my machine...
Re: Develop your own operating system
I have not read all the info as of yet. I do have plans to go over it as I think this information is really quite incredible.
I have plans to write a custom kernel for a unix system and this information will be very helpful.
Thank you Gogeta for all of the hard work put into this.
*cheers Maboroshi
I have plans to write a custom kernel for a unix system and this information will be very helpful.
Thank you Gogeta for all of the hard work put into this.
*cheers Maboroshi